Unable to apply configuration for 2 instances

[uma K]

Hi,
My Nagios log server works with 4 instances, when verify configuration changes it says "Configuration is OK"

When I try to apply configuration,2 instances get applied successfully and 2 instances throws exception as "The instance is likely offline, please check and try again"

And another issue that I face is , today date is 04/25/2018 but my logstash create an index for tomorw date also 04/26/2018.
Please help me to resolve the issue because of which I have multiple exception on logstash logs.

{:timestamp=>"2018-04-25T18:08:14.632000-0700", :message=>"failed action with response of 403, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2018.04.26\", :_type=>\"eventlog\",,

[cdienger]

Try restarting the services on the two instances that are having problems:

service logstash restart
service elasticsearch restart

If the problem persists or if you receive an error running these, please provide copies of /var/log/logstash/logstash.log and /var/log/elasticsearch/<cluster_id>.log

For the second issue, check the date and time on the NLS server as well as the clients. Also, have a look at https://support.nagios.com/forum/viewto ... 38&t=48245 which sounds very similar to this. If you're unable to resolve it, please open a new thread to avoid having to troubleshoot two issues in a single thread.

[uma K]

EVen after the restart I do see the same issue.
Please find the attached logs for all the 4 servers.
I have issue with server 3 and 4 to apply configuration.

[cdienger]

Are the 4 appliances on the same network segment? The logs indicate there is a problem with port 9300. Make sure port 9300 is up and listening on all devices and that there isn't a firewall blocking the traffic.

You can check the port is up with:

netstat -an | grep 9300

and check local iptables with:

iptables -L

the above iptables will show you local filter rules. If there is a firewall or other device between the machines you'll need to look on those as well to make sure the port isn't getting dropped/blocked.