Nagios Log server logstash behaving crazy

[anish]

Hi Team ,

There is some problem with the Nagios log server . Suddenly was able to see indexes for older dates like logstash -2014.10.05 .We didn't even built the Nagios Log server in 2014. Please check the screen shot attached . What could be the reason for the indexes to appear in the Nagios Log console for the older dates.

[cdienger]

Is the date set properly on the machine or has it been modified recently? What is the output of:

Code: Select all

date +%F
date %s

also check the the index names under /usr/local/nagioslogserver/elasticsearch/data/*CLUSTERID*/nodes/0/indices. Are there any indices in there with a 2014 timestamp?

[anish]

Please find the output

[root@SESKLNGLSIPD01 ~]# date +%F
2017-06-16
[root@SESKLNGLSIPD01 ~]# date %s
date: invalid date `%s'

Checked the under /usr/local/nagioslogserver/elasticsearch/data/*CLUSTERID*/nodes/0/indices ,,I can see 2014 dated indices and logstash-2014.06.16 was created today . May i know the reason behind.


drwxr-xr-x 8 nagios users 4096 Oct 20 2016 kibana-int
drwxr-xr-x 8 nagios users 4096 Jun 15 10:36 logstash-2014.06.15
drwxr-xr-x 8 nagios users 4096 Jun 15 20:00 logstash-2014.06.16
drwxr-xr-x 8 nagios users 4096 Jun 10 14:01 logstash-2017.06.11
drwxr-xr-x 8 nagios users 4096 Jun 14 09:56 logstash-2017.06.12
drwxr-xr-x 8 nagios users 4096 Jun 14 14:56 logstash-2017.06.13
drwxr-xr-x 8 nagios users 4096 Jun 14 09:11 logstash-2017.06.14
drwxr-xr-x 8 nagios users 4096 Jun 14 14:58 logstash-2017.06.15
drwxr-xr-x 8 nagios users 4096 Jun 15 14:01 logstash-2017.06.16
drwxr-xr-x 4 nagios users 4096 Oct 20 2016 nagioslogserver
drwxr-xr-x 8 nagios users 4096 Mar 28 09:54 nagioslogserver_log

[cdienger]

Hi Anish,

Can you PM me a profile or otherwise make it available somewhere for me to download? If you'd like to password protect it, please PM me the password. I'd like to see what you have setup for logstash filters. I'd also like to get copies of the files in /var/log/elasticsearch and /var/log/logstash.

[anish]

Please find the attachment regarding Nagios Log Profile

[anish]

Please find the attachments regarding the copies of the files in /var/log/elasticsearch

[anish]

Please find the attachments regarding the copies of the files in /var/log/logstash

[cdienger]

Some of the data was removed or not collected in the profile that I was hoping to verify. The main part being the output filter. Can you PM that to me along with the other files? I'd also like to see the output of:

Code: Select all

curl -XGET 'http://localhost:9200/_search/template?pretty'

[anish]

please find the output

[cdienger]

The 2014 indices don't appear to contain much(a few kb). Go ahead and delete them and then restart the service with:

Code: Select all

service elasticsearch restart

Restart the service on one of the nodes and after it comes back up, restart the service on the other node.