Page 1 of 2
Nagios Log server logstash behaving crazy
Posted: Thu Jun 15, 2017 7:00 am
by anish
Hi Team ,
There is some problem with the Nagios log server . Suddenly was able to see indexes for older dates like logstash -2014.10.05 .We didn't even built the Nagios Log server in 2014. Please check the screen shot attached . What could be the reason for the indexes to appear in the Nagios Log console for the older dates.
Re: Nagios Log server logstash behaving crazy
Posted: Thu Jun 15, 2017 10:53 am
by cdienger
Is the date set properly on the machine or has it been modified recently? What is the output of:
also check the the index names under /usr/local/nagioslogserver/elasticsearch/data/*CLUSTERID*/nodes/0/indices. Are there any indices in there with a 2014 timestamp?
Re: Nagios Log server logstash behaving crazy
Posted: Fri Jun 16, 2017 7:34 am
by anish
Please find the output
[root@SESKLNGLSIPD01 ~]# date +%F
2017-06-16
[root@SESKLNGLSIPD01 ~]# date %s
date: invalid date `%s'
Checked the under /usr/local/nagioslogserver/elasticsearch/data/*CLUSTERID*/nodes/0/indices ,,I can see 2014 dated indices and logstash-2014.06.16 was created today . May i know the reason behind.
drwxr-xr-x 8 nagios users 4096 Oct 20 2016 kibana-int
drwxr-xr-x 8 nagios users 4096 Jun 15 10:36 logstash-2014.06.15
drwxr-xr-x 8 nagios users 4096 Jun 15 20:00 logstash-2014.06.16
drwxr-xr-x 8 nagios users 4096 Jun 10 14:01 logstash-2017.06.11
drwxr-xr-x 8 nagios users 4096 Jun 14 09:56 logstash-2017.06.12
drwxr-xr-x 8 nagios users 4096 Jun 14 14:56 logstash-2017.06.13
drwxr-xr-x 8 nagios users 4096 Jun 14 09:11 logstash-2017.06.14
drwxr-xr-x 8 nagios users 4096 Jun 14 14:58 logstash-2017.06.15
drwxr-xr-x 8 nagios users 4096 Jun 15 14:01 logstash-2017.06.16
drwxr-xr-x 4 nagios users 4096 Oct 20 2016 nagioslogserver
drwxr-xr-x 8 nagios users 4096 Mar 28 09:54 nagioslogserver_log
Re: Nagios Log server logstash behaving crazy
Posted: Fri Jun 16, 2017 10:22 am
by cdienger
Hi Anish,
Can you PM me a profile or otherwise make it available somewhere for me to download? If you'd like to password protect it, please PM me the password. I'd like to see what you have setup for logstash filters. I'd also like to get copies of the files in /var/log/elasticsearch and /var/log/logstash.
Re: Nagios Log server logstash behaving crazy
Posted: Thu Jun 22, 2017 7:24 am
by anish
Please find the attachment regarding Nagios Log Profile
Re: Nagios Log server logstash behaving crazy
Posted: Thu Jun 22, 2017 7:33 am
by anish
Please find the attachments regarding the copies of the files in /var/log/elasticsearch
Re: Nagios Log server logstash behaving crazy
Posted: Thu Jun 22, 2017 7:34 am
by anish
Please find the attachments regarding the copies of the files in /var/log/logstash
Re: Nagios Log server logstash behaving crazy
Posted: Thu Jun 22, 2017 10:03 am
by cdienger
Some of the data was removed or not collected in the profile that I was hoping to verify. The main part being the output filter. Can you PM that to me along with the other files? I'd also like to see the output of:
Code: Select all
curl -XGET 'http://localhost:9200/_search/template?pretty'
Re: Nagios Log server logstash behaving crazy
Posted: Thu Jun 22, 2017 10:18 am
by anish
please find the output
Re: Nagios Log server logstash behaving crazy
Posted: Thu Jun 22, 2017 10:37 am
by cdienger
The 2014 indices don't appear to contain much(a few kb). Go ahead and delete them and then restart the service with:
Restart the service on one of the nodes and after it comes back up, restart the service on the other node.