Page 1 of 2
Problems after upgrade to 1.4
Posted: Thu Jan 07, 2016 11:43 am
by gregwhite
After upgrading to NLS 1.4 the cluster status is red and the current index is 0. I closed the index and the cluster status turned yellow and it said it was receiving data from all the hosts. However, no data appears to be coming in. Also, Logstash would periodically crash.
Re: Problems after upgrade to 1.4
Posted: Thu Jan 07, 2016 12:02 pm
by jolson
Are you certain that Logstash is running on your systems (Run these commands on all nodes)?
Code: Select all
service logstash status
service logstash restart
I would also like the following information from you (Run these commands on a single node):
Code: Select all
curl 'localhost:9200/_cat/nodes?v'
curl -XGET 'localhost:9200/_cat/recovery?v'
curl 'localhost:9200/_cluster/health?level=indices&pretty'
Re: Problems after upgrade to 1.4
Posted: Thu Jan 07, 2016 1:26 pm
by gregwhite
output.txt
Here is the output from the 3 CURL commands.
I did the service status for logstash and it was running but it keeps stopping.
Re: Problems after upgrade to 1.4
Posted: Thu Jan 07, 2016 1:33 pm
by jolson
Is there anything notable in the Logstash log?
Code: Select all
tail -n50 /var/log/logstash/logstash.log
You might try bumping the Logstash HEAP up to 1024m and giving it another try:
LS_HEAP_SIZE="1024m"
Please note that you must remove the comment (#) in front of the LS_HEAP_SIZE setting.
After upping the HEAP to 1024, try restarting Logstash again:
I'm also interested in the memory consumption of your system as a whole:
Re: Problems after upgrade to 1.4
Posted: Thu Jan 07, 2016 2:32 pm
by gregwhite
ls_heap_size was already increased to 1024 last month.
Free Memory = [root@lkenmycroftp01 logstash]# free -m
total used free shared buffers cached
Mem: 32071 31882 189 0 61 12709
-/+ buffers/cache: 19111 12960
Swap: 16103 72 16031
Re: Problems after upgrade to 1.4
Posted: Thu Jan 07, 2016 2:40 pm
by jolson
Anything in the logstash log, or did it turn up blank?
Re: Problems after upgrade to 1.4
Posted: Thu Jan 07, 2016 4:01 pm
by gregwhite
I sent a copy of the log to support hoping it would get to you. The only thing that stood out to me was:
ar/perfdata/APO_F3_2960-S13/If_FastEthernet0_31.rrd) failed with status -1. (/usr/local/nagios/
var/perfdata/APO_F3_2960-S13/If_FastEthernet0_31.rrd: found extra data on update argument: 0:0:
0:0)\\\\\\\\n\\\\\\\", \\\\\\\"@version\\\\\\\"=>\\\\\\\"1\\\\\\\", \\\\\\\"@timestamp\\\\\\\"=
>\\\\\\\"2016-01-06T06:21:33.000Z\\\\\\\", \\\\\\\"type\\\\\\\"=>\\\\\\\"syslog\\\\\\\", \\\\\\
\"host\\\\\\\"=>\\\\\\\"172.22.3.18\\\\\\\", \\\\\\\"priority\\\\\\\"=>29, \\\\\\\"timestamp\\\
\\\\"=>\\\\\\\"Jan 6 01:21:33\\\\\\\", \\\\\\\"logsource\\\\\\\"=>\\\\\\\"lkennagiost02\\\\\\\
", \\\\\\\"program\\\\\\\"=>\\\\\\\"rrdcached\\\\\\\", \\\\\\\"pid\\\\\\\"=>\\\\\\\"27330\\\\\\
\", \\\\\\\"severity\\\\\\\"=>5, \\\\\\\"facility\\\\\\\"=>3, \\\\\\\"facility_label\\\\\\\"=>\
\\\\\\"system\\\\\\\", \\\\\\\"severity_label\\\\\\\"=>\\\\\\\"Notice\\\\\\\"}, \\\\\\\"type\\\
\\\\"], \\\\\\\"[program]\\\\\\\"=>[{\\\\\\\"message\\\\\\\"=>\\\\\\\"queue_thread_main: rrd_up
date_r (
Re: Problems after upgrade to 1.4
Posted: Fri Jan 08, 2016 11:18 am
by jolson
I sent a copy of the log to support hoping it would get to you.
I did not receive this - did you send it to
[email protected]? If not, please do so - I'd like to get a remote going as soon as possible to get you back online.
Jesse
Re: Problems after upgrade to 1.4
Posted: Fri Jan 08, 2016 11:21 am
by gregwhite
Yes I did send it to
[email protected]. Should I send it again?
Re: Problems after upgrade to 1.4
Posted: Fri Jan 08, 2016 11:34 am
by jolson
I did not see your ticket come through - if you could send it again that would be great. Thank you!