Problems after upgrade to 1.4

[gregwhite]

After upgrading to NLS 1.4 the cluster status is red and the current index is 0. I closed the index and the cluster status turned yellow and it said it was receiving data from all the hosts. However, no data appears to be coming in. Also, Logstash would periodically crash.

[jolson]

Are you certain that Logstash is running on your systems (Run these commands on all nodes)?

Code: Select all

service logstash status
service logstash restart

I would also like the following information from you (Run these commands on a single node):

Code: Select all

curl 'localhost:9200/_cat/nodes?v'
curl -XGET 'localhost:9200/_cat/recovery?v'
curl 'localhost:9200/_cluster/health?level=indices&pretty'

[gregwhite]

output.txt

Here is the output from the 3 CURL commands.

I did the service status for logstash and it was running but it keeps stopping.

[jolson]

Is there anything notable in the Logstash log?

Code: Select all

tail -n50 /var/log/logstash/logstash.log

You might try bumping the Logstash HEAP up to 1024m and giving it another try:

Code: Select all

vi /etc/sysconfig/logstash

LS_HEAP_SIZE="1024m"

Please note that you must remove the comment (#) in front of the LS_HEAP_SIZE setting.

After upping the HEAP to 1024, try restarting Logstash again:

Code: Select all

service logstash restart

I'm also interested in the memory consumption of your system as a whole:

Code: Select all

free -m

[gregwhite]

ls_heap_size was already increased to 1024 last month.

Free Memory = [root@lkenmycroftp01 logstash]# free -m
total used free shared buffers cached
Mem: 32071 31882 189 0 61 12709
-/+ buffers/cache: 19111 12960
Swap: 16103 72 16031

[jolson]

Anything in the logstash log, or did it turn up blank?

[gregwhite]

I sent a copy of the log to support hoping it would get to you. The only thing that stood out to me was:


ar/perfdata/APO_F3_2960-S13/If_FastEthernet0_31.rrd) failed with status -1. (/usr/local/nagios/
var/perfdata/APO_F3_2960-S13/If_FastEthernet0_31.rrd: found extra data on update argument: 0:0:
0:0)\\\\\\\\n\\\\\\\", \\\\\\\"@version\\\\\\\"=>\\\\\\\"1\\\\\\\", \\\\\\\"@timestamp\\\\\\\"=
>\\\\\\\"2016-01-06T06:21:33.000Z\\\\\\\", \\\\\\\"type\\\\\\\"=>\\\\\\\"syslog\\\\\\\", \\\\\\
\"host\\\\\\\"=>\\\\\\\"172.22.3.18\\\\\\\", \\\\\\\"priority\\\\\\\"=>29, \\\\\\\"timestamp\\\
\\\\"=>\\\\\\\"Jan 6 01:21:33\\\\\\\", \\\\\\\"logsource\\\\\\\"=>\\\\\\\"lkennagiost02\\\\\\\
", \\\\\\\"program\\\\\\\"=>\\\\\\\"rrdcached\\\\\\\", \\\\\\\"pid\\\\\\\"=>\\\\\\\"27330\\\\\\
\", \\\\\\\"severity\\\\\\\"=>5, \\\\\\\"facility\\\\\\\"=>3, \\\\\\\"facility_label\\\\\\\"=>\
\\\\\\"system\\\\\\\", \\\\\\\"severity_label\\\\\\\"=>\\\\\\\"Notice\\\\\\\"}, \\\\\\\"type\\\
\\\\"], \\\\\\\"[program]\\\\\\\"=>[{\\\\\\\"message\\\\\\\"=>\\\\\\\"queue_thread_main: rrd_up
date_r (

[jolson]

I sent a copy of the log to support hoping it would get to you.

I did not receive this - did you send it to [email protected]? If not, please do so - I'd like to get a remote going as soon as possible to get you back online.

Jesse

[gregwhite]

Yes I did send it to [email protected]. Should I send it again?

[jolson]

I did not see your ticket come through - if you could send it again that would be great. Thank you!