Can't start logstash after 2.1 Upgrade

[stecino]

tcp_receiver at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:175
tcp_listener at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:161

I also noticed, if I login to the cluster nodes that haven't been upgraded I can see index details from there, but not from the nodes that have been upgraded

[jolson]

I'd like to see a screenshot of the problem, in addition to the following output:

Code: Select all

curl 'localhost:9200/_cluster/health?level=indices&pretty'

[stecino]

I'd like to see a screenshot of the problem, in addition to the following output:

Code: Select all

curl 'localhost:9200/_cluster/health?level=indices&pretty'

Here is a snippet, also attached the screenshot

"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 166,
"active_shards" : 246,
"relocating_shards" : 0,
"initializing_shards" : 4,
"unassigned_shards" : 82,
"number_of_pending_tasks" : -1,
"number_of_in_flight_fetch" : 0,
"indices" : {
"logstash-2015.07.30" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 6,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 4
},
"logstash-2015.07.31" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 8,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2
},
"nagioslogserver" : {
"status" : "green",
"number_of_shards" : 1,
"number_of_replicas" : 1,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.07.09" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 8,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2
},
"nagioslogserver_log" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 8,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2
},
"logstash-2015.07.15" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 8,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2
},
"logstash-2015.07.14" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 7,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 3
},
"logstash-2015.07.13" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 7,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 3
},
"logstash-2015.07.12" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 7,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 3
},
"logstash-2015.07.19" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 8,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2
},
"logstash-2015.07.18" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 8,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2
},

[jolson]

Am I correct that your cluster now consists of different versions of Nagios Log Server? If so, please get them all upgraded - that could very well resolve the Index display issue that you're experiencing.

[stecino]

Also seeing this on the node that was upgraded:

INFO: [00a07fa5-7288-4d96-880c-474285dad017] loaded [], sites []
Aug 05, 2015 7:35:58 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [00a07fa5-7288-4d96-880c-474285dad017] loaded [], sites []
Aug 05, 2015 7:35:58 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [00a07fa5-7288-4d96-880c-474285dad017] loaded [], sites []
Aug 05, 2015 7:35:58 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [00a07fa5-7288-4d96-880c-474285dad017] loaded [], sites []
Aug 05, 2015 7:35:58 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [00a07fa5-7288-4d96-880c-474285dad017] loaded [], sites []

[stecino]

Am I correct that your cluster now consists of different versions of Nagios Log Server? If so, please get them all upgraded - that could very well resolve the Index display issue that you're experiencing.

Correct. What are the chances if it will make it even worse? Although I can see things are being logged

[jolson]

Also seeing this on the node that was upgraded:

INFO: [00a07fa5-7288-4d96-880c-474285dad017] loaded [], sites []
Aug 05, 2015 7:35:58 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [00a07fa5-7288-4d96-880c-474285dad017] loaded [], sites []
Aug 05, 2015 7:35:58 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [00a07fa5-7288-4d96-880c-474285dad017] loaded [], sites []
Aug 05, 2015 7:35:58 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [00a07fa5-7288-4d96-880c-474285dad017] loaded [], sites []
Aug 05, 2015 7:35:58 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [00a07fa5-7288-4d96-880c-474285dad017] loaded [], sites []

That is normal behavior - Logstash will output startup information to stdout when it's just spinning up.

Correct. What are the chances if it will make it even worse? Although I can see things are being logged

That is hard for me to say - is it possible that you could take snapshots/backups of your Nagios Log Server instances prior to upgrading? For the cluster to function optimally all of the versions need to be similar - it is my understanding that display bugs like this could crop up due to the version differences.

[stecino]

I have upgraded all the nodes

What does this mean?

/etc/init.d/logstash stop
Stopping Logstash Daemon: Errno::EBADF: Bad file descriptor - Bad file descriptor
each at org/jruby/RubyIO.java:3542
tcp_receiver at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:175
tcp_listener at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:161

[stecino]

I updated all the nodes, everything is working fine now. I just wanted to check and see what it was reporting version wise. So ran this: curl -XGET localhost:9200/_nodes/jvm?pretty

getting this:

"version" : "1.6.0",
"build" : "cdd3ac4",

The NLS page reports 2015R1.3

[jolson]

"version" : "1.6.0",

I wouldn't be concerned with the version of Java in use here - it is the same on my test cluster.

The NLS page reports 2015R1.3

Try running the 'update check' from 'Administration -> Command Subsystem'. Does the version number correct itself?