Logstash Daemon dead but pid file exists

[benhank]

We upgraded to 2.0 and now the logstash collector keeps bombing out.
so I ran the following

Code: Select all

[root@lkenmycroftp01 nagioslogserver]# service logstash status
Logstash Daemon dead but pid file exists
[root@lkenmycroftp01 nagioslogserver]# service logstash restart
Restarting Logstash Daemon:                                [FAILED]
                                                           [  OK  ]
[root@lkenmycroftp01 nagioslogserver]# service logstash status
Logstash Daemon (pid  20500) is running...
[root@lkenmycroftp01 nagioslogserver]# ps aux | grep java
nagios   10949 40.5 63.2 298613896 20765960 ?  SLl  11:13  18:22 /usr/bin/java -Xms16035m -Xmx16035m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Des.cluster.name=c9dc126e-346d-4bfa-a30e-14b849c50ab5 -Des.node.name=4832fc3b-a2d4-429c-80f1-60f23afa08c3 -Des.discovery.zen.ping.unicast.hosts=172.22.3.187 -Des.path.repo=/ -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/local/nagioslogserver/elasticsearch -cp :/usr/local/nagioslogserver/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/local/nagioslogserver/elasticsearch/lib/*:/usr/local/nagioslogserver/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/local/nagioslogserver/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/usr/local/nagioslogserver/elasticsearch/data -Des.default.path.work=/usr/local/nagioslogserver/tmp/elasticsearch -Des.default.path.conf=/usr/local/nagioslogserver/elasticsearch/config org.elasticsearch.bootstrap.Elasticsearch
root     20500  0.0  0.1 2925136 52808 pts/0   SNl  11:58   0:00 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx1024m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
root     20559  0.0  0.0 103316   852 pts/0    S+   11:58   0:00 grep java
[root@lkenmycroftp01 nagioslogserver]# ls -al /var/run/logstash
total 12
drwxr-xr-x.  2 nagios nagios 4096 Nov 14 11:58 .
drwxr-xr-x. 37 root   root   4096 Nov 14 10:27 ..
-rw-r--r--   1 nagios nagios    0 Nov 14 11:27 logstash
-rw-r--r--   1 root   nagios    6 Nov 14 11:58 logstash.pid

and also:

Code: Select all

tail -n 100 /var/log/logstash/logstash.log

and see attachment.

logstashlog.txt

[cdienger]

There was an issue with the upgrade script that prevented the logstash config files from being updated properly. This will be fixed asap on our end so a new download later today should fix it. In the meantime though you can run:

Code: Select all

/usr/bin/php /var/www/html/nagioslogserver/www/index.php configure/write_configs_for_node

[benhank]

does the command you said to run address this issue ?:

Code: Select all

{:timestamp=>"2017-11-14T11:59:05.156000-0500", :message=>"Pipeline aborted due to error", :exception=>"LogStash::ConfigurationError", :error=>"The setting `host` in plugin `elasticsearch` is obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask.", :backtrace=>["/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/config/mixin.rb:88:in `config_init'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/config/mixin.rb:72:in `config_init'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/outputs/base.rb:79:in `initialize'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:74:in `register'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:181:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:181:in `start_workers'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:136:in `run'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error}
{:timestamp=>"2017-11-14T11:59:08.165000-0500", :message=>"stopping pipeline", :id=>"main"}

[cdienger]

Correct. And it is covered in https://support.nagios.com/kb/article/n ... o-2-0.html as well.

[benhank]

Thanks you can lock this up