logstash daemon stopping

[benhank]

Happy new year guys!
this most likely an easy fix

my logstash deamon stops running periodically, the error I get (from nagios) is:

Code: Select all

the logstash daemon dead, but pid file exists.

When this happens, the number of objects sending logs to the logserver drops to 1
after restarting the daemon via the logserver's admin page, everything goes back to normal.
how can I prevent this fellas?
thanks in advance

[tmcdonald]

Could you post some logs from when this is happening?

Code: Select all

tail -100 /var/log/logstash/logstash.log

As a bonus, try saying this 5 times fast: slash var slash log slash logstash slash logstash dot log

[benhank]

But.. we don't have any logs on the logserver.

[sreinhardt]

/var/log/logstash is standard for all log stash installs. Log server does not actually take in this log in any way unless its provided to the local syslog facility. If this is pretty much a stock log server, they definitely should be available.

[benhank]

dude i know lol i was kidding. lol. hold off on this till tomorrow, just got a project dropped on me

[cmerchant]

Could you post some logs from when this is happening?

Code: Select all
tail -100 /var/log/logstash/logstash.log

Just send us something tomorrow, thanks.

[benhank]

here as a creepy aside, even tho my NLS is showing that it is receiving data from 1 source, the lil green arrows are both green and nagios is reporting that the logstash daemon is running.

[sreinhardt]

Looks like you're hitting this guy: https://github.com/elasticsearch/logstash/issues/1604, you should be able to follow the patch here: https://github.com/jordansissel/ruby-ft ... 59eccc18a3. Try replacing theses files with the ones attached to this post:

/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb
/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/pool.rb

[eloyd]

As a bonus, try saying this 5 times fast: slash var slash log slash logstash slash logstash dot log

slash var slash log slash logstash slash logstash dot log
slash var slash log stash logstash slash logslash dot log
slash var stash logo stash logslash stash logoslash dot log
stash var stash logo stash logslasht stash logoslash dot slog
stash vars tash log ostash logslashts tashlo go shash dots logs

[sreinhardt]

You, ... thats enough out of you! Ben, let us know if that works out for you or not. The change is needed because a part of logstash does not close its open sockets when done with them, and since linux treats everything as a file.. you get an too many open files error. This just makes it play nice with everyone else and close it's sockets.