Page 1 of 6
logstash daemon stopping
Posted: Tue Jan 06, 2015 1:07 pm
by benhank
Happy new year guys!
this most likely an easy fix
my logstash deamon stops running periodically, the error I get (from nagios) is:
Code: Select all
the logstash daemon dead, but pid file exists.
When this happens, the number of objects sending logs to the logserver drops to 1
after restarting the daemon via the logserver's admin page, everything goes back to normal.
how can I prevent this fellas?
thanks in advance
Re: logstash daemon stopping
Posted: Tue Jan 06, 2015 1:11 pm
by tmcdonald
Could you post some logs from when this is happening?
Code: Select all
tail -100 /var/log/logstash/logstash.log
As a bonus, try saying this 5 times fast: slash var slash log slash logstash slash logstash dot log
Re: logstash daemon stopping
Posted: Tue Jan 06, 2015 4:20 pm
by benhank
But.. we don't have any logs on the logserver.
Re: logstash daemon stopping
Posted: Tue Jan 06, 2015 4:25 pm
by sreinhardt
/var/log/logstash is standard for all log stash installs. Log server does not actually take in this log in any way unless its provided to the local syslog facility. If this is pretty much a stock log server, they definitely should be available.
Re: logstash daemon stopping
Posted: Tue Jan 06, 2015 5:21 pm
by benhank
dude i know lol i was kidding. lol. hold off on this till tomorrow, just got a project dropped on me
Re: logstash daemon stopping
Posted: Tue Jan 06, 2015 5:48 pm
by cmerchant
Could you post some logs from when this is happening?
Code: Select all
tail -100 /var/log/logstash/logstash.log
Just send us something tomorrow, thanks.
Re: logstash daemon stopping
Posted: Thu Jan 08, 2015 12:27 pm
by benhank
here as a creepy aside, even tho my NLS is showing that it is receiving data from 1 source, the lil green arrows are both green and nagios is reporting that the logstash daemon is running.
Re: logstash daemon stopping
Posted: Thu Jan 08, 2015 2:06 pm
by sreinhardt
Looks like you're hitting this guy:
https://github.com/elasticsearch/logstash/issues/1604, you should be able to follow the patch here:
https://github.com/jordansissel/ruby-ft ... 59eccc18a3. Try replacing theses files with the ones attached to this post:
/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb
/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/pool.rb
Re: logstash daemon stopping
Posted: Thu Jan 08, 2015 2:20 pm
by eloyd
tmcdonald wrote:As a bonus, try saying this 5 times fast: slash var slash log slash logstash slash logstash dot log
slash var slash log slash logstash slash logstash dot log
slash var slash log stash logstash slash logslash dot log
slash var stash logo stash logslash stash logoslash dot log
stash var stash logo stash logslasht stash logoslash dot slog
stash vars tash log ostash logslashts tashlo go shash dots logs
Re: logstash daemon stopping
Posted: Thu Jan 08, 2015 3:18 pm
by sreinhardt
You, ... thats enough out of you! Ben, let us know if that works out for you or not. The change is needed because a part of logstash does not close its open sockets when done with them, and since linux treats everything as a file.. you get an too many open files error. This just makes it play nice with everyone else and close it's sockets.