Logstash stopped working - Too many Open files issue

[teirekos]

Logstash stopped working normally. The following error appears in the logstash log.

{:timestamp=>"2015-02-12T10:04:29.478000+0200", :message=>"Failed to flush outgoing items", :outgoing_count=>1, :exception=>#<SocketError: initialize: java.net.SocketException: Too many open files>, :backtrace=>["org/jruby/ext/socket/RubySocket.java:190:in `initialize'", "org/jruby/RubyIO.java:852:in `new'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb:146:in `connect'", "org/jruby/RubyArray.javain `each'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb:139:in `connect'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:406:in `connect'", "org/jruby/RubyProc.java:271:in `call'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/pool.rb:48:in `fetch'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:403:in `connect'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:319:in `execute'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:217:in `post!'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:106:in `bulk_ftw'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:80:in `bulk'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch.rb:315:in `flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.javain `each'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:216:in `buffer_flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:193:in `buffer_flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:159:in `buffer_receive'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch.rb:311:in `receive'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/base.rb:86:in `handle'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/base.rb:78:in `worker_setup'"], :level=>:warn}

[scottwilkerson]

Follow instruction here for each of your servers:
http://support.nagios.com/forum/viewtop ... 83#p122363

and then restart logstash (on each server)

Code: Select all

service logstash restart

Finally, this is a known bug that will be fixed in the next release after final testing (next week at the latest)

[teirekos]

thanx a lot.
When I restarted the service I got the following:

[root@NagiosLogServer ftw]# service logstash restart
Failed to set capabilities on file `/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/bin/java' (No such file or directory)
usage: setcap [-q] [-v] (-r|-|<caps>) <filename> [ ... (-r|-|<capsN>) <filenameN> ]

Note <filename> must be a regular (non-symlink) file.
Restarting Logstash Daemon: [ OK ]
[ OK ]
Any ideas?

[teirekos]

I also have this now:

{:timestamp=>"2015-02-12T16:53:52.370000+0200", :message=>"syslog udp listener died", :address=>"0.0.0.0:5544", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:116:in `udp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:76:in `run'"], :level=>:warn}
{:timestamp=>"2015-02-12T16:53:57.371000+0200", :message=>"syslog tcp listener died", :address=>"0.0.0.0:5544", :exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:118:in `initialize'", "org/jruby/RubyIO.java:852:in `new'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:135:in `tcp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:90:in `run'"], :level=>:warn}

[scottwilkerson]

I'm guessing you had followed the Listening on privileged ports doc?
http://assets.nagios.com/downloads/nagi ... Server.pdf


Can you show the following

Code: Select all

cat /etc/sysconfig/logstash|grep setcap

If you get a line like this, it is possible that the version of java updated

Code: Select all

setcap 'cap_net_bind_service=+ep' /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/bin/java

Lets delete that line from /etc/sysconfig/logstash, and run the following

Code: Select all

echo -e "\nsetcap 'cap_net_bind_service=+ep' \$(readlink -f \$(which java))" >> /etc/sysconfig/logstash

This will continuously update if the java version updates. I'm also going to modify the doc so others will not have the same issue.

[scottwilkerson]

After making the changes, you should restart logstash again

[teirekos]

Yes indeed, I am strugling to sent logs from AIX machines via syslog 514 ...

Yes logstash restart is clear now.

You might also be interesting in the following error:


{:timestamp=>"2015-02-12T17:06:57.279000+0200", :message=>"syslog tcp listener died", :address=>"0.0.0.0:5544", :exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:118:in `initialize'", "org/jruby/RubyIO.java:852:in `new'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:135:in `tcp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:90:in `run'"], :level=>:warn}
{:timestamp=>"2015-02-12T17:07:02.281000+0200", :message=>"syslog tcp listener died", :address=>"0.0.0.0:5544", :exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:118:in `initialize'", "org/jruby/RubyIO.java:852:in `new'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:135:in `tcp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:90:in `run'"], :level=>:warn}

[teirekos]

For some strange reason the Input regarding syslog on port 5544 was duplicated! :-0
I've deleted one of them and now seems ok.
Thanx for your help. Pls close the thread.