Post 2.16 - Logstash stops ingesting logs

[jaimie.livingston]

Post the 2.16 update, logstash appears to stop ingesting logs from any source at seeming random times. This is occurring on 3 different 3-node clusters in 3 different datacenters that have been running very well since the initial installations in Nov of 2019.

Rebooting all of the nodes in the cluster gets log ingestion going again.

The only thing I can find in the logs that corresponds with this behaviour is the log snippet (copied below from the messages log, but could come from the logstash log as well) from one of the servers. This seems to match a previous set of issues that were resolved in 2015, but that's the only thing I could find that seems out of the norm.

Is this a known issue?
Are there additional troubleshooting steps I can pursue?

Code: Select all

messages-20200712:Jul  7 12:34:38 atlenglog01 logstash: Errno::EBADF: Bad file descriptor - Bad file descriptor
messages-20200712:Jul  7 12:34:38 atlenglog01 logstash: each at org/jruby/RubyIO.java:3565
messages-20200712:Jul  7 12:34:38 atlenglog01 logstash: tcp_receiver at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:173
messages-20200712:Jul  7 12:34:38 atlenglog01 logstash: tcp_listener at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:159
messages-20200712:Jul  8 10:47:32 atlenglog01 logstash: Starting Logstash Daemon: [  OK  ]
messages-20200712:Jul  9 09:40:21 atlenglog01 logstash: Errno::EBADF: Bad file descriptor - Bad file descriptor
messages-20200712:Jul  9 09:40:21 atlenglog01 logstash: each at org/jruby/RubyIO.java:3565
messages-20200712:Jul  9 09:40:21 atlenglog01 logstash: tcp_receiver at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:173
messages-20200712:Jul  9 09:40:21 atlenglog01 logstash: tcp_listener at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:159

Thanks,

Jaimie Livingston

[ssax]

Please PM me a copy of your profile, you can download it from Admin > System Status by clicking the Download System Profile button​.

[jgsupport]

Good luck with this problem, if you find something let me know. I have raised this many times with some suggestions but no resolution to date. Seems to be some bug.. I have been just restarting logstash everyday for the last 2 years.

[ssax]

Most times it stops processing is because of lack of memory or too many logs and not enough nodes.

@jaimie.livingston, please attach these files too (as well as sending the profile I requested in the last response):
- You'll only have two of them, I don't yet know what distro you're running so just send me whichever ones you have

Code: Select all

/etc/sysconfig/logstash
/etc/sysconfig/elasticsearch
/etc/default/logstash
/etc/default/elasticsearch

@jgsupport, feel free to create another topic/ticket if you would like someone to look at your system.