Every few days all our systems stop sending logs

[Frouldeste]

Every few days all our systems stop sending logs (or so it appears). But, I can get the logs to start up again once I restart logstash (via "service logstash restart"). I assume I shouldn't need to continuously restart logstash. What are some possible causes and what logs on the OS or application can I look at to try and troubleshoot the issue?

[jmichaelson]

You can check the logstash logs by entering journalctl -xeu logstash in a terminal window.

The logstash logs can be found in /usr/local/nagioslogserver/logstash/logs.

Look for anything relating to an unhandled exception. Feel free to post snippets here (sanitized, if necessary, to remove private data) and we can provide further help.

[dscrimpsher]

Hi, I am seeing this also.
I am running NLS 2024R1.0.1 on a 2-note cluster.

I have 38 unique hosts sending logs. After a couple days the number of unique hosts drops to ZERO. I know there is nothing wring with all those hists be cause when I reboot the Nagios Log Servers (2-node cluster) I get tons of logs from the last couple days suddenly showing up, including entries that should have been displayed in the log server history from those past few days. Then a couple days later same thing all over again. Nothing shows up, no new logs from my hosts. I reboot the log server and see log data gain. (Repeats...)
The NLS GUI show both nodes green, no errors indicates. Both instances are up and green check marks. No indication anything is wrong.

The command journalctl -xeu logstash shows this:
~
~
~
~
-- Logs begin at Thu 2024-05-02 15:30:01 MDT, end at Mon 2024-05-06 07:28:16 MDT. --
May 02 15:30:24 logserver1.csi.edu systemd[1]: Starting LSB: Logstash...
-- Subject: Unit logstash.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/li ... temd-devel
--
-- Unit logstash.service has begun starting up.
May 02 15:30:24 logserver1 runuser[1301]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 02 15:30:24 logserver1 logstash[1207]: Starting Logstash Daemon: [ OK ]
May 02 15:30:24 logserver1 systemd[1]: Started LSB: Logstash.
-- Subject: Unit logstash.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/li ... temd-devel
--
-- Unit logstash.service has finished starting up.
--
-- The start-up result is done.
May 03 21:00:06 logserver1 logstash[1207]: Errno::EBADF: Bad file descriptor - Bad file descriptor
May 03 21:00:06 logserver1u logstash[1207]: each at org/jruby/RubyIO.java:3565
May 03 21:00:06 logserver1 logstash[1207]: tcp_receiver at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:173
May 03 21:00:06 logserver1 logstash[1207]: tcp_listener at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:159
May 03 21:00:06 logserver1 runuser[1301]: pam_unix(runuser:session): session closed for user nagios


After a reboot that command gives me:
~
~
~
-- Logs begin at Mon 2024-05-06 07:39:01 MDT, end at Mon 2024-05-06 07:41:07 MDT. --
May 06 07:39:24 logserver1 systemd[1]: Starting LSB: Logstash...
-- Subject: Unit logstash.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/li ... temd-devel
--
-- Unit logstash.service has begun starting up.
May 06 07:39:24 logserver1 runuser[1289]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 06 07:39:25 logserver1 logstash[1197]: Starting Logstash Daemon: [ OK ]
May 06 07:39:25 logserver1 systemd[1]: Started LSB: Logstash.
-- Subject: Unit logstash.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/li ... temd-devel
--
-- Unit logstash.service has finished starting up.
--
-- The start-up result is done.

[jmichaelson]

Can you get me more of the logs from the logstash log right before that bad file descriptor line?

[kg2857]

In the meantime you may want to set a cron to restart logstash.
In my experience logstash was pretty reliable BTW.

[jmichaelson]

In the meantime you may want to set a cron to restart logstash.
In my experience logstash was pretty reliable BTW.

Yes, it is generally quite reliable. The big question largely across the board with the ELK stack is system RAM.

[dscrimpsher]

I rebooted the server Monday, May 6. This is all that the log shows.

journalctl -xu logstash

-- Logs begin at Mon 2024-05-06 07:39:01 MDT, end at Wed 2024-05-08 08:16:56 MDT. --
May 06 07:39:24 logserver1.csi.edu systemd[1]: Starting LSB: Logstash...
-- Subject: Unit logstash.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/li ... temd-devel
--
-- Unit logstash.service has begun starting up.
May 06 07:39:24 logserver1.csi.edu runuser[1289]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 06 07:39:25 logserver1.csi.edu logstash[1197]: Starting Logstash Daemon: [ OK ]
May 06 07:39:25 logserver1.csi.edu systemd[1]: Started LSB: Logstash.
-- Subject: Unit logstash.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/li ... temd-devel
--
-- Unit logstash.service has finished starting up.
--
-- The start-up result is done.
May 07 13:00:05 logserver1.csi.edu logstash[1197]: Errno::EBADF: Bad file descriptor - Bad file descriptor
May 07 13:00:05 logserver1.csi.edu logstash[1197]: each at org/jruby/RubyIO.java:3565

May 07 13:00:05 logserver1.csi.edu logstash[1197]: tcp_receiver at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:173
May 07 13:00:05 logserver1.csi.edu logstash[1197]: tcp_listener at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:159
May 07 13:00:05 logserver1.csi.edu runuser[1289]: pam_unix(runuser:session): session closed for user nagios

[jmichaelson]

I'm sorry for the confusion I created. The bits that I'm looking for are going to be in /usr/local/nagioslogserver/logstash/logs, most likely the plaintext file.

[dscrimpsher]

I found the logstash logs in the folder /var/log/logstash.
This does not look good. The last few lines of logstash.log show this:

{:timestamp=>"2024-05-08T15:36:33.611000-0600", :message=>"syslog listener died", :protocol=>:tcp, :address=>"0.0.0.0:514", :exception=>#<Errno::EACCES: Permission denied - bind(2)>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:871:in `new'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:152:in `tcp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:101:in `run'"], :level=>:warn}
{:timestamp=>"2024-05-08T15:36:33.673000-0600", :message=>"syslog listener died", :protocol=>:udp, :address=>"0.0.0.0:514", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:135:in `udp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:97:in `run'"], :level=>:warn}
{:timestamp=>"2024-05-08T15:36:38.610000-0600", :message=>"syslog listener died", :protocol=>:tcp, :address=>"0.0.0.0:514", :exception=>#<Errno::EACCES: Permission denied - bind(2)>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:871:in `new'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:152:in `tcp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:101:in `run'"], :level=>:warn}
{:timestamp=>"2024-05-08T15:36:38.674000-0600", :message=>"syslog listener died", :protocol=>:udp, :address=>"0.0.0.0:514", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:135:in `udp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:97:in `run'"], :level=>:warn}


I have checked the both servers and they are showing the same log entries like this.
I wanted to capture a logstash restart and those errors persist. It looks like I definitely have something wring with my setup.


{:timestamp=>"2024-05-08T15:46:29.393000-0600", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2024-05-08T15:46:29.393000-0600", :message=>"stopping pipeline", :id=>"main"}
{:timestamp=>"2024-05-08T15:46:38.939000-0600", :message=>"syslog listener died", :protocol=>:udp, :address=>"0.0.0.0:514", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:135:in `udp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:97:in `run'"], :level=>:warn}
{:timestamp=>"2024-05-08T15:46:38.941000-0600", :message=>"syslog listener died", :protocol=>:tcp, :address=>"0.0.0.0:514", :exception=>#<Errno::EACCES: Permission denied - bind(2)>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:871:in `new'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:152:in `tcp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:101:in `run'"], :level=>:warn}

{:timestamp=>"2024-05-08T15:46:39.354000-0600", :message=>"Pipeline main started"}

{:timestamp=>"2024-05-08T15:46:43.940000-0600", :message=>"syslog listener died", :protocol=>:udp, :address=>"0.0.0.0:514", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:135:in `udp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:97:in `run'"], :level=>:warn}

[dscrimpsher]

I noticed that the OS I am using is no longer on the supported list for NLS. The server OS is CentOS Linux release 7.9.2009. I wonder if I might be better off starting over. I am by no means a expert managing linux systems, but i do pretty well with what i have learned over the years and it also helps when there are good instructions to follow.
Is it possible to install a clean OS on the boot drive and then somehow reattach the /data drive when installing NLS? I would like to retain the historical data I already have in NLS if it is possible.