LogStash Crash

[jhahn]

Hi all,
I have been having an issue with Logstash on my NagiosLog Instance. every so often the service shows running but the sub status of the port listener shows exited. I have composed a check tool with NagiosXI, as the standard NCPA service check shows that the service is still running but does not indicate that the service thread for the ports has exited. I use this command to check it

systemctl list-units --type=service --all |{ head -1; grep "logstash.service"; }

and most times it shows running in SUB but when it fails it shows exited.

From the time it shows exited I receive no logs in the log server at all.

I have expanded the java vm settings for LogStash to
echo $LS_JAVA_OPTS
-Xmx2g -Xmx1g

I have checked the logs and I am not seeing anything that is unexpected in the Logstash logs.

Any Thoughts?

Thanks,
Joe Hahn

[jmichaelson]

The first question i have is what version of Nagios Log Server are you running. The infrastructure was massively upgraded in 2024R2.

The second question I have is how much RAM does your system have? This sounds like a situation where its possible the process is being killed by the system's out of memory killer. Is there any evidence of that in the system logs or the Logstash logs (the location for those will dependd on which version of NLS you're running, either /var/log/logstash for 2024R1 or earlier or /usr/local/nagioslogserver/logstash/logs for 2024R2 or newer)

[jhahn]

Thank you for answering me.

the version I am running currently is 2024R1.3.2 we have not updated yet as we are in an air gaped network.I have to manually pull it and request a maintenance window, that can take a bit.

Current ram is set to 32 GB with 16 vCPU.

Again thank you for the reply.

Joe H

[jhahn]

Also I adjusted the LogStash Java VM options today to be -Xms3g -Xmx8g I initially had the wrong setting. only supplying the maximum ram twice. I took the opportunity to go to the maximum allowed according to LogStash's Documentation.

[jmichaelson]

Was that successful?

If not, have a look in /var/log/logstash to see if that shows why it died, and if that's not a help, look for java in the syslog entries within your log server.

[jmichaelson]

That's actually a good point. The processes can fall victim to the system out of memory killer. There should be evidence of that in the system logs if it happened.