Issues with LogStash and falling number of received logs

[ejd4389]

Hi all,

Can you please help us try to understand our current predicament with a constant falling number of log entries after every logstash restart.

We currently feed NLS with quite a number of logs, and what we have noticed is that there is always a surge in entries (between 50k and 150k) and then it looks like the system struggles to manage and the entries per 15min fall to only 1500 entries.

After investigations within our team we have noticed that we do have an issue with logstash in NLS. The CPU for logstash increases around 400 to 500% and then the shipping of logs to NLS reduces significantly.
The logstash process seems to be in hung state or hibernating and no logs produced in /var/log/logstash/logstash.log

The issue resolves after we restart logstash and problem comes back after few hours.
Below are the loadavg and cpu of logstash process.

The allocated max memory for the process is 500mb and it seems to be insufficient.

[cdienger]

Increase the allocated memory to 2048m. See https://support.nagios.com/kb/article/n ... g-576.html for the steps.

Let's also increase the number of logstash workers - this can be done by editing /etc/sysconfig/logstash and changing this line:

Code: Select all

LS_OPTS=" -w 4"

Increase it to 16:

Code: Select all

LS_OPTS=" -w 8"

and restart logstash:

Code: Select all

systemctl restart logstash

[ejd4389]

Thank you for the support.

I have made the requested changes, so I will monitor the incoming logs for the next few hours and report back if needed. Appreciate the help!

[cdienger]

Sounds good. Look forward to your results!

[ejd4389]

Hi again,

Unfortunately it did not fix the problem.

After a logstash restart we still see a surge in events, which result in everything slowing down.
Logheap has been set to 2048m as well as adjusting the workers as per your suggestion.





That said, we are sending alot of events in these logs spread over 12 hosts. In the last 24hours you can see how many have been received. Is it possible that Nagios struggles to handle this many?



Thanks in advance for the help!

[cdienger]

Please provide a profile from the system. It can be gathered under Admin > System > System Status > Download System Profile or from the command line with:

Code: Select all

/usr/local/nagioslogserver/scripts/profile.sh

This will create /tmp/system-profile.tar.gz. Please send me this through private message.

[ssax]

Locking thread, ticket received, we will continue support through the ticket.

Thank you!