logstash is crashing

[comfone]

Hi All
We are using 2 Nagios Log Server Instances Version 1.4.2 (VM appliance)
The logstash is crashing several times a day.
Attached the logstash.log file.
Does anybody experiance the same problem?

[tmcdonald]

How much memory do each of these servers have? Usually when there are issues in Logserver, upping the RAM is a good first step since it can be memory-intensive.

[comfone]

I gave 10GB RAM each of them.
Attached you see the RAM usage measured by Nagios XI.

Is there a way to monitor and alarm using Nagios XI if logstash and elasticsearch are running on NLS?

[rkennedy]

I would add two checks to your machine using the NRPE wizard. One making sure that logstash is running, and the other for elasticsearch.

[comfone]

Thank you for the feeback.
I'll monitore NLS but I would need your help to troubleshoot why Logstash is crashing.
Attached you will find an extract of the "secure" log as well as "logstash" log.
I need your urgent support as NLS is not usable with logstash crashing several times a day.
Thank you.

[rkennedy]

Could you post your Elasticsearch log as well? It's located at /var/log/elasticsearch/ - the name of it will be yourcluster.log (where yourcluster is the actual cluster id).

Could you also post a screenshot of your Backup & Maintenance page, and the output of curl 127.0.0.1:9200/_cat/indices?v

[comfone]

Attached the requested information.
Thank you in advance for your quick help.

[mcapra]

With 10GB allocated to each machine, you're likely encountering memory issues. Looking at the list of open indices, there's about 211GB worth of indices open currently that is trying to fit into the Java heap. Even with the compression done by elasticsearch on the back-end, this exceeds what your environment has available in terms of resources. With your nodes only having 20GB allocated between them (this being reduced to 10GB total to leave room for maintenance tasks), you're likely exhausting the cluster.

I would suggest increasing the memory available to each of these nodes (up to 64GB max) and seeing if that solves the issue. You could also try reducing the number of open indices you have at any given moment via the "Backup & Maintenance" page by adjusting the "Close indexes older than" value.