Periodically stop receiving logs

[xdatanet]

We migrated our old debian to Ubuntu and reinstalled NLS.

We updated to the last version.

Our two-node cluster works like a charm a part for RAM.
It consumes more ram every day and, after couple of days (4 or 5), it stops to listen (usually the node with more connections).
If I reapply the conf by UI it frees about 5 of 14/15GB and works normally.

Now I am on top of the curve (it works since last monday) and I've tryed to reapply the conf before it stops (usually during the weekend).
It freed immediatly 4GB of RAM and I think I will not have a problem for a couple fo days.

Two question:
1. Why???
2. If I cannot discover the cause can I try to reapply the conf by command line otherwise using UI by cron???

Regards,
Graziano.

[jmichaelson]

Greetings!

How much RAM do you have overall in your systems? I'm wondering if you're not possibly running low on RAM in the system. Check your system logs for messages from the OOM killer:

Code: Select all

journalctl | grep -i oom

If that's happening, you may want to consider disabling the OOM killer:

Code: Select all

systemctl disable --now systemd-oomd

and upgrading the amount of RAM in the system. Elasticsearch can grow very large in its RAM usage depending on the number of indexes it has open and the amount of data being ingested.

[xdatanet]

No, any message from OOM Killer...

I use an Azure VM to run this system and I must limit the RAM to 16GB otherwise I have to jump to 32GB doubling the costs.
My cluster is made of two nodes with the same resources (4core, 16GB and 1TB).
It receives about 1,5GB a day of logs. It retains online about 90 days and offline 1 year.

It works like a charm for 5-6 days since the RAM reachs 12/14GB. Sometimes stops reaching the 12GB, sometimes the 14GB.
Now I doscovered that I can restart logstash and I planned to do it every week via cron. This action frees the RAM like GUI "Apply configuration" and seems to avoid the systems to stop receiving.

Regards,
Graziano.

[jmichaelson]

So it is the logstash process that is using up all the RAM and not ElasticSearch?

That is indeed unusual. As you noted, applying the configuration does indeed restart logstash itself.

This might not be feasible, but could you turn off your logstash filters one at a time and monitor the memory usage? That's really the only configuration you can control for it.