logstash seems to close connections to nxlog

[_asp_]

Hi there,

is there a limit on how many logshippers (nxlog) the logstash instance of nagioslogserver can handle?

We have a single instance which is used by multiple nxlog services to process and shipper the logs to elasticsearch.
All instances are sending more or less continuously data to logstash.

Now I have the behavior, that logstash seems to close connections to the clients, as if there is a limit?
Logstash is not really very speaky, I did not find anything interesting in logstash.log.

[_asp_]

more details: problem occurred when I wanted to add more servers (activate nxlog).

If I activate more servers, some other nxlog shippers stop shipping data. No error in nxlog found.
If I stop the new started nxlogs, nothing changes, but if I restart the "broken" nxlog, it starts working again.

I am not sure jet, if it is connection count or throughput related, but i assume that logstash is only able to hold a limited number of connections.

[_asp_]

new details:

I noticed, that the problem is load initiated. The connections stays open, as I can see with netstat.

I also noticed, in top, that the load is significantly increasing, if I shipper the new logfile from an additional machine.
But there is something I do not understand:

I added a new logfile. The logfile is the logfile output of elastic's topbeat. Then it is parsed like a common logfile in logstash.
This one has been added to 4 servers, which runs smooth:

4 logfiles.JPG

But if I enable the same configuration on the fifth machine, the load is increasing significantly, so that logstash / elasticsearch will not be able to process fast enough, so the events are processed with an increasing big delay.

5 logfiles.JPG

We can see the that the througput of processed events is decreased significantly.

througput drop.JPG

Also the processed count of topbeat events is decreased significantly.

The logfile parsing should also not be that expensive. It only uses grok to parse, mutate for converting to values to float and to remove some fields and some small ruby elements which do some simple calculations like converting bytes to GB.

[_asp_]

the throughput decreases and decreases. Less and less events are processed until it is stucked completely

processing goes stucking.JPG

[mcapra]

This is probably related to the following github issues:
https://github.com/elastic/logstash/issues/4815
https://github.com/elastic/logstash/issues/4225

I have also filed an internal bug report for this issue (ID 9305).

See if restarting the logstash service once you notice the events dropping fixes the issue (at least temporarily). If so, you may need to restart the logstash on a regular interval via cron. I realize this is less than ideal, but it seems to be the only low-level solution others have found. My understanding of the problem is that logstash is not properly closing out "dead" connections which bogs down things on the back-end.