Nagios Support Forum

Hello,

my nagioslogserver installation stopped ingesting logs on the 31/12/2017 @ around mignight. I had to reboot for all ingest functions to start back today.

Is that a known behaviour ?

REgards,
Saleem
Paris

This would be the first instance we've heard of related to the new year and It could just be coincidence. I would check the logs in /var/log/elasticsearch/ and /var/log/logstash/ for errors and warnings around the time the problem was noticed. The problem you described could indicate a crash of either the logstash or elasticsearch process.

We had the same problem, at midnight new year almost all logging stopped. After restarting all logstash and elasticsearch services (NLS 1.4.4) on all cluster nodes everything was back to normal. So that makes 2....

Greetings..Hans Blom

@CBoekhuis,

That would make 2 cases.

Did you happen to see any notable logs in /var/log/elasticsearch/ and /var/log/logstash/.

Hi Kyang,

no nothing in the elasticssearch/logstash logfiles. That's also the reason I didn't investigate it any further.

For posterity, this appears to be a "gotcha" between logstash-output-elasticsearch and Joda:
https://github.com/logstash-plugins/log ... issues/541
https://github.com/logstash-plugins/log ... issues/354

I think the solution is to change the default index template used by Nagios Log Server for its ElasticSearch output, but I don't have an instance to check against currently.

Thanks for the help @mcapra! It certainly fits the part.

Strange enough, but since a restart of logstash and elasticsearch or rebooting Nagios Log Server got everything back to normal. I'm not sure where to exactly classify this.

If we see this as a strong enough issue, I'm sure someone will look into it.