Logs Stop Being Received - Logstash Error

[NCATmax]

Hello,

We are having an issue where logs stop being received in NLS. I have 3 machines sending syslog logs to a syslog input in NLS. This works fine.

The issue is that eventually, these logs will stop appearing in NLS. No more logs are received, and NLS will list these hosts in the "Not Sending" section on the Unique Hosts page.

Additionally, I have noticed that logs stop being received every day at exactly 8pm.

I have also been able to fix the issue. I found that after the issues occurs, if the current index is deleted, all of the logs will immediately start being collected again.

Looking at the Logstash log, I see the the same error, repeated for every event from these machines. I have attached a sample error line from the log. I believe that the significant part is:

Code: Select all

response=>{"create"=>{"_index"=>"logstash-2019.09.18", "_type"=>"syslog", "_id"=>"AW1E4WbF-7bFpjTmq8fG", "status"=>400, "error"=>"MapperParsingException[failed to parse [timestamp]]; nested: MapperParsingException[failed to parse date field [Sep 18 10:58:05], tried both date format [dateOptionalTime], and timestamp number with locale []]; nested: IllegalArgumentException[Invalid format: \"Sep 18 10:58:05\"]; "}}

I cannot figure out why the format of the date is a problem. It works some of the time, and seems to be a common syslog date format.

While searching for solutions, I found a similar issue in the Nagios Support Forum. That post indicates that the index has the wrong date format? If this is the case, how should that be addressed?


Would you have any advice for how to solve this issue? I would be happy to provide any additional information.

Thank you for your assistance.

[mbellerue]

Can you single out the logs that came just before 8pm, through to 8pm? There may be a log that's coming in about the time that Log Server is migrating to a new index. Do you know if Log Server's system time is in UTC?

[NCATmax]

The day before I posted, I had reinstalled NLS from scratch, as well as updated from 2.0.8 to 2.1.0. Since then, deleting the current index no longer makes the logs start showing up again. I'm not sure what changed, that consistently worked for several weeks.

Because of that re-installation, I don't have the logs from 8pm when the logs stopped appearing. I was hoping to replicate the issue after reinstalling, but that behavior is no longer happening.

The time zones on all involved servers is EDT. I did notice that 8pm is midnight UTC, which may not be coincidental.

In any case, Logstash is still having an issue with the log entries. All the events are being recorded in the Logstash log with the "MapperParsingException" error shown above.

Thank you for your assistance.

[mbellerue]

Could you PM your system profile to me? It can be downloaded from Admin -> System Status -> Download System Profile.

[NCATmax]

I have attempted to send you a PM containing the system profile, but on the private message page, after I select the file to upload, I click "Add file" and the page loads for about three seconds, and then the page seems to completely reset itself.

Is there a size limitation on attachments? The file NLS gave me is 153 MB in size.

[mbellerue]

Ah, yes I bet that is too large for our system. This is usually due to the logs in the Logstash and/or Elasticsearch directories found in it. Could you please open the profile, extract these directories/files and send them separately.

[NCATmax]

I would like to follow up on this issue. I attempted to send you a message containing my system profile back on Sept 30. Did you receive that message? I would be glad to send it again if need be.

Thank you!

[benjaminsmith]

Hello,

Michael is out of the office today, and I don't see the profile in your account. Would you be able to send it once more? You can PM the profile to my account ( just click the PM icon under my name). Thanks.