New Year mismatch

[patalenszki.zoltan]

Dear Support,

After the new year both of our nagios log servers started to send their own syslog (messages) to the nagios log repository for the previous year. (same date and time, with year 2016).
I've restarted the nodes yesterday and problem disappeared.

support.JPG

After that, the nightly maintenance proces got confused. It used 30Gb disk space instead of the normal 4-5Gb.

Could you please help me, what could cause this strange behaviour?

Thanks in advance,
Zoltán Patalenszki

[mcapra]

Have you tried restarting the rsyslog daemon on the remote machines shipping the logs? I saw this on a few of my test machines and rsyslog wound up being the culprit.

[patalenszki.zoltan]

No, I reboot the servers.
We have ~50 servers with RHEL and we did not experience the same problem on them despite of that timestamp format is the same: RSYSLOG_TraditionalFileFormat.
Only on the nagios nodes.

[mcapra]

I would stop the NLS machines' rsyslog processes, delete the incorrect Jan 2016 indices via the GUI, then restart the rsyslog process. See if that kicks it into the correct year.

[patalenszki.zoltan]

You may misunderstood me.
I solved the problem with rebooting servers and because of that "only" the nagios log servers own syslogs were impacted, wrong timestamps are not so critical issue.
I would like to know the reasons in order to avoid further occurences. We have some business critical servers and I am afraid of that next time will happen on one of them.

Thanks for your help!
Zoltan

[mcapra]

Ah, I apologize for the misunderstanding.

I wasn't able to pin it down on my testing machines unfortunately. It seemed to be a mismatch between how rsyslog and logstash were calculating the year. Each was assuming the other had taken care of it, when in reality neither of them did. This should be fixed in a future version of Logstash (and therefore a future version of NLS).