All Linux server logs stopped

[jcrea]

Out of the blue, all at once, all of my Linux servers log files stopped coming in. Running Nagios Log Server 2024R1.3.2. Under Admin\Unique Hosts, it shows that all of them stopped sending logs. I rebooted the log server. after that I can see a lot of these servers have established a connection to the log server on port 5544 (tcp6 0 0 X.X.X.X:5544 X.X.X.X:34700 ESTABLISHED) but the log counts of these hosts in the list keep dropping. Even local host is in the not sending list. All of my windows servers and firewall logs are still working.

[jcrea]

I found that the logstash was getting time stamp errors:
status"=>400, "error"=>"MapperParsingException[failed to parse [timestamp]]; nested: MapperParsingException[failed to parse date field [Apr 15 09:14:55], tried both date format [dateOptionalTime], and timestamp number with locale []]; nested: IllegalArgumentException[Invalid format: \"Apr 15 09:14:55\"]; "}}, :level=>:warn}
So I added a new input for my linux servers:

Code: Select all

syslog {
    type => 'rsyslog'
    port => 55455
}

Opened firewall ports:

Code: Select all

firewall-cmd --zone=public --add-port=55445/udp
firewall-cmd --zone=public --add-port=55445/udp --permanent
firewall-cmd --zone=public --add-port=55445/tcp
firewall-cmd --zone=public --add-port=55445/tcp --permanent

Applied the config.
Changed my servers to use the new port and the logs started flowing again.
Not sure how to change the local host though.

[jmichaelson]

the localhost rsyslogd configuration for Log Server can be found in /etc/rsyslog.d/nagioslogserver.conf. Change the port from 5544 to your new port in that file and then restart rsyslogd, and you should have logs again.

[jcrea]

Yes, that worked for the local host.
Thanks.
It's still strange how it just stopped working though. Haven't had any issues like this since we started using Nagios Log Server back in 2022. Then for it to just start complaining about the time stamps. Just strange.
Thanks again.

[jmichaelson]

All other things being equal, that is rather unusual.

For what its worth, NLS2024R2's recommended rsyslog configuration adds in the full time zone and microseconds to the log entry, so that the timestamp on logs is unambiguous and absolute. There is a R2.0.1 bugfix release in the works for a few issues found in R2. We don't yet have a release date for it, but it will be coming soon.