Recent log entires into LOG server are hours old

[dlukinski]

Hello LOG support

We are having issue as follows:

LOG server configured in UTC
Most servers it collects logs from are configured in UTC + 2

How as we look for recent messages (just arrived to the log server), they are few hours old. 6 hours in the recent investigation.

So logs on server keep updating, but at the same time LOG receives their much earlier entries.

How we could investigate and resolve this?

Thank you
---------------------------------------------------

Wonder if logs "from the future" is an issue - https://support.nagios.com/forum/viewto ... 1467745040 ?

But in this case how to deal with multiple timezone logging and why we were never advised during installation and later questions about having LOG server in UTC as common denominator, while the servers are not there?

[mcapra]

Can you share a screenshot of the events that are off by 6 hours as well as the plain-text event itself?

Can I also see the full output of the following command:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/*.conf

[dlukinski]

Can you share a screenshot of the events that are off by 6 hours as well as the plain-text event itself?

Can I also see the full output of the following command:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/*.conf

Not anymore: no more events shown after 1.4.2 upgrade (just attempted)
- created ticket and emailed to XI, asking to fwd to LOG
Once fixed, we are to get back to this one:

Any messages at least from the groups of servers, located in UTC +2, shown in 15 min dashboard are actually 4-6 hours old checking the server logs directly. Timestamps would be correct, but server get them after significant dealy, while showing in the past 15 min (!). Server is set for UTC time.

[mcapra]

Ticket received, will pick it up

[dlukinski]

Ticket received, will pick it up

Here you (attached)

This is a 15 min view (Where events shown are few hour old)
- went over this one multiple times with specific Application server (I have no access to)

[rkennedy]

Can you show us the full page, and the full messages expanded? It looks like your device is sending every x interval.

[dlukinski]

Can you show us the full page, and the full messages expanded? It looks like your device is sending every x interval.

Here expanded

1 hour
Timescale in my local time (US Eastern)

Timestamps in UTC

today's checks should these logged by the server itself 4 hours prior or average (servers are UTC + 2 / currently 3 hour difference)

Saw similar posts about LOG server in the past.
Unsure if programming issue where rsyslog sends messages when LOG server reaches same time as server containing log (with some delay of course)
or just have to dig into rsyslog scheduling or else.

I'll try to get more local log files (already requested) to match with LOG server timing

[mcapra]

There may be some inconsistencies with how the timezone is configured on your NLS machine. Please run the following script:

Code: Select all

/usr/local/nagioslogserver/scripts/change_timezone.sh -z America/Chicago

Replacing America/Chicago with your preferred locale. If that doesn't resolve things, please share the contents of your rsyslog configs on the machine sending the logs. You can gather them like so:

Code: Select all

cat /etc/rsyslog.d/*.conf
cat /etc/rsyslog.conf

[dlukinski]

There may be some inconsistencies with how the timezone is configured on your NLS machine. Please run the following script:

Code: Select all

/usr/local/nagioslogserver/scripts/change_timezone.sh -z America/Chicago

Replacing America/Chicago with your preferred locale. If that doesn't resolve things, please share the contents of your rsyslog configs on the machine sending the logs. You can gather them like so:

Code: Select all

cat /etc/rsyslog.d/*.conf
cat /etc/rsyslog.conf

Changed timezone to UTC, still waiting for rsyslog logs.

Got attachmets as follows:

1. Dashboard-BIP3384E shows event timestamp at 8:43 AM

2. log-entry-time from user.log is also 8:43 AM

3. Event over time is 3 hours off (I set my PC time to be the same as the server event is collected from) / when I set my PC to UTC (like the LOG server), time would match, but the event will arrive to log (from the server in question) after hours... / done in second Screenshot

[rkennedy]

Closing this up as a ticket has been received, we'll continue there.