@Timestamp Time offset

[polarbear1]

The @timestamp field on my dashboard is offeset 5 hours into the future from the actual time of the alert

This happens on dashboards with data originating on a Linux client using rsyslog and Windows client using nxlog.

dashboard.PNG

Tried this, no dice --
I'm on 2015R2.1 so I am betting the bug this talks about was fixed, but I tried it anyway.

Code: Select all

# cat /etc/sysconfig/clock
ZONE="America/Chicago"
# cat /etc/php.ini | grep date.timezone
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.timezone
date.timezone = America/Chicago

settings.PNG

The "Nagios Log Server Search" dashboard, the timestamp is correct --

NLS_Search.PNG

Thanks.

[jolson]

Nagios Log Server will by default tag any inbound logs with the current localtime of your instance. Check the localtime with the following command:

Code: Select all

date

Is that time accurate? If not, you can update it by setting up ntpd.

I would like you to expand one of the problem logs in question, and click the 'raw' button - what does the log look like at that point?

2015-08-06 10_45_36-Dashboard • Nagios Log Server - Firefox Developer Edition.png

If the end device (Windows/Linux device) is reporting its time in syslog format, Nagios Log Server could pick up that time and use it *instead* of using the localtime of the Nagios Log Server instance - this could result in the discrepancy you're seeing. Please ensure that your Windows/Linux devices are using Central Time as well.

It's also worth noting that when you're viewing the dashboard, Nagios Log Server will adjust the time according the the localtime your *browser* is reporting. For example if your logs were coming in and tagged in central time, they would be stored in the database using central time.

Now, you view your cluster with a browser from a Windows host in GMT - your logs would appear to be tagged as 5 hours in the future due to the difference between your browser time and the time of Nagios Log Server - the end result is to reduce confusion on your end so that you can troubleshoot using your localtime as a reference instead of having to do the timezone translation in your head.

This is a pretty complicated topic, let me know if you need any clarification.

[polarbear1]

All our servers (windows and linux) are set up with ntp to sync to our AD server (which itself syncs to Microsoft's default windows ntp), so they should all be in sync and running date on several linux boxes confirms that the time shown is accurate and in sync.

My workstation is also in Central timezone and the Windows Date and Time settings screen confirms the timezone is set for Central Time (w/ automatic adjustment for daylight savings). To confirm there's nothing fishy with my browser, I checked out the dashboard in IE and Chrome.

raw.png

[jolson]

I was wrong about one of my previous statements. I stated that:

Nagios Log Server will by default tag any inbound logs with the current localtime of your instance.

This statement is incorrect. Nagios Log Server will take logs in and automatically translate them into UTC time (GMT).

The confusion arises because the Web interface translates those UTC logs *back* to your local timezone - which is why several of your logs are displayed in UTC time (-5 hours from CT).

Does that make sense or am I missing something here?

[polarbear1]

It makes sense: Log Timestamp (Central) ----> Log Server (UTC) ----> Dashboard (Central)

But still doesn't resolve my problem. My servers, NLS instance, and web browser (dashboard) are both in Central and set for America/Chicago.

Right now its 13:52 CDT (UTC-5).

Server:13:52:30 CDT
Node: 13:52:30 CDT
My Desktop: 13:52:30 CDT

An alert (ignore that it came in about a minute later...took a minute to put this together):

Code: Select all

{
  "_index": "logstash-2015.08.06",
  "_type": "syslog",
  "_id": "AU8EXItfDNpcdLzFzJqT",
  "_score": null,
  "_source": {
    "message": "2015/08/06 13:53:36 Verbose: xxxx",
    "@version": "1",
    "@timestamp": "2015-08-06T18:53:37.000Z",
    "type": "syslog",
    "host": "192.168.1.70",
    "priority": 133,
    "timestamp": "Aug  6 13:53:37",
    "logsource": "xxxxxx",
    "program": "xxxxxxx",
    "severity": 5,
    "facility": 16,
    "facility_label": "local0",
    "severity_label": "Notice"
  },
  "highlight": {
    "logsource": [
      "@start-highlight@xxxx@end-highlight@"
    ],
    "logsource.raw": [
      "@start-highlight@xxxx@end-highlight@"
    ]
  },
  "sort": [
    1438887217000
  ]
}

[jolson]

Excuse me if I don't understand the problem, but:

"message": "2015/08/06 13:53:36 Verbose: xxxx",

Was set by your remote machine - this looks good since the remote box pulled the local timezone (CT).

"timestamp": "Aug 6 13:53:37",

Was also set by the remote machine.

"@timestamp": "2015-08-06T18:53:37.000Z",

Was set by Nagios Log Server (in UTC) for backend purposes.

All of this looks correct to me. It looks like the issue might be here?

2015-08-06 14_47_48-dashboard.PNG (PNG Image, 1415 × 204 pixels) - Firefox Developer Edition.png

It is my understanding that Kibana will take the UTC time of the '@timestamp' field and replace it with the localtime of your browser - which should be 10:05:xx in the above picture.

Could you let me know how I could replicate this? I've gone through all of my dashboards, and the timestamps are translated by the dashboard properly (from UTC to CT). If you could upload the dashboard that's having problems, I'd be happy to import it on my end to see if the problem is the same:

2015-08-06 14_50_24-Dashboard • Nagios Log Server - Firefox Developer Edition.png