trial issue, setup multiple sources, none showing up

[jolson]

That said, why would NLS care? Shouldn't the data show up in the log list when it was received regardless of what time the OS thinks it is?

The logs are time-stamped at the source - in this case the source would be your Linux source. This timestamp is displayed in the @timestamp field, and does not affect how logs are tracked internally. (Logs have two timestamps - the external @timestamp field, and the internal timestamp used to display data).

Logstash will then mark logs using UTC, and then Logstash pushes them to the elasticsearch database. Kibana reads from elasticsearch - adjusting the @timestamp based on your browser-reported timezone.

The ELK stack is designed this way so that you can have logs come in from different timezones be displayed using your local timezone - yet still retain their original @timestamp field for your records.

[ucemike]

So it seems once I told it to use GMT the log entries for my log sources was fine... except for the OVF/NSL log entries. I can't seem to figure out why but if I add a log entry I have to look back 4 hours in the search for them to show up instead of "last 5 minutes" like I do the rest.

I ran...

Code: Select all

 /usr/local/nagioslogserver/scripts/change_timezone.sh -z GMT

on the NLS.

Code: Select all

[root@nagios-log1 log]# date
Thu May  7 18:44:29 GMT 2015
[root@nagios-log1 log]# hwclock
Thu 07 May 2015 06:44:31 PM GMT  -0.299307 seconds

Code: Select all

[root@nagios-log1 log]# cat /etc/sysconfig/clock
ZONE="GMT"

Code: Select all

[root@nagios-log1 log]# cat /etc/php.ini | grep date.time
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.timezone
date.timezone = GMT

Code: Select all

[root@nagios-log1 log]# ls -al /etc/localtime
lrwxrwxrwx 1 root root 23 May  7 18:40 /etc/localtime -> /usr/share/zoneinfo/GMT

I've restarted httpd and logstash a few times to make sure it took the new config changes as well.

[jolson]

So it seems once I told it to use GMT the log entries for my log sources was fine... except for the OVF/NSL log entries.

Do you mean that when Nagios Log Server sends logs to itself, the logs show up 4 hours prior to when they're supposed to? I am experiencing a similar issue on one of my lab boxes, and have been working with it today with no success. I'll let you know as I make more progress.

[ucemike]

So it seems once I told it to use GMT the log entries for my log sources was fine... except for the OVF/NSL log entries.

Do you mean that when Nagios Log Server sends logs to itself, the logs show up 4 hours prior to when they're supposed to? I am experiencing a similar issue on one of my lab boxes, and have been working with it today with no success. I'll let you know as I make more progress.

Yes, thats exactly the same thing I am seeing. I can send a test log entry and unless I search for >4 hours it won't show up. RUn the same test on my source sites and I can use a search of 5m.

[jolson]

Did you restart rsyslog and SSH? I am not sure which restart resolved my problem, but restarting both rsyslog and sshd resolved my issues. I assume this is because they need restarts to pull the new system time properly.

Code: Select all

service rsyslog restart && service sshd restart

[ucemike]

Did you restart rsyslog and SSH? I am not sure which restart resolved my problem, but restarting both rsyslog and sshd resolved my issues. I assume this is because they need restarts to pull the new system time properly.

Code: Select all

service rsyslog restart && service sshd restart

I am certain I've restarted rsyslog but had not sshd. I restarted both and indeed the logs are showing up immediately now. Odd but satisfactory!

Thanks once again.

[jolson]

No problem! I'll lock this thread - feel free to open up a new one if you have any additional questions or problems. Thanks!