NLS Dashboard shows logs with a several hours delay.

[Dante]

Yesterday I started sending logs from multiple hosts to NLS. I couldn't see the logs appear in the dashboard at the time. This morning, without me doing anything, the logs started appearing. Now, I have powered down one of the hosts which was sending multiple logs a minute. I have confirmed using tcpdump that NLS is not receiving any traffic from that host. However NLS dashboard keeps showing new logs with recent timestamps from that host.

Code: Select all

2016-04-04T13:47:02.000+01:00	YY.YY.YY.YY	syslog	Failed password for root from ZZ.ZZ.ZZ.ZZ port PPPP ssh2

My current guess is that the logs are delayed because of time zone mismatch. It sounds similar to "trial issue, setup multiple sources, none showing up" thread.

In case this is related to time zones, here is my console output.

Code: Select all

[ec2-user@ip-XX-XX-XX-XX ~]$ ls -al /etc/localtime
lrwxrwxrwx 1 root root 33 Apr  4 09:15 /etc/localtime -> /usr/share/zoneinfo/Europe/London
[ec2-user@ip-XX-XX-XX-XX ~]$ date
Mon Apr  4 13:40:53 BST 2016
[ec2-user@ip-XX-XX-XX-XX ~]$ hwclock
Cannot access the Hardware Clock via any known method.
Use the --debug option to see the details of our search for an access method.
[ec2-user@ip-XX-XX-XX-XX ~]$ cat /etc/php.ini | grep date.time
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.timezone
date.timezone = Europe/London
[ec2-user@ip-XX-XX-XX-XX ~]$ cat /etc/sysconfig/clock
ZONE="Europe/London"
UTC=True

The system which is the source of the logs shows the same console output as above.

These show time and timezone which are correct for me. However after reboot of NLS server, the delayed logs are still appearing as if they are recent.

[Dante]

In order to make things clearer, I'm trying to wipe all the logs from NLS. Is there an easy way to do this?

I tried:

Code: Select all

curl -XDELETE "http://localhost:9200/nagioslogserver/"
{"acknowledged":true}

but this didn't seem to have the intended effect.
Edit: This command has actually deleted my NLS admin login and I had to search the forum to find a way to recover it.

[rkennedy]

What kind of resources do you have allocated to this machine? It sounds like you're hitting a throttle somewhere.

[Dante]

I run it on EC2 m3.medium instance. My cpu load average <0.1.

[jolson]

These show time and timezone which are correct for me. However after reboot of NLS server, the delayed logs are still appearing as if they are recent.

I am interested in seeing one of the problem logs that you're receiving via the web GUI (click on a particular log to expand it). It would help a lot if you could send us a screenshot of that log, in addition to the output of the following command:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/*

[Dante]

The delayed logs have eventually stopped appearing in the Dashboard. It now works as expected.

My guess is that changing the time zone settings didn't affect already existing logs which were registered as if they were from the future and it took time to catch up to those 'future' logs.

I think the thread can be closed now.

Thanks.