Page 1 of 2
How long should it take for a new sources to appear?
Posted: Thu Jan 22, 2015 12:59 am
by Box293
I've configured a host to send syslogs to Log Server 2015R1.2b.
I can see the traffic hitting the Log Server by watching a TCP Dump.
On the Home page, BEFORE I configured the remote host it said "Receiving logs from 5 hosts." AFTER I configured the remote host it said "Receiving logs from 6 hosts."
I go to the Dashboard to see the new logs coming in, query is:
Nothing shows on the screen.
Under Administration > System Status:
I tried restarting Logstash Collector -> no change
I tried restarting Elasticsearch Database -> no change
logstash.log
elasticsearch.log
Code: Select all
cat /etc/localtime
AEST-10AEDT,M10.1.0,M4.1.0/3
Code: Select all
cat /etc/php.ini | grep date.timezone
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.timezone
date.timezone = Australia/Melbourne
I had this problem the other day and it seemed to work fine when I came back later.
Re: How long should it take for a new sources to appear?
Posted: Thu Jan 22, 2015 1:05 am
by itsupport_sgp
Hi,
I encounter this before as well.
For ours, it is due to the datetime, you might want to double check on it.
If it works for you in the past, it should not be the NLS firewall too.
Cheers.
Re: How long should it take for a new sources to appear?
Posted: Thu Jan 22, 2015 1:49 am
by Box293
Where did you check the date and time?
I checked on the sending server and discovered it's timezone was not correctly set. I just fixed that and rebooted the sending server however nothing is show in the Dashboard (tcpdump shows incoming data).
Re: How long should it take for a new sources to appear?
Posted: Thu Jan 22, 2015 3:34 am
by itsupport_sgp
Hi,
Try to type 'datetime' in the NLS CLI.
It will shows you the time.
Regards,
Re: How long should it take for a new sources to appear?
Posted: Thu Jan 22, 2015 2:18 pm
by slansing
You can also check for differences between the system time and php time by opening up php.ini. The main commands I run to check time for both XI and NLS are:
Code: Select all
date
hwclock
cat /etc/php.ini | grep date.time
Re: How long should it take for a new sources to appear?
Posted: Thu Jan 22, 2015 3:45 pm
by Box293
These all seem correct
Code: Select all
[root@lsproduction ~]# date
Fri Jan 23 07:42:52 AEDT 2015
[root@lsproduction ~]# hwclock
Fri 23 Jan 2015 07:42:56 AM AEDT -0.489382 seconds
[root@lsproduction ~]# cat /etc/php.ini | grep date.time
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.timezone
date.timezone = Australia/Melbourne
[root@lsproduction ~]#
tcpdump still shows logs hitting server but dashboard result returns 0
Code: Select all
tcpdump src host 10.25.6.1 and tcp dst port 5544 and dst host 10.25.5.80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
07:45:00.245023 IP vCenter.box293.local.35235 > lsproduction.box293.local.5544: Flags [P.], seq 4069869443:4069869605, ack 2323282946, win 115, options [nop,nop,TS val 12552869 ecr 139734191], length 162
07:45:00.247060 IP vCenter.box293.local.35235 > lsproduction.box293.local.5544: Flags [P.], seq 162:435, ack 1, win 115, options [nop,nop,TS val 12552869 ecr 139744208], length 273
07:45:00.251540 IP vCenter.box293.local.35235 > lsproduction.box293.local.5544: Flags [P.], seq 435:596, ack 1, win 115, options [nop,nop,TS val 12552870 ecr 139744211], length 161
07:45:00.252993 IP vCenter.box293.local.35235 > lsproduction.box293.local.5544: Flags [P.], seq 596:865, ack 1, win 115, options [nop,nop,TS val 12552871 ecr 139744215], length 269
07:45:00.258910 IP vCenter.box293.local.35235 > lsproduction.box293.local.5544: Flags [P.], seq 865:1038, ack 1, win 115, options [nop,nop,TS val 12552872 ecr 139744216], length 173
07:45:00.259327 IP vCenter.box293.local.35235 > lsproduction.box293.local.5544: Flags [P.], seq 1038:1281, ack 1, win 115, options [nop,nop,TS val 12552872 ecr 139744222], length 243
07:45:00.267497 IP vCenter.box293.local.35235 > lsproduction.box293.local.5544: Flags [P.], seq 1281:1461, ack 1, win 115, options [nop,nop,TS val 12552874 ecr 139744222], length 180
07:45:00.271406 IP vCenter.box293.local.35235 > lsproduction.box293.local.5544: Flags [P.], seq 1461:2062, ack 1, win 115, options [nop,nop,TS val 12552874 ecr 139744230], length 601
07:45:00.271602 IP vCenter.box293.local.35235 > lsproduction.box293.local.5544: Flags [P.], seq 2062:2170, ack 1, win 115, options [nop,nop,TS val 12552875 ecr 139744234], length 108
^C
9 packets captured
173 packets received by filter
78 packets dropped by kernel
Re: How long should it take for a new sources to appear?
Posted: Thu Jan 22, 2015 4:14 pm
by scottwilkerson
Something to look at, if you go to Administration -> Index Status
Do you have any indexes that are coming in with old dates?
I ask because the logs are placed in the index based on the time in the logs and not the current time.
So you have have them in there, but might need to look at a day/month/year/decade ago to see them
Re: How long should it take for a new sources to appear?
Posted: Thu Jan 22, 2015 4:54 pm
by Box293
I do have some that date back to 2014.01.01
How can I determine when an index was last updated?
If I do a Dashboard search for the past 2 years with the query host:"10.25.6.1" I do not see anything dating back that far. There are some entries for this host that came in yesterday but no entries today. However the events over time only seem to reflect a 24 hour period, not a 2 year period.
Screenshot.png
ls -al /usr/local/nagioslogserver/elasticsearch/data/ebb8a63c-023a-421f-9bd5-7ec355a6ce8f/nodes/0/indices/
total 332
drwxrwxr-x 83 nagios nagios 4096 Jan 22 19:00 .
drwxrwxr-x 4 nagios nagios 4096 Nov 24 12:46 ..
drwxrwxr-x 8 nagios nagios 4096 Nov 24 12:46 kibana-int
drwxrwxr-x 8 nagios nagios 4096 Jan 1 00:00 logstash-2014.01.01
drwxrwxr-x 8 nagios nagios 4096 Jan 1 19:00 logstash-2014.01.02
drwxrwxr-x 8 nagios nagios 4096 Jan 2 19:00 logstash-2014.01.03
drwxrwxr-x 8 nagios nagios 4096 Jan 3 19:00 logstash-2014.01.04
drwxrwxr-x 8 nagios nagios 4096 Jan 4 19:00 logstash-2014.01.05
drwxr-xr-x 8 nagios users 4096 Jan 5 19:00 logstash-2014.01.06
drwxr-xr-x 8 nagios users 4096 Jan 6 19:00 logstash-2014.01.07
drwxr-xr-x 8 nagios users 4096 Jan 7 19:00 logstash-2014.01.08
drwxr-xr-x 8 nagios users 4096 Jan 8 19:00 logstash-2014.01.09
drwxr-xr-x 8 nagios users 4096 Jan 9 19:00 logstash-2014.01.10
drwxr-xr-x 8 nagios users 4096 Jan 10 19:00 logstash-2014.01.11
drwxr-xr-x 8 nagios users 4096 Jan 11 19:00 logstash-2014.01.12
drwxr-xr-x 8 nagios users 4096 Jan 12 19:00 logstash-2014.01.13
drwxr-xr-x 8 nagios users 4096 Jan 13 19:00 logstash-2014.01.14
drwxr-xr-x 8 nagios users 4096 Jan 14 19:00 logstash-2014.01.15
drwxr-xr-x 8 nagios users 4096 Jan 15 19:00 logstash-2014.01.16
drwxrwxr-x 8 nagios nagios 4096 Nov 24 12:47 logstash-2014.11.17
drwxrwxr-x 8 nagios nagios 4096 Nov 24 12:47 logstash-2014.11.24
drwxrwxr-x 8 nagios nagios 4096 Nov 24 19:00 logstash-2014.11.25
drwxrwxr-x 8 nagios nagios 4096 Nov 25 19:00 logstash-2014.11.26
drwxrwxr-x 8 nagios nagios 4096 Nov 26 19:00 logstash-2014.11.27
drwxrwxr-x 8 nagios nagios 4096 Nov 27 19:00 logstash-2014.11.28
drwxrwxr-x 8 nagios nagios 4096 Nov 28 19:00 logstash-2014.11.29
drwxrwxr-x 8 nagios nagios 4096 Nov 29 19:00 logstash-2014.11.30
drwxrwxr-x 8 nagios nagios 4096 Nov 30 19:00 logstash-2014.12.01
drwxrwxr-x 8 nagios nagios 4096 Dec 1 19:00 logstash-2014.12.02
drwxrwxr-x 8 nagios nagios 4096 Dec 2 19:00 logstash-2014.12.03
drwxrwxr-x 8 nagios nagios 4096 Dec 3 19:00 logstash-2014.12.04
drwxrwxr-x 8 nagios nagios 4096 Dec 4 19:00 logstash-2014.12.05
drwxrwxr-x 8 nagios nagios 4096 Dec 5 19:00 logstash-2014.12.06
drwxrwxr-x 8 nagios nagios 4096 Dec 6 19:00 logstash-2014.12.07
drwxrwxr-x 8 nagios nagios 4096 Dec 7 19:00 logstash-2014.12.08
drwxrwxr-x 8 nagios nagios 4096 Dec 8 19:00 logstash-2014.12.09
drwxrwxr-x 8 nagios nagios 4096 Dec 9 19:00 logstash-2014.12.10
drwxrwxr-x 8 nagios nagios 4096 Dec 10 19:00 logstash-2014.12.11
drwxrwxr-x 8 nagios nagios 4096 Dec 11 19:00 logstash-2014.12.12
drwxrwxr-x 8 nagios nagios 4096 Dec 12 19:00 logstash-2014.12.13
drwxrwxr-x 8 nagios nagios 4096 Dec 13 19:00 logstash-2014.12.14
drwxrwxr-x 8 nagios nagios 4096 Dec 14 19:00 logstash-2014.12.15
drwxrwxr-x 8 nagios nagios 4096 Dec 15 19:00 logstash-2014.12.16
drwxrwxr-x 8 nagios nagios 4096 Dec 16 19:00 logstash-2014.12.17
drwxrwxr-x 8 nagios nagios 4096 Dec 17 19:00 logstash-2014.12.18
drwxrwxr-x 8 nagios nagios 4096 Dec 18 19:00 logstash-2014.12.19
drwxrwxr-x 8 nagios nagios 4096 Dec 19 19:00 logstash-2014.12.20
drwxrwxr-x 8 nagios nagios 4096 Dec 20 19:00 logstash-2014.12.21
drwxrwxr-x 8 nagios nagios 4096 Dec 22 08:51 logstash-2014.12.22
drwxrwxr-x 8 nagios nagios 4096 Dec 22 19:00 logstash-2014.12.23
drwxrwxr-x 8 nagios nagios 4096 Dec 23 19:00 logstash-2014.12.24
drwxrwxr-x 8 nagios nagios 4096 Dec 24 19:00 logstash-2014.12.25
drwxrwxr-x 8 nagios nagios 4096 Dec 25 19:00 logstash-2014.12.26
drwxrwxr-x 8 nagios nagios 4096 Dec 26 19:00 logstash-2014.12.27
drwxrwxr-x 8 nagios nagios 4096 Dec 27 19:00 logstash-2014.12.28
drwxrwxr-x 8 nagios nagios 4096 Dec 28 19:00 logstash-2014.12.29
drwxrwxr-x 8 nagios nagios 4096 Dec 29 19:00 logstash-2014.12.30
drwxrwxr-x 8 nagios nagios 4096 Dec 30 19:00 logstash-2014.12.31
drwxrwxr-x 8 nagios nagios 4096 Dec 31 19:00 logstash-2015.01.01
drwxrwxr-x 8 nagios nagios 4096 Jan 2 11:00 logstash-2015.01.02
drwxrwxr-x 8 nagios nagios 4096 Jan 3 11:00 logstash-2015.01.03
drwxrwxr-x 8 nagios nagios 4096 Jan 4 11:00 logstash-2015.01.04
drwxrwxr-x 8 nagios nagios 4096 Jan 5 11:00 logstash-2015.01.05
drwxr-xr-x 8 nagios users 4096 Jan 6 11:00 logstash-2015.01.06
drwxr-xr-x 8 nagios users 4096 Jan 7 11:00 logstash-2015.01.07
drwxr-xr-x 8 nagios users 4096 Jan 8 11:00 logstash-2015.01.08
drwxr-xr-x 8 nagios users 4096 Jan 9 11:00 logstash-2015.01.09
drwxr-xr-x 8 nagios users 4096 Jan 10 11:00 logstash-2015.01.10
drwxr-xr-x 8 nagios users 4096 Jan 11 11:00 logstash-2015.01.11
drwxr-xr-x 8 nagios users 4096 Jan 12 11:00 logstash-2015.01.12
drwxr-xr-x 8 nagios users 4096 Jan 13 11:00 logstash-2015.01.13
drwxr-xr-x 8 nagios users 4096 Jan 14 11:00 logstash-2015.01.14
drwxr-xr-x 8 nagios users 4096 Jan 15 11:00 logstash-2015.01.15
drwxr-xr-x 8 nagios users 4096 Jan 16 11:00 logstash-2015.01.16
drwxr-xr-x 8 nagios users 4096 Jan 16 19:00 logstash-2015.01.17
drwxr-xr-x 8 nagios users 4096 Jan 17 19:00 logstash-2015.01.18
drwxr-xr-x 8 nagios users 4096 Jan 18 19:00 logstash-2015.01.19
drwxr-xr-x 8 nagios users 4096 Jan 19 19:00 logstash-2015.01.20
drwxr-xr-x 8 nagios users 4096 Jan 20 19:00 logstash-2015.01.21
drwxr-xr-x 8 nagios users 4096 Jan 21 19:00 logstash-2015.01.22
drwxr-xr-x 8 nagios users 4096 Jan 22 19:00 logstash-2015.01.23
drwxrwxr-x 4 nagios nagios 4096 Nov 24 12:46 nagioslogserver
drwxrwxr-x 8 nagios nagios 4096 Nov 24 12:46 nagioslogserver_log
Re: How long should it take for a new sources to appear?
Posted: Thu Jan 22, 2015 5:10 pm
by scottwilkerson
Can you post the following
Lets make sure it has
then restart logstash
Re: How long should it take for a new sources to appear?
Posted: Thu Jan 22, 2015 5:16 pm
by Box293
That seems the be a problem:
Code: Select all
cat /etc/sysconfig/clock
ZONE="US/Eastern"
UTC=False
Now it is:
Code: Select all
cat /etc/sysconfig/clock
ZONE="Australia/Melbourne"
UTC=False
I restarted logstash and BANG new entries are coming on.
Awesome, great stuff, case closed
Certainly needs to be a procedure for setting the timezone for Log Server.