Page 1 of 3
VMware Logs not coming in
Posted: Mon Jan 26, 2015 10:54 am
by WillemDH
hello,
I've jsut installed our first Log Server instance. I followed this guide to get ESXi logs:
http://assets.nagios.com/downloads/nagi ... Server.pdf but I don't see any logs coming in.
syslog.globalhost.loghost is set top tcp://ip-address:1514 and firewall port 1514 is open.
Grtz
Willem
Re: VMware Logs not coming in
Posted: Mon Jan 26, 2015 12:13 pm
by lgroschen
Did you apply config after adding the input into Log server?
Also, try this on your logserver machine:
Re: VMware Logs not coming in
Posted: Mon Jan 26, 2015 1:45 pm
by WillemDH
Hey Luke,
netstat -nlp | grep 1514
Gives no result.
I think I applied configuration, the "save" and the "save and apply" button is greyed out by the way.
Grtz
Willem
Re: VMware Logs not coming in
Posted: Mon Jan 26, 2015 2:18 pm
by tgriep
Can you go to the Logserver's Global Configuration and verify that the Input you created for the ESXi server is active?
Can you run the following from a command line and post the output?
Re: VMware Logs not coming in
Posted: Mon Jan 26, 2015 4:28 pm
by WillemDH
Luke,
Output iptables on the Nagios Log Server:
Code: Select all
service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:1514
2 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:1514
3 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2057
4 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2056
5 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:5544
6 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:3515
7 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpts:9300:9400
8 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443
9 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
And see attached screenshot for view of the global configuration.
Seem correct right?
Grtz
Willem
Re: VMware Logs not coming in
Posted: Mon Jan 26, 2015 5:20 pm
by tgriep
The attached screen shot never made it to the post, can you try adding it again?
Lets install tcpdump on the log server and see if the ESXi server is sending data to the log server.
Run these commands on the logserver.
Code: Select all
yum -y install tcpdump
tcpdump port 1514
Let it run for a while to see if data is getting to the log server.
If you don't see any data, check the ESXi server's settings for syslog and the firewall to see if they are correct.
Re: VMware Logs not coming in
Posted: Mon Jan 26, 2015 7:41 pm
by Box293
If you see data coming in, they this may be a timezone configuration issue as I've come across similar problems recently.
Run this command:
Which will output something like:
lrwxrwxrwx 1 root root 39 Jun 26 2014 /etc/localtime -> /usr/share/zoneinfo/Australia/Melbourne
Make sure this is your correct timezone.
Run these two commands
Make sure they have the same date/time.
Run this command:
Make sure it has the correct timezone, in my example above it should be:
Code: Select all
date.timezone = Australia/Melbourne
If you do change it restart the httpd service
Run this command:
Make sure it has the correct timezone, in my example it should be
If you change it, restart logstash
Also, I've had issues where the syslog service on the ESXi hosts stops working when the Log Server goes offline (you might reboot it). After that it doesn't restart when the Log Server is back online. This VMware article points to a known issue and is fixed in later versions of ESXi. Check the link for detailed information:
http://kb.vmware.com/selfservice/micros ... Id=2003127
Re: VMware Logs not coming in
Posted: Tue Jan 27, 2015 2:34 am
by WillemDH
Hey,
Installed tcpdump and started listening on port 1514. It seems not much is flowing in.
Code: Select all
ls -al /etc/localtime
lrwxrwxrwx 1 root root 35 Jan 26 16:43 /etc/localtime -> /usr/share/zoneinfo/Europe/Brussels
Date was ok:
But hwclock was not:
Code: Select all
hwclock
Tue 27 Jan 2015 07:27:15 AM CET -0.306870 seconds
So I synced it with
Code: Select all
hwclock --systohc
[root@srvnaglog01 ~]# hwclock
Tue 27 Jan 2015 08:28:35 AM CET -0.931657 seconds
So that should be ok now.
PHP time:
Code: Select all
cat /etc/php.ini | grep date.time
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.timezone
date.timezone = Europe/Brussels
Code: Select all
cat /etc/sysconfig/clock
ZONE="Europe/Brussels"
UTC=False
How frequent should an ESXi server write logs? Are these syslog messages the events in the task & events tab of an ESXi server?
I'll enable cli and check if the syslog daemon is running.
EDIT: It seems I was able to reload syslog :
Code: Select all
~ # esxcli system syslog reload
~ # ps -Cuv | grep vmsyslogd
8666 8666 vmsyslogd /bin/python
~ #
Which would imply it was still running.
Grtz
Re: VMware Logs not coming in
Posted: Tue Jan 27, 2015 2:51 am
by WillemDH
Hello,
I think the command
must have kickstarted the daemon, as logs are coming in now. I hope the syslog daemon is not failing every time I restart the server..., as I have like 35 ESXi servers...
I'm not looking forward to enable cli and reload every time it fails.
Can I monitor the ESXi syslog service state with Nagios XI? And maybe make a Reactor Event Handler when it fails?
Grtz and thanks.
Willem
Re: VMware Logs not coming in
Posted: Tue Jan 27, 2015 3:02 am
by WillemDH
Troy,
I did the test and indeed. After restarting the logstash service the logs stop flowing in from the ESXi server.
How would I best deal with this? Executing
Solved the problem again. We are on ESXi 5.5. I don't seem to find any patches for ESXi 5.5..
Grtz
Willem