VMware Logs not coming in

[WillemDH]

hello,

I've jsut installed our first Log Server instance. I followed this guide to get ESXi logs: http://assets.nagios.com/downloads/nagi ... Server.pdf but I don't see any logs coming in.

syslog.globalhost.loghost is set top tcp://ip-address:1514 and firewall port 1514 is open.

Grtz

Willem

[lgroschen]

Did you apply config after adding the input into Log server?

Also, try this on your logserver machine:

Code: Select all

netstat -nlp | grep 1514

[WillemDH]

Hey Luke,

netstat -nlp | grep 1514

Gives no result.

I think I applied configuration, the "save" and the "save and apply" button is greyed out by the way.

Grtz

Willem

[tgriep]

Can you go to the Logserver's Global Configuration and verify that the Input you created for the ESXi server is active?
Can you run the following from a command line and post the output?

Code: Select all

service iptables status

[WillemDH]

Luke,

Output iptables on the Nagios Log Server:

Code: Select all

service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:1514
2    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:1514
3    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:2057
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:2056
5    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:5544
6    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:3515
7    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpts:9300:9400
8    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:443
9    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:80

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

And see attached screenshot for view of the global configuration.

Seem correct right?

Grtz

Willem

[tgriep]

The attached screen shot never made it to the post, can you try adding it again?

Lets install tcpdump on the log server and see if the ESXi server is sending data to the log server.

Run these commands on the logserver.

Code: Select all

yum -y install tcpdump
tcpdump port 1514

Let it run for a while to see if data is getting to the log server.
If you don't see any data, check the ESXi server's settings for syslog and the firewall to see if they are correct.

[Box293]

If you see data coming in, they this may be a timezone configuration issue as I've come across similar problems recently.

Run this command:

Code: Select all

ls -al /etc/localtime

Which will output something like:

lrwxrwxrwx 1 root root 39 Jun 26 2014 /etc/localtime -> /usr/share/zoneinfo/Australia/Melbourne

Make sure this is your correct timezone.

Run these two commands

Code: Select all

date

hwclock

Make sure they have the same date/time.

Run this command:

Code: Select all

cat /etc/php.ini | grep date.time

Make sure it has the correct timezone, in my example above it should be:

Code: Select all

date.timezone = Australia/Melbourne

If you do change it restart the httpd service

Code: Select all

service httpd restart

Run this command:

Code: Select all

cat /etc/sysconfig/clock

Make sure it has the correct timezone, in my example it should be

Code: Select all

ZONE="Australia/Melbourne"

If you change it, restart logstash

Code: Select all

service logstash restart

Also, I've had issues where the syslog service on the ESXi hosts stops working when the Log Server goes offline (you might reboot it). After that it doesn't restart when the Log Server is back online. This VMware article points to a known issue and is fixed in later versions of ESXi. Check the link for detailed information:
http://kb.vmware.com/selfservice/micros ... Id=2003127

[WillemDH]

Hey,

Installed tcpdump and started listening on port 1514. It seems not much is flowing in.

Code: Select all

ls -al /etc/localtime
lrwxrwxrwx 1 root root 35 Jan 26 16:43 /etc/localtime -> /usr/share/zoneinfo/Europe/Brussels

Date was ok:

Code: Select all

date
Tue Jan 27 08:28:59 CET 2015

But hwclock was not:

Code: Select all

hwclock
Tue 27 Jan 2015 07:27:15 AM CET  -0.306870 seconds

So I synced it with

Code: Select all

hwclock --systohc
[root@srvnaglog01 ~]# hwclock
Tue 27 Jan 2015 08:28:35 AM CET  -0.931657 seconds

So that should be ok now.

PHP time:

Code: Select all

cat /etc/php.ini | grep date.time
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.timezone
date.timezone = Europe/Brussels

Code: Select all

cat /etc/sysconfig/clock
ZONE="Europe/Brussels"
UTC=False

How frequent should an ESXi server write logs? Are these syslog messages the events in the task & events tab of an ESXi server?

I'll enable cli and check if the syslog daemon is running.

EDIT: It seems I was able to reload syslog :

Code: Select all

~ # esxcli system syslog reload
~ # ps -Cuv | grep vmsyslogd
8666 8666 vmsyslogd /bin/python
~ #

Which would imply it was still running.

Grtz

[WillemDH]

Hello,

I think the command

Code: Select all

esxcli system syslog reload

must have kickstarted the daemon, as logs are coming in now. I hope the syslog daemon is not failing every time I restart the server..., as I have like 35 ESXi servers...
I'm not looking forward to enable cli and reload every time it fails.
Can I monitor the ESXi syslog service state with Nagios XI? And maybe make a Reactor Event Handler when it fails?

Grtz and thanks.

Willem

[WillemDH]

Troy,

I did the test and indeed. After restarting the logstash service the logs stop flowing in from the ESXi server.

How would I best deal with this? Executing

Code: Select all

esxcli system syslog reload

Solved the problem again. We are on ESXi 5.5. I don't seem to find any patches for ESXi 5.5..

Grtz

Willem