VMware Logs not coming in

[tgriep]

Here is a link to VMware's knowledge base talking about ESXi not reconnecting syslog if it lost connection to the remote server.

http://kb.vmware.com/selfservice/micros ... Id=2003127

[WillemDH]

Yes, I know tgriep. Troy already posted that link.

It does not give any solution though, nor any update on when this issue would be solved. As in order to execute "esxcli [--server hostname --user username] system syslog reload" we have to enable ssh etc, login, I was hoping to work towards a solution where "esxcli [--server hostname --user username] system syslog reload" would be executed at the moment the problem arises. I suspect I could use the vma for that, but it's only used for the box293 plugin atm, so before I start looking into this option, I was hoping to hear Troy's opinion.

Grtz

Willem

[tgriep]

Sorry, Missed Troy's post.

Are your server patched to the minimums that VMware recommends?

[WillemDH]

Well Tgriep, I'm sorry too. I thought we were on 5.5, but when execute ']vmware -vl' on the ESX server I added, I get
[code
VMware ESXi 5.1.0 build-2000251
VMware ESXi 5.1.0 Update 2[/code]

It seems we are on ESXi 5.1.0 build 2000251, which seems higher then the proposed build with the 'patch' for tcp:

vSphere ESXi 5.1 build 1483097 or higher. For more information, see VMware ESXi 5.1 Update 2 Release Notes.

But we still experience the issue...

Grtz

Willem

[tgriep]

I found this in VMware's article for ESXi 5.1

After a network or storage interruption, syslog over TCP, syslog over SSL, and storage logging do not restart automatically.
After a network or storage interruption, the syslog service does not restart automatically in certain configurations. These configurations include syslog over TCP, syslog over SSL, and the interrupt storage logging.

Workaround: Restart syslog explicitly by running the following command:
esxcli system syslog reload You can also configure syslog over UDP, which restarts automatically.

I looks like if you setup syslog to use UDP instead of TCP, it will reconnect automatically.

[WillemDH]

Tgriep, that sounds like a nice plan.

So I reconfigured the syslog host to udp://<ip-naglog-server>:1514

The firewal settings on the esxi host was still correct, just like the iptables on the NLS.

But the syslog messages are not flowing in. I did a tcpdump port 1514 , but nothing to see. So I did another

esxcli [--server hostname --user username] system syslog reload

one the esxi server, but still nothing.

I must be missing something, as tcp worked fine...

Did another

Code: Select all

Type iptables -I INPUT -p tcp --destination-port 1514 -j ACCEPT and press Enter
• Type iptables -I INPUT -p udp --destination-port 1514 -j ACCEPT and press Enter
• Type service iptables save and press Enter 

Rechecked firewall settings on the esxi host, udp 1514 is open...

Grtz

Willem

[WillemDH]

Ok, I think I know why. UDP seems to use 514 by default in all the examples I saw. That will be it. I do some tests and will let you know.

[tgriep]

To do ports below 1024 on the log server, you will need to follow this document.
http://assets.nagios.com/downloads/nagi ... Server.pdf

[WillemDH]

Changed the LS_USER to root as described in http://assets.nagios.com/downloads/nagi ... erver.pdf

Restarted logstash service

tcpdumpport 514

And logs seem to flow in again. After applying config in global configuration, the logs were visible in NLS again.

Restarted the logstash service again and it seems the logs keep flowing in after the restart. Thanks Tgriep for the help! I'll add some more ESX servers tomorrow and will do some more tests and will let you know if the thread can- be closed.

Grtz

Willem

[tgriep]

Keep us informed.