grokparsefailure on ESX syslog message

[krobertson71]

Getting "some" of our syslog events from ESX hosts showing up as _grokparsefailure.

Code: Select all

 	<166>Dec 14 13:37:45 10.0.106.152 f46715f-f625-002481e8dc5 esxhost.me.net Vpxa: / /vmfs/volumes/55cb3008-4356f04a-1d59-002481e8dc5c/ /vmfs/volumes/55c34512-4dc40a26-a712-002481e8dc5c/ /vmfs/volumes/55cb3064-927b61d2-c3a7-002481e8dc5c/ /vmfs/volumes/55c345c4-21f16885-19ee-002481e8dc5c/ /vmfs/volumes/55cb3049-dc9f4850-41d3-002481e8dc5c/

We get millions of events from all our ESX nodes and all via syslog so not sure why some events are not parsing properly. This is just one example.

Any ideas?

Log Server 1.4

[jolson]

Are these events simply being sent to a syslog input, or are there any filters involved? If it's just the syslog input, let's observe the following:

Default syslog input filter:

Code: Select all

<%{POSINT:priority}>(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:%{SYSLOGBASE2} %{GREEDYDATA:message}

Your log:

Code: Select all

<166>Dec 14 13:37:45 10.0.106.152 f46715f-f625-002481e8dc5 esxhost.me.net Vpxa: / /vmfs/volumes/55cb3008-4356f04a-1d59-002481e8dc5c/ /vmfs/volumes/55c34512-4dc40a26-a712-002481e8dc5c/ /vmfs/volumes/55cb3064-927b61d2-c3a7-002481e8dc5c/ /vmfs/volumes/55c345c4-21f16885-19ee-002481e8dc5c/ /vmfs/volumes/55cb3049-dc9f4850-41d3-002481e8dc5c/

Plugging both of them into a debugger gives us the following result:

2015-12-14 16_51_48-Grok Debugger.png

Not the result we were looking for. After some playing around, it looks like your log doesn't match the syslog input because it doesn't match the standard syslog format:

2015-12-14 16_59_00-Test grok patterns.png

Notice that the default filter matches:
%{SYSLOGHOST:logsource} %{SYSLOGPROG}:

Meaning that the program must come immediately after the logsource. In a log that follows the format exactly, you'll see something like this:
Dec 14 08:51:01 localhost kernel:

For some reason your log includes the 'f46715f-f625-002481e8dc5 esxhost.me.net ' bit of information, which appears to throw out filter off. You'll likely have to design a custom filter to fit your ESXI syslogs:
http://support.nagios.com/forum/viewtop ... 37&t=32221
http://support.nagios.com/forum/viewtop ... 68#p134768
Less long:
http://support.nagios.com/forum/viewtop ... 28#p137728

This pattern may give you a good start:
https://github.com/harrytruman/logstash ... stash.conf