grok parsefalure

[jolson]

Willem,

Can you try changing hostname to the following pattern:

Code: Select all

\b(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62})(?:\.(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62}))*(\.?|\b)

The above filter should match your entire hostname.

[WillemDH]

jolson,

I managed to solve it!!
It was not this pattern:

Code: Select all

HOSTNAME \b(?:[_0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[_0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)

but this pattern I needed:

Code: Select all

HOSTNAME \b(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62})(?:\.(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62}))*(\.?|\b)

The hostname is now recognized correctly. I do still get the _grokparsefailure tag. Should this tag go away automatically once you get the correct match or do I ened to do a remove_tag grok?

Grtz

Willem

[jolson]

Willem,

Looks like we're on the same page! The _grokparsefailure tag:

Append values to the ‘tags’ field when there has been no successful match

If it is helpful, you can also add the following tag to each Grok to not add any tag on failure.

Code: Select all

tag_on_failure => []

Looking around the internet, this is often caused by unescaped double quotes or slightly improper characters.
http://stackoverflow.com/questions/2240 ... eing-happy

Use the Grok debugger tool to figure out what might be the cause:
https://grokdebug.herokuapp.com/
If you cannot find out what might be causing the parsefailure tag, I would like you to post your current input filter and an example log - I would be happy to give it a shot.

Jesse

[WillemDH]

Hey Jesse,

Thanks for the input. I'm already using http://grokconstructor.appspot.com/, which I think is a little better then heroku grok debugger.

Another question:

if I make a grok filter, with a match that would catch 75 % of the syslog events for the type this grok filter applies too. What happens with the other 25 % of the logs which don't match with the created filter?

I've been working for several hours to get this grok filter for my Brocade switches correct. I've learned alot, but I would really like to find our why I keep getting the _grokparsefailure tag. So you say if I can find the correct filter, Nagios Log Server would stop tagging the logs with _grokparsefailure?

Some example logs:

Code: Select all

<188>mrt 18 15:59:14 10.54.22.160 raslogd: 2015/03/18-14:59:14, [TS-1001], 442, WWN 10:00:00:05:1e:8f:54:8c | FID 128, WARNING, DGSG_FSENC02_SANSWB01, NTP Query failed: 256.

Timestamp => 2015-03-18T14:59:14.254Z

Code: Select all

<190>mrt 18 10:14:38 10.41.37.172 raslogd: 2015/03/18-09:14:38, [SNMP-1005], 116, WWN 10:00:00:05:1e:8f:54:8a | FID 128, INFO, CPF_FSENC02_SANSWB01, SNMP configuration attribute, SNMPv3 Trap Recipient Port 1, has changed from 1162 to 162.

Timestamp => 2015-03-18T09:14:38.465Z

The filter I'm using at the moment (with an edited HOSTNAME in grok patterns file)

Code: Select all

if [type] == "syslog-brocade" {
    grok {
      match => { "message" => "<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR}\/%{MONTHNUM}\/%{MONTHDAY}-%{TIME}%{GREEDYDATA}WWN %{IPV6:wwn}%{GREEDYDATA}%{LOGLEVEL}\, %{HOSTNAME:hostname}" }  
    add_tag => "grokked"
    }      
  }

Apart from the _grokparsefaulure issue I have another issue with the syslog messages from our Brocade switches, as the hour is one hour off it seems. I added the created timestamps by NLS under the log examples. Syslog messages from sources which do automatically get parsed do ahve the correct timestamp. Do i have to specify timezone in my filter or something?

Grtz and tx for helping me with this.

Willem

[jolson]

Willem,

Please see the attached screenshot. It looks like you have some trailing information that grok doesn't parse - mainly "SNMP configuration attribute, SNMPv3 Trap Recipient Port 1, has changed from 1162 to 162.". If this is data that you do not care about, you could add a simple '.+' (similar to a wildcard) to the end of your pattern, making it look something like this:

Code: Select all

<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR}\/%{MONTHNUM}\/%{MONTHDAY}-%{TIME}%{GREEDYDATA}WWN %{IPV6:wwn}%{GREEDYDATA}%{LOGLEVEL}\, %{HOSTNAME:hostname}.+

If that information is something you'd like to index as a type, there are a few logstash defaults that can do that for you. Mainly {GREEDYDATA}. The filter might then look like the following:

Code: Select all

<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR}\/%{MONTHNUM}\/%{MONTHDAY}-%{TIME}%{GREEDYDATA}WWN %{IPV6:wwn}%{GREEDYDATA}%{LOGLEVEL}\, %{HOSTNAME:hostname}\, %{GREEDYDATA:info}

One thing that I want to make a note of: it looks like you're using a lot of default grok patterns, but you aren't matching them to a variable. For instance - %{YEAR} will match the YEAR part of your log file, but it will not be indexed under the 'year' field unless you have a variable assigned to it. %{YEAR:year} is an example of what it could look like.

Following that path, I matched all of your information to a variable appropriately as follows:

Code: Select all

<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day}-%{TIME:time}%{GREEDYDATA:data1}WWN %{IPV6:wwn}%{GREEDYDATA:data2}%{LOGLEVEL:loglevel}\, %{HOSTNAME:hostname}\, %{GREEDYDATA:info}

I hope that clears some things up for you!

[WillemDH]

Jesse,

Thank you very much for your input. It helped me a lot. But the syslog messages for my Brocade switches keep getting _grokparsefailures tag. And the @timestamp is still one hour off. You can clearly see the time difference in the screenshot. The time in the syslog event is correct, but the @timestamp is not... (Using your latest filter:)

Code: Select all

if [type] == "syslog-brocade" {
    grok {
      match => { "message" => "<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day}-%{TIME:time}%{GREEDYDATA:data1}WWN %{IPV6:wwn}%{GREEDYDATA:data2}%{LOGLEVEL:loglevel}\, %{HOSTNAME:hostname}\, %{GREEDYDATA:info}" }  
    add_tag => "grokked"
    }      
  }

Grtz

Willem

[jolson]

Please try this:

Code: Select all

if [type] == "syslog-brocade" {
    grok {
      match => [ "message", "<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day}-%{TIME:time}%{GREEDYDATA:data1}WWN %{IPV6:wwn}%{GREEDYDATA:data2}%{LOGLEVEL:loglevel}\, %{HOSTNAME:hostname}\, %{GREEDYDATA:info}" ]
    add_tag => "grokked"
    }     
  }

I added square brackets to the beginning and end, and changed the modifier after 'message' from '=>' to ','

Let me know if this helps. Thanks!

[WillemDH]

Jesse,

Hmm, seems your latest suggestion still gives me grokparsefailures. I'm starting to wonder if it's even possible to Automatically get rid of these tags. I will try your suggestion

Code: Select all

  tag_on_failure => []

And see if that helps.

Grtz

Willem

[WillemDH]

Using this:

Code: Select all

if [type] == "syslog-brocade" {
    grok {
      match => [ "message", "<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day}-%{TIME:time}%{GREEDYDATA:data1}WWN %{IPV6:wwn}%{GREEDYDATA:data2}%{LOGLEVEL:loglevel}\, %{HOSTNAME:hostname}\, %{GREEDYDATA:info}" ]
    tag_on_failure => []
    add_tag => "grokked"
    }     
  }

Did not remove the grokparsefailure tag. Gonan try with remove tag now.

[jolson]

Sounds good - let us know your results when you have them. Thanks!

Also, one thing that I'd like you to verify is that the changes you're making on the GUI are pushing to your nodes properly:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf

I have seen situations where Apply Config looks like it works fine however the changes don't push out. I just want to make sure that isn't the case here.