grok parsefalure

[WillemDH]

Hello,

We started sending ourbrocade sanswitch syslog messages to NLS last week. It seems some things don't get parses like it should, so I guess I'll have to start writing my first filter.

I made a specific port for all syslog messages from all our Brocade switches like this:

Code: Select all

syslog {
    type => 'syslog-brocade'
    port => 5547
}

See screenshot for a view of a message where I get a grokparsefailure. Could I please get some help to filter some values from the messages. The message looks like this:

Code: Select all

<188>mrt 02 08:51:25 10.54.97.17 raslogd: 2015/03/02-08:51:25, [TS-1001], 1545, WWN 10:00:00:05:33:72:f5:4f | FID 128, WARNING, AC_FSENC04_SANSWB02, NTP Query failed: 256.

As we don't get a source ip. Source ip is the minimum requirement I would need to retrieve. In the above example message '10.54.97.17' would be the source ip. Even better would be if we were able to make a seperate field of the WWN of the interface in the above example WWN '10:00:00:05:33:72:f5:4f'

I guess the filter would have to start with

Code: Select all

if [type] == 'syslog-brocade' {
    grok {
        match => [ 'message', '%{?}']
    }
    date {
        match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
    }
    mutate {
                        ??
    }
}

Thanks for any help creating this filter for our Brocade switches.

Willem

[tmcdonald]

Looks to be pretty close to a standard syslog format. Have you looked at the Grok Debugger?

https://grokdebug.herokuapp.com/

[WillemDH]

hey Trevor,

I'm sorry but this whole grok thing is really new to me.

I've tried some things, but i'm not sure if I'm heading in the right direction..
Should I try to make a complete match?

Code: Select all

if [type] == 'syslog-brocade' {
    grok {
        match => [ %{SYSLOG5424PRI}mrt 03 %{TIME} %{IP} raslogd: 20%{DATESTAMP}, %{SYSLOG5424SD}, 4884, WWN %{IP} | FID 128, WARNING, AC_FSENC03_SANSWB02, NTP Query failed: 256.]
    }
}

or should I use add_field to add the fields I need?

Code: Select all

if [type] == 'syslog-brocade' {
    grok {
        add_field => { "WWN %{IPV6}" => "%{IPV6}" }
    }
}

Thanks for guiding me in the right direction? Why would the default syslog filter not get applied by the way?

[jolson]

Willem,

I am by no means a grok expert, but the following tutorial helped me get a handle on it: http://logstash.net/docs/1.4.2/filters/grok. Please give that a read - it will answer your questions regarding how to build your filter.

[WillemDH]

Well I've been trying to make the correct filter for some time, learned alot about grok, but I'm stuck at the hostname at the moment. So this log:

Code: Select all

<190>mrt 04 11:06:50 10.54.22.40 raslogd: 2015/03/04-11:06:50, [SNMP-1005], 2890, WWN 10:00:00:05:33:53:42:4c | FID 128, INFO, DGOG_FSENC02_SANSWB01, SNMP configuration attribute, Trap recipient port 2 , has changed from 1162 to 162.

With this filter:

Code: Select all

<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:ip}%{GREEDYDATA:program}: %{YEAR}\/%{MONTHNUM}\/%{MONTHDAY}-%{TIME}%{GREEDYDATA}WWN %{IPV6:wwn}%{GREEDYDATA}%{LOGLEVEL:severity_label}\, %{HOSTNAME}

Gives:

Code: Select all

{
  "ip": [
    [
      "10.54.22.40"
    ]
  ],
  "program": [
    [
      " raslogd"
    ]
  ],
  "YEAR": [
    [
      "2015"
    ]
  ],
  "MONTHNUM": [
    [
      "03"
    ]
  ],
  "MONTHDAY": [
    [
      "04"
    ]
  ],
  "TIME": [
    [
      "11:06:50"
    ]
  ],
  "HOUR": [
    [
      "11"
    ]
  ],
  "MINUTE": [
    [
      "06"
    ]
  ],
  "SECOND": [
    [
      "50"
    ]
  ],
  "GREEDYDATA": [
    [
      ", [SNMP-1005], 2890, ",
      " | FID 128, "
    ]
  ],
  "wwn": [
    [
      "10:00:00:05:33:53:42:4c"
    ]
  ],
  "severity_label": [
    [
      "INFO"
    ]
  ],
  "HOSTNAME": [
    [
      "DGOG"
    ]
  ]
}

The problem is that as we use underscores in hostnames, the grok pattern HOSTNAME only reads the first part of the hostname. I read something about custom grok patterns we can save to a file. Where on the NLS would I be able to create custom grok patterns?

It seems my above grok filter is kind of working (except for the hostname), as NLS now has a separate field for program, severity_label and wwn. But the syslog messages still have the tag _grokparefailure. What else would I need to do to get rid of the groparsefailure tags?

Thanks.

Willem

[ssax]

The grok-patterns file is in the location below:

Code: Select all

/usr/local/nagioslogserver/logstash/patterns/grok-patterns

You can change the HOSTNAME pattern from:

Code: Select all

HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)

To:

Code: Select all

HOSTNAME \b(?:[_0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[_0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)

[WillemDH]

Thanks ssax,

I've replaced the HOSTNAME as you suggested. I keep getting grokparesefailures though and my hostname with underscores still isn't recognized..

Code: Select all

HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)

Do I need to restart a service or something after editing a grok pattern?

Grtz

Willem

[jolson]

After adding a filter at Administration > Global Configuration > Add Filter, you will need to Apply Configuration for the changes to take effect. Please verify that the changes have taken effect by running the following command on each node:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf

[WillemDH]

jolson,

I edited a grok pattern, not an input or a filter. Please read ssax post.

The problem is that my hostnames have underscores...

But even after replacing the default HOSTNAME I still get grokparsefailures. In the meantime I rebooted the servers and I still get grokparsefailures and the HOSTNAME is still not recognized..

Grtz

Willem

[jolson]

Looks like I need some more coffee.

You will likely need to do a logstash restart after applying the custom pattern:

Code: Select all

service logstash restart

Please let us know if that works out for you.

Edit: Just saw your update, I am testing in my lab currently.