_grokparsefailure on Cisco Logs

[egrimisu]

We have been looking for a si,ple logserver solution for some time now. We do find that Nagios LogServer is quite adecvate to our needs. Our goal for the time beeing is to collect syslog from some linux server, windows eventlogs and syslog from Cisco switched. i have been testing this solution for a few days now in order to validate the solution. Windows eventlogs and linux syslogs are parsed Ok but the Cisco logs not. I have created a Grok pattern and validated it using grokdebug.herokuapp.com. So here is the config:

Cisco log format:

Code: Select all

<189>166936: .Mar 27 2015 10:00:11.061 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/9, changed state to up

Configuration used:

Code: Select all

input {
    syslog {
        type => 'syslog'
        port => 5544
    }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
      tcp {
        port => 514
        type => "syslog"
        tags => [ "Cisco" ]
      }
      udp {
        port => 514
        type => "syslog"
        tags => [ "Cisco" ]
      }
}

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
    if "Cisco" in [tags] {
      grok {
        patterns_dir => "/usr/local/nagioslogserver/logstash/patterns"
        match => [ "message", "<%{POSINT:seqnum1}>%{POSINT:seqnum2}:%{SPACE} .%{CISCOTIMESTAMPZ:cisco_timestamp}: \%%{DATA:facility}-%{POSINT:severity}-%{DATA:mnemonic}: %{GREEDYDATA:log_message}" ]
      }
    }
}

output {
    elasticsearch { host => localhost }
    stdout { codec => rubydebug }
}

Grok pattern created and added to a new file in /usr/local/nagioslogserver/logstash/patterns:

Code: Select all

CISCOTIMESTAMPZ %{MONTH} +%{MONTHDAY}(?: %{YEAR})? %{TIME} %{TZ}

And in the end the logs are not grokked correctly, could someone help?

[jolson]

First thing - I see that you are using a privileged port (514) to import your Cisco logs. If you have not done so already, please follow this guide to listen on privileged ports properly: http://assets.nagios.com/downloads/nagi ... Server.pdf

Assuming you have already done that, I took a look at your pattern and found the following:
-You need to escape the 'period' character, as period in regex is a wildcard for any 1 character. Since you already defined {SPACE}, I removed the actual 'space' between {SPACE} and '.':
from:

Code: Select all

%{SPACE} .%

to:

Code: Select all

%{SPACE}\.%

After doing this, everything on my end matched. Please try it out and let me know if it works for you. The final pattern is changed as follows:
from:

Code: Select all

<%{POSINT:seqnum1}>%{POSINT:seqnum2}:%{SPACE} .%{CISCOTIMESTAMPZ:cisco_timestamp}: \%%{DATA:facility}-%{POSINT:severity}-%{DATA:mnemonic}: %{GREEDYDATA:log_message}

to:

Code: Select all

<%{POSINT:seqnum1}>%{POSINT:seqnum2}:%{SPACE}\.%{CISCOTIMESTAMPZ:cisco_timestamp}: \%%{DATA:facility}-%{POSINT:severity}-%{DATA:mnemonic}: %{GREEDYDATA:log_message}

I hope this helps. Thank you!

[egrimisu]

No change

[jolson]

Did you apply configuration after making the change, and verify that it took in /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf? If you did, please report the output of your logstash log:

Code: Select all

tail /var/log/logstash/logstash.log

Maybe it's your timestamp that isn't taking? To test this, please use the following block of regex:

Code: Select all

<%{POSINT:seqnum1}>%{POSINT:seqnum2}:%{SPACE}\.%{GREEDYDATA:cisco_timestamp}: \%%{DATA:facility}-%{POSINT:severity}-%{DATA:mnemonic}: %{GREEDYDATA:log_message}

I have inserted GREEDYDATA where your timestamp was previously, which should capture all data up to the ':' following your timestamp. Let's see if this makes a difference. Another thing to try would be replacing your space with the regex space expression:
Replace:

Code: Select all

%{SPACE}

With:

Code: Select all

\s

This is mostly to make your code easier to follow, as it removes a variable for simple regex. I don't think it will make a usability difference, but it's worth a try.

Give the above a try and get back to us - if this test works, we will have to revise your CISCOTIMESTAMPZ definition.

[egrimisu]

That seemed to do the trick, thanks for your help.

[jolson]

No problem - would it be alright if I closed this thread?