How to query complex regex pattern in nagios log server

[lmiltchev]

Can you try changing "GREEDYDATA" to "DATA" and post your patterns file? Have you tried using the grok debugger?

https://grokdebug.herokuapp.com/

[Jklre]

Can you try changing "GREEDYDATA" to "DATA" and post your patterns file? Have you tried using the grok debugger?

https://grokdebug.herokuapp.com/

Yes. everything works perfectly in the debugger. I changed greedydata to data and get the same result.

ss4.jpg

here are the custom patterns attached.

Mitchell.txt

[scottwilkerson]

What do you get if you check "Named Captures Only"?

[Jklre]

What do you get if you check "Named Captures Only"?

I get the same thing as in the nagios logserver dashboard

So it looks like I needed to name each of these grok rules for them to show up as a table.

I'm still seeing _grokparsefailure not quite sure why.... but the rules are working and the data I want is showing up now.

re-cap for anyone else interested in this:

I had to create the custom grok rules in a file in the "/usr/local/nagioslogserver/logstash/patterns/" folder

Create a file and create whatever custom rules I want.

ruleName regex-match (1 per line) example :
COMPANY_CODE (CoCode=[A-Z0-9]{2}|coCode=[A-Z0-9]{2})

Then when I create the input filter I need to have the grok rules as %{CustomGrokRule:Whatever_I_want_the_table_to_be_named}

if [type] == 'syslog' {
grok {
patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
match => [ 'message', '%{SYSLOG5424PRI}%{MITCH_HOST:Mitch_host}%{DATA:syslog_message}%%{CSTTERROR:CSTTError}%%%{MITCHERRORNUM}%%%{MITCHPRIORITY:Mitch_Priority}%%%{MITCH_ORG_HOST:Mitch_org_host}%{MITCHPROG:Mitch_prog}%{MITCHTIME:Mitchell_time}%{MITCHMESSAGE:Mitch_Message}%{MITCHMESSAGE2:Mitch_Message1}' ]
}
}

ss5.jpg

[ssax]

Thanks for posting your recap Jklre, that will hopefully help future visitors that may be experiencing the same issue. I see you and OP work for the same company, can you confirm that this was resolved for OP?

[Jklre]

Thanks for posting your recap Jklre, that will hopefully help future visitors that may be experiencing the same issue. I see you and OP work for the same company, can you confirm that this was resolved for OP?

We are a lot closer than we were but we are still seeing '_grokparsefailure' on all of these syslog messages. Not quite sure why or even if it matters. The next challenge is mutating some of the data and more thorough testing of our rules. I'll be sure to post any challenges I come across. I have several other types of logging challenges coming up.

[scottwilkerson]

We are a lot closer than we were but we are still seeing '_grokparsefailure' on all of these syslog messages. Not quite sure why or even if it matters. The next challenge is mutating some of the data and more thorough testing of our rules. I'll be sure to post any challenges I come across. I have several other types of logging challenges coming up.

This could be because the syslog input tries to parse the message as a RFC3164 syslog message BEFORE passing to the filter you have specified, if it does not match the format exactly, it will add the "_grokparsefailure" tag.

You could change the syslog input to a tcp input and it would not try to parse it before passing to your filter, or, you could remove the tag from within your filter..

[Jklre]

We are a lot closer than we were but we are still seeing '_grokparsefailure' on all of these syslog messages. Not quite sure why or even if it matters. The next challenge is mutating some of the data and more thorough testing of our rules. I'll be sure to post any challenges I come across. I have several other types of logging challenges coming up.

This could be because the syslog input tries to parse the message as a RFC3164 syslog message BEFORE passing to the filter you have specified, if it does not match the format exactly, it will add the "_grokparsefailure" tag.

You could change the syslog input to a tcp input and it would not try to parse it before passing to your filter, or, you could remove the tag from within your filter..

Thanks Scott. I'm still seeing some other behavior that's a little bit not what i'm expecting.

1.) I'm seeing that for each input filter I add it will add a tag to it regardless of if there is a match or not. Is there a way to make it tag a log only if it matches that input? I tried using the Break_on_match option but its still tagging it with all of them.

My Inputs
if [type] == 'syslog' {
grok {
patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
break_on_match => true
match => [ 'message', '%{SYSLOG5424PRI}%{MITCH_HOST:Mitch_host}%{GREEDYDATA:syslog_message}%%{CSTTERROR:CSTTError}%%%{MITCHERRORNUM:Mitch_ErrorNum}%%%{MITCHPRIORITY:Mitch_Priority}%%%{MITCH_ORG_HOST:Mitch_org_host}%{MITCHPROG:Mitch_prog}%{MITCHTIME:Mitchell_time}%{MITCHMESSAGE:Mitch_Message}%{MITCHMESSAGE2:Mitch_Message1}' ]
}
mutate {
add_tag => "Mitch_Jboss_Log"
}
}
if [type] == 'syslog' {
grok {
patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
match => [ 'message', '%{SYSLOG5424PRI}Message forwarded from %{MITCH_IBM_HOST:Mitch_host}:%{GREEDYDATA:Mitch_Message}' ]
}
mutate {
add_tag => "Mitch_IBM"
}
}
if [type] == 'syslog' {
grok {
patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
match => [ 'message', '%{SYSLOG5424PRI}%{HEARTBEAT:Mitch_Message}' ]
}
mutate {
add_tag => "Mitchell_Heart"
}
}
}

2.) When setting up alerts for a specific query with filters. I click on the display alert in dashboard or on the link the alert e-mail sends I am sent to a dashboard that does not match the query i'm trying to alert on. see below.

MY saved Query

ss6.jpg

What I get in from the e-mail link / view this alert in the dashboard link.

ss7.jpg

Also I haven't looked into it thoroughly yet but is there a way to customize those e-mail notifications that go out? Ideally id like to include some table names form the alerts into the e-mail message if possible.

Thank you

[scottwilkerson]

1. can be simplified, you are mutating either way, and we can put the add_tag inside the grok filter, change them like so

Code: Select all

if [type] == 'syslog' {
  grok {
    patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
    match => [ 'message', '%{SYSLOG5424PRI}%{MITCH_HOST:Mitch_host}%{GREEDYDATA:syslog_message}%%{CSTTERROR:CSTTError}%%%{MITCHERRORNUM:Mitch_ErrorNum}%%%{MITCHPRIORITY:Mitch_Priority}%%%{MITCH_ORG_HOST:Mitch_org_host}%{MITCHPROG:Mitch_prog}%{MITCHTIME:Mitchell_time}%{MITCHMESSAGE:Mitch_Message}%{MITCHMESSAGE2:Mitch_Message1}' ]
    add_tag => "Mitch_Jboss_Log"
  }

}
if [type] == 'syslog' {
  grok {
    patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
    match => [ 'message', '%{SYSLOG5424PRI}Message forwarded from %{MITCH_IBM_HOST:Mitch_host}:%{GREEDYDATA:Mitch_Message}' ]
    add_tag => "Mitch_IBM"
  }

}
if [type] == 'syslog' {
  grok {
    patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
    match => [ 'message', '%{SYSLOG5424PRI}%{HEARTBEAT:Mitch_Message}' ]
    add_tag => "Mitchell_Heart"
  }
}
}

Thesecond item, I'm going to have to have someone dig into, it should bring you to the same query ( not dashboard ) having all of the query elements and filters, but yours appears to be missing some of the filter items...

[jomann]

It looks like the issue with clicking into alert dashboards was a bug. I've attached a file that you can replace that will fix it.

Copy/replace the file in:
/var/www/html/nagioslogserver/www/app/dashboards