Dashboard query

[patalenszki.zoltan]

Dear All,

I would like to create a query on Dashboard for pattern <128>* but it seem that it ignores angle brackets.
So message <134> Oct 20 18:23:03 HOST.DOMAIN /LM/W3SVC/1/ROOT/MobilSign.Interface.NN.Wcf-4-131214475152949404: [Interface.NN.ACC] [128] [] [CallId: ] [NotifyDocument] [signDoc... also matches with query.
I tries to supress with escape character( \) but with no succes.
Could you pleae help me?

Thanks in advance!

regards,
Zoltan Patalenszki

[mcapra]

So the issue is that we would like to capture messages like the one highlighted in red and exclude the other matching messages:

2016_10_20_12_27_30_Dashboard_Nagios_Log_Server.png

The problem is with how elasticsearch is storing the "message" field internally. Without getting too technical, it's basically breaking down the message field into individual bits and often excludes special characters in it's tokenization of the "message" field. So elasticsearch really only sees "128" when the original event contains "<128>".

What we can do to mitigate this is plug the value between those brackets <128> into it's own field and query that field specifically. I wrote this very simply filter to accomplish this:

Code: Select all

if [type] == 'import_raw' {

    grok {
          match => [ 'message', '\<%{NUMBER:special_number}\>' ]
    }

}

Which will target the import_raw event type, find a number between the <> characters, and store it in the "special_number" field like so:

2016_10_20_12_38_16_Dashboard_Nagios_Log_Server.png

Now if I revise my query to target the special_number field specifically, special_number:128, I get the following results:

2016_10_20_12_39_55_Dashboard_Nagios_Log_Server.png

Which has eliminated the entries with [128] effectively.

[patalenszki.zoltan]

Sorry, but i'm newbie in Nagios environment.
Should i add the new filter on Administration/Global confoguration page?
I did that, but don't find the new field after that.

Thanks in advance!

regards,
Zoltan

[mcapra]

Can you share a screenshot of one of the messages fully expanded like so:

2016_10_24_09_07_51_Dashboard_Nagios_Log_Server.png

The filter may need to be revised to match your specific message/environment. I used if [type] == 'import_raw' as the conditional because I was testing on the import_raw input type.

[patalenszki.zoltan]

raw_data.JPG

I modified type to syslog when i tried your suggestion.
I already have a syslog filter. I tried to modify(replace the first pattern in match) it and to create a new, additional one as well.

Please find here the filter already exists in our nagios environment.

Code: Select all

if [type] == "syslog" and "%ASA-" not in [message] and "localhost" not in [host] {
    grok {
      match => [ "message", "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]
    }
  }

Thanks in advance!

[mcapra]

This site is your very best friend when creating grok filters:

http://grokdebug.herokuapp.com/

In the top box you put your sample message, and in the bottom box your grok filter rules:

2016_10_24_11_16_38_Grok_Debugger.png

You will see your field matches in the results box underneath.

Here's the filter pattern i've used to deconstruct this data completely:

Code: Select all

\<%{NUMBER:special_number}\> %{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} /%{DATA:syslog_program}: \[%{DATA:field1}\] \[%{DATA:field2}\] \[%{DATA:field3}\] \[CallId\: %{DATA:CallId}\] \[%{DATA:field4}\] \[%{DATA:field5}\] %{GREEDYDATA:field6}

Use that in the grok debugger linked above and rename field1, field2, ... field6 to more appropriate names for your use case.

[patalenszki.zoltan]

Thank you very much! It works.

Regards,
Zoli

[avandemore]

Great to hear. Is it okay to lock this thread?

[patalenszki.zoltan]

Yes. Thank You!