Page 1 of 1
Dashboard query
Posted: Thu Oct 20, 2016 11:39 am
by patalenszki.zoltan
Dear All,
I would like to create a query on Dashboard for pattern <128>* but it seem that it ignores angle brackets.
So message <134> Oct 20 18:23:03 HOST.DOMAIN /LM/W3SVC/1/ROOT/MobilSign.Interface.NN.Wcf-4-131214475152949404: [Interface.NN.ACC] [128] [] [CallId: ] [NotifyDocument] [signDoc... also matches with query.
I tries to supress with escape character( \) but with no succes.
Could you pleae help me?
Thanks in advance!
regards,
Zoltan Patalenszki
Re: Dashboard query
Posted: Thu Oct 20, 2016 12:40 pm
by mcapra
So the issue is that we would like to capture messages like the one highlighted in red and exclude the other matching messages:
2016_10_20_12_27_30_Dashboard_Nagios_Log_Server.png
The problem is with how elasticsearch is storing the "message" field internally. Without getting too technical, it's basically breaking down the message field into individual bits and often excludes special characters in it's tokenization of the "message" field. So elasticsearch really only sees "128" when the original event contains "<128>".
What we can do to mitigate this is plug the value between those brackets <
128> into it's own field and query that field specifically. I wrote this very simply filter to accomplish this:
Code: Select all
if [type] == 'import_raw' {
grok {
match => [ 'message', '\<%{NUMBER:special_number}\>' ]
}
}
Which will target the import_raw event type, find a number between the <> characters, and store it in the "special_number" field like so:
2016_10_20_12_38_16_Dashboard_Nagios_Log_Server.png
Now if I revise my query to target the special_number field specifically,
special_number:128, I get the following results:
2016_10_20_12_39_55_Dashboard_Nagios_Log_Server.png
Which has eliminated the entries with [128] effectively.
Re: Dashboard query
Posted: Mon Oct 24, 2016 6:54 am
by patalenszki.zoltan
Sorry, but i'm newbie in Nagios environment.
Should i add the new filter on Administration/Global confoguration page?
I did that, but don't find the new field after that.
Thanks in advance!
regards,
Zoltan
Re: Dashboard query
Posted: Mon Oct 24, 2016 9:09 am
by mcapra
Can you share a screenshot of one of the messages fully expanded like so:
2016_10_24_09_07_51_Dashboard_Nagios_Log_Server.png
The filter may need to be revised to match your specific message/environment. I used
if [type] == 'import_raw' as the conditional because I was testing on the
import_raw input type.
Re: Dashboard query
Posted: Mon Oct 24, 2016 11:07 am
by patalenszki.zoltan
raw_data.JPG
I modified type to syslog when i tried your suggestion.
I already have a syslog filter. I tried to modify(replace the first pattern in match) it and to create a new, additional one as well.
Please find here the filter already exists in our nagios environment.
Code: Select all
if [type] == "syslog" and "%ASA-" not in [message] and "localhost" not in [host] {
grok {
match => [ "message", "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]
}
}
Thanks in advance!
Re: Dashboard query
Posted: Mon Oct 24, 2016 11:21 am
by mcapra
This site is your very best friend when creating grok filters:
http://grokdebug.herokuapp.com/
In the top box you put your sample message, and in the bottom box your grok filter rules:
2016_10_24_11_16_38_Grok_Debugger.png
You will see your field matches in the results box underneath.
Here's the filter pattern i've used to deconstruct this data completely:
Code: Select all
\<%{NUMBER:special_number}\> %{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} /%{DATA:syslog_program}: \[%{DATA:field1}\] \[%{DATA:field2}\] \[%{DATA:field3}\] \[CallId\: %{DATA:CallId}\] \[%{DATA:field4}\] \[%{DATA:field5}\] %{GREEDYDATA:field6}
Use that in the grok debugger linked above and rename field1, field2, ... field6 to more appropriate names for your use case.
Re: Dashboard query
Posted: Tue Oct 25, 2016 8:38 am
by patalenszki.zoltan
Thank you very much! It works.
Regards,
Zoli
Re: Dashboard query
Posted: Tue Oct 25, 2016 9:50 am
by avandemore
Great to hear. Is it okay to lock this thread?
Re: Dashboard query
Posted: Tue Oct 25, 2016 10:15 am
by patalenszki.zoltan
Yes. Thank You!