How to query complex regex pattern in nagios log server

[Mitchell]

Hi ,

I am new to nagios log server , recently we installed and configured syslog stream inputs , now my question is if i want make complex regex query on nagios log server how can i do that , for exaple below is the regex match
[(?:wlimtch-prod.*|wliprod_core1|p.+lxv jboss_jbc.+):.*%%.*%%(9300[0-5]|930[1245]0|93[23]00|933[3-7]0|9338[2389]|9339[01])%%]

i tried to put the above regex in the nagios log server query dashboard it didnot work , please can you help me with this ?

Thnaks,
Mitchell

[scottwilkerson]

Did you specifically select a regex query type?

You can do this by clicking on the colored icon on the left of the query field and selecting regex from the dropdown.

[scottwilkerson]

I ran the regex you provided through several tools
http://www.softlion.com/webTools/RegExp ... fault.aspx
https://www.regex101.com/
http://www.freeformatter.com/regex-tester.html

and it came back invalid regex with too many )

[Jklre]

I ran the regex you provided through several tools
http://www.softlion.com/webTools/RegExp ... fault.aspx
https://www.regex101.com/
http://www.freeformatter.com/regex-tester.html

and it came back invalid regex with too many )

I'm actually having similar issues with complex regex log matching problems. Were moving from an older log analyzer that uses complex regex rules that I am attempting to port over to Nagios log server. I ran all these rules through similar tools where they all show successful matches in testing but Nagios Log server will match a random character instead of a full pattern match.

For example using a regex string like: (ID [0-9]{4}).+(139945289).+(10710[0-9]).+(jboss_jbc[0-9]{1,2}).+

Will match a random character in a log like "1" in field syslog priority instead of the whole pattern in the message field like I'm expecting.

I'm assuming there's a better way of writing these queries besides pure regex.

[scottwilkerson]

I would write them differently, but would need to see what you are matching against.

One thing to note is normal fields are "analyzed" for search strings and the default will match the analyzed values of all fields, however if you want to match the not_naalyzed version you can add .raw to the field and your query will match on the raw value, such as

Code: Select all

message.raw:(ID [0-9]{4}).+(139945289).+(10710[0-9]).+(jboss_jbc[0-9]{1,2}).+

this will not use the analyzed version but match on the raw string that is in the message field

[Jklre]

I would write them differently, but would need to see what you are matching against.

One thing to note is normal fields are "analyzed" for search strings and the default will match the analyzed values of all fields, however if you want to match the not_naalyzed version you can add .raw to the field and your query will match on the raw value, such as

Code: Select all

message.raw:(ID [0-9]{4}).+(139945289).+(10710[0-9]).+(jboss_jbc[0-9]{1,2}).+

this will not use the analyzed version but match on the raw string that is in the message field

Thank you for the reply Scott.

Here is an example of one of the messages i'm trying to match.

message text:
<11>pjbc7n4lxv jboss_jbc7: node4: [ID 21526 Feb 3, 2015 10:45:05 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%139950883%%82008%%WARNING%%papp11lxv.mitchell.com:jboss_jbc7:node2%%TOTALLOSS%%TOTALLOSS%%2015-02-03T10:45:04.699-08:00%%NotificationUtil%%sendEventNofitication%%%%W1%%%%[jboss_jbc7] Error calling custom setting:com.mitchell.common.dao.MICommonDAOException - Type: 67000, ErrorDateTime: Tue Feb 03 10:45:04 PST 2015, ClassName: BaseDAO, MethodName: openConnection, SeverName: papp11lxv.mitchell.com, Description: Error opening connection.

Regex Search query:
%%([0-9]{9})%%(82008)%%(WARNING|FATAL)%%(papp[0-9]{1,2}lxv.+):(jboss_jbc[0-9]{1,2}):(node[0-9])%%([a-z]+)%%([a-z]+)%%

which should match: "%%139950883%%82008%%WARNING%%papp11lxv.mitchell.com:jboss_jbc7:node2%%TOTALLOSS%%TOTALLOSS%%" from the message field.

I tried this with the message.raw: to see if that made a difference but i got the same result.

I can do a simple search and match for just a small part of the field (82008) and get results but matching the whole part of the message field does not seem to come up with a match.

Any ideas on what I'm missing? Or do you think there is a better way of tackling this problem? I have about 2000 other similar rules to go through.

[scottwilkerson]

Actually based on your screenshot, it looks like that regex matched 2 items.

I do note that only 82008 is highlighted, but I will admin the highlighting could have a bug on the regex queries and is just highlighting the first portion.

However, it matched the whole thing, or you would not have any results.

[Jklre]

Actually based on your screenshot, it looks like that regex matched 2 items.

I do note that only 82008 is highlighted, but I will admin the highlighting could have a bug on the regex queries and is just highlighting the first portion.

However, it matched the whole thing, or you would not have any results.

Sorry for the confusion its actually 2 screen shots merged into one..(my lack of photoshop skills) the top portion was the query

%%([0-9]{9})%%(82008)%%(WARNING|FATAL)%%(papp[0-9]{1,2}lxv.+):(jboss_jbc[0-9]{1,2}):(node[0-9])%%([a-z]+)%%([a-z]+)%%

ss1.jpg

The second one is just the simple regex query (82008)



I hope this clears things up

[scottwilkerson]

I looked a little closer at the query syntax and it does say in the elasticsearch guide it is not fully perl compliant, and I in fact couldn't make your item match
http://www.elasticsearch.org/guide/en/e ... exp-syntax

These may take some trial and error to match exactly what you need.

[Jklre]

I looked a little closer at the query syntax and it does say in the elasticsearch guide it is not fully perl compliant, and I in fact couldn't make your item match
http://www.elasticsearch.org/guide/en/e ... exp-syntax

These may take some trial and error to match exactly what you need.

I kept on running into issues with the longer regex queries so I'm looking into using a grok input to break everything up into tables so I can get better use of this data.
I created a bunch of grok rules in the patterns folder. "/usr/local/nagioslogserver/logstash/patterns/CUSTOMPATTERNFILENAME"
and created a custom input filter:

if [type] == 'syslog' {
grok {
patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
match => [ 'message', '%{SYSLOG5424PRI}%{MITCH_HOST}%{GREEDYDATA:syslog_message}%%{CSTTERROR}%%%{MITCHERRORNUM}%%%{MITCHPRIORITY}%%%{MITCH_ORG_HOST}%{MITCHPROG}%{MITCHTIME}%{MITCHMESSAGE}%{MITCHMESSAGE2}' ]
}
}

The new issue I'm running into is that the custom rules are only matching some rules but none of the Custom ones.

ss3.jpg