Need help with requested alerting criteria

[krobertson71]

Someone in our organization has requested the following:

Look for "x" phrase in the message field and email if it appears more than once in a 5 minute interval an another alert if it appears more then 5 times in a 15 minute period.

Can I get a hand up on how to accomplish this with a proper query? I know how to find the events and save that query, the alerting criteria has me a bit stumped. I know I can tell it to check every 15 minutes and look 15 minutes back, but I need it to count the instances first. Or, I need to create a query in such a way that it only returns results if the above criteria is met, then I just set up alerts as normal.

Any help here would be great!

[jolson]

krobertson71,

The answer here is that you need to design a query that returns only the results that will trigger your alert. For instance, if the string I'm concerned about is 'bad things are happening', I would start by querying for that string.

Once I found the exact log in question, I'd begin filtering it based on appropriate fields as much as possible by clicking the appropriate 'action' button:

2015-06-25 14_13_57-Dashboard • Nagios Log Server - Firefox Developer Edition.png

Once filtered, you can see the end result here:

2015-06-25 14_15_58-Dashboard • Nagios Log Server - Firefox Developer Edition.png

Now that I have an appropriate dashboard, I can save it as a query.

2015-06-25 14_17_18-Dashboard • Nagios Log Server - Firefox Developer Edition.png

Once saved, we can make an alert based on the query. This part is pretty self-explanatory.

Every 5 minutes, look back 5 minutes and alert on more than 1 entry found:

2015-06-25 14_19_18-Alerting • Nagios Log Server - Firefox Developer Edition.png

Let me know if you have any questions!

[krobertson71]

I forgot totally about the number of events.

So 1 1 #number of event is "More than 1?"

I thought that would fire for every event.

I forget what the 1 1 means since this is not a strict warning critical threshold.. or is it?

So if I want to watch for 3 events in 10 minutes would I set 1 3?

[jolson]

The warning and critical thresholds use the 'nagios plugins' syntax, you can read more about that syntax here: https://nagios-plugins.org/doc/guidelines.html

See 'table 3'.

Setting '1' for warning means 'send a WARNING alert on anything more than 1' - the critical field uses the same syntax.

If you wanted to watch for 3 events in 10 minutes you would set both fields to '2 2' to alert on 'more than 2' logs found. You would set 'lookback' to 10m'.

Let me know if you have any questions!

[krobertson71]

Sorry for the late reply. Had a family matter that put me out of touch for a bit.

Thank you very much for the explanation. I was just unsure about the two event # boxes. I thought it was warning/critical but wanted to be sure.

Thanks again and you can close this thread.