Alerting settings to avoid duplicate alerts from firing
Posted: Tue May 16, 2017 8:58 pm
I have the following settings configured for an alert:
Check Interval = 5m
Loopback Period = 60m
Warning = 0
Critical = 0
The following events occurred that met the query criteria of the alert:
Event #1 occurred @ 2017-05-16T15:27:02.000
Event #2 occurred @ 2017-05-16T15:27:02.000
Event #3 occurred @ 2017-05-16T15:46:46.000
Event #4 occurred @ 2017-05-16T15:46:46.000
Event #5 occurred @ 2017-05-16T15:46:46.000
From the audit logs, type=ALERT:
2017-05-16T15:29:14.020-04:00 ALERT Alert Name TEST returned CRITICAL: 2 matching entries found |logs=2;0;0
2017-05-16T15:34:29.049-04:00 ALERT Alert Name TEST returned CRITICAL: 2 matching entries found |logs=2;0;0
2017-05-16T15:39:44.065-04:00 ALERT Alert Name TEST returned CRITICAL: 2 matching entries found |logs=2;0;0
2017-05-16T15:44:49.037-04:00 ALERT Alert Name TEST returned CRITICAL: 2 matching entries found |logs=2;0;0
2017-05-16T15:49:54.133-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T15:55:04.888-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:00:04.113-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:05:09.166-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:10:14.407-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:15:29.380-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:20:44.503-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:25:49.476-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:31:08.524-04:00 ALERT Alert Name TEST returned CRITICAL: 3 matching entries found |logs=3;0;0
2017-05-16T16:36:24.356-04:00 ALERT Alert Name TEST returned CRITICAL: 3 matching entries found |logs=3;0;0
2017-05-16T16:41:29.427-04:00 ALERT Alert Name TEST returned CRITICAL: 3 matching entries found |logs=3;0;0
2017-05-16T16:46:34.491-04:00 ALERT Alert Name TEST returned CRITICAL: 3 matching entries found |logs=3;0;0
I am looking for a way to check the logs every X minutes (or seconds) and look for new events and fire the alert if found.
What is the best practice for doing this without missing any alerts and without having duplicate alerts occur for the same events? In the case above, I would expect 2 alerts, not 16 alerts.
Check Interval = 5m
Loopback Period = 60m
Warning = 0
Critical = 0
The following events occurred that met the query criteria of the alert:
Event #1 occurred @ 2017-05-16T15:27:02.000
Event #2 occurred @ 2017-05-16T15:27:02.000
Event #3 occurred @ 2017-05-16T15:46:46.000
Event #4 occurred @ 2017-05-16T15:46:46.000
Event #5 occurred @ 2017-05-16T15:46:46.000
From the audit logs, type=ALERT:
2017-05-16T15:29:14.020-04:00 ALERT Alert Name TEST returned CRITICAL: 2 matching entries found |logs=2;0;0
2017-05-16T15:34:29.049-04:00 ALERT Alert Name TEST returned CRITICAL: 2 matching entries found |logs=2;0;0
2017-05-16T15:39:44.065-04:00 ALERT Alert Name TEST returned CRITICAL: 2 matching entries found |logs=2;0;0
2017-05-16T15:44:49.037-04:00 ALERT Alert Name TEST returned CRITICAL: 2 matching entries found |logs=2;0;0
2017-05-16T15:49:54.133-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T15:55:04.888-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:00:04.113-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:05:09.166-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:10:14.407-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:15:29.380-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:20:44.503-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:25:49.476-04:00 ALERT Alert Name TEST returned CRITICAL: 5 matching entries found |logs=5;0;0
2017-05-16T16:31:08.524-04:00 ALERT Alert Name TEST returned CRITICAL: 3 matching entries found |logs=3;0;0
2017-05-16T16:36:24.356-04:00 ALERT Alert Name TEST returned CRITICAL: 3 matching entries found |logs=3;0;0
2017-05-16T16:41:29.427-04:00 ALERT Alert Name TEST returned CRITICAL: 3 matching entries found |logs=3;0;0
2017-05-16T16:46:34.491-04:00 ALERT Alert Name TEST returned CRITICAL: 3 matching entries found |logs=3;0;0
I am looking for a way to check the logs every X minutes (or seconds) and look for new events and fire the alert if found.
What is the best practice for doing this without missing any alerts and without having duplicate alerts occur for the same events? In the case above, I would expect 2 alerts, not 16 alerts.