Service handler triggered by SOFT alerts
Posted: Mon Oct 14, 2019 4:15 pm
Hello,
We're using a service handler to send alerts to a third-party product when a critical alert fires. Our general service monitoring setup is to check every 15 minutes, then if there's a problem, check once every minute for five minutes before firing an alert.
We ran into an issue today: the service handler in question was firing every minute on SOFT alerts instead of only once for the HARD alert. This resulted in a ton of spam being sent to the team and on-call person.
Is there any way to have the service handler only fire on the final HARD alert?
As a nb: the service check in question is binary: it's either CRITICAL or it's not; there's no WARN state. I don't know if this makes a difference, but I thought I'd throw it out there anyway.
Thanks,
-- Mike Beebe
We're using a service handler to send alerts to a third-party product when a critical alert fires. Our general service monitoring setup is to check every 15 minutes, then if there's a problem, check once every minute for five minutes before firing an alert.
We ran into an issue today: the service handler in question was firing every minute on SOFT alerts instead of only once for the HARD alert. This resulted in a ton of spam being sent to the team and on-call person.
Is there any way to have the service handler only fire on the final HARD alert?
As a nb: the service check in question is binary: it's either CRITICAL or it's not; there's no WARN state. I don't know if this makes a difference, but I thought I'd throw it out there anyway.
Thanks,
-- Mike Beebe