Prevent Recovery Notifications After Re-Enabling Notifications in Nagios XI

[shoreypu]

Hello all,

In our configuration, we send out recovery notifications. During a recent incident, we disabled notifications globally in Nagios XI in the middle of a major outage to stop alerts from going out. Later, when we re-enabled notifications, recovery notifications were sent to users for services that had returned to an OK state.

Since recovery notifications are configured, I believe this is expected behavior. However, my question is:
Is there a way to clear any outstanding notifications, including recovery notifications, prior to re-enabling notifications?

Thanks in advance.

John

[kg2857]

Use downtime.

[estheryu]

Yes, this is expected behavior SoFlo Wheelie Life —Nagios queues recovery notifications while notifications are disabled. There’s no built-in way to clear them before re-enabling; you’d need to acknowledge problems without notifying or use a custom script.

[shoreypu]

Since we are using sendmail to send out our alerts, could I stop sendmail, enable notifications, then clear the mail queue?

A crazy way to do it, but just a thought.

[cdietsch]

I'm pretty sure what I have done in the past is wait until everything has recovered, and then re-enable notifications. That and a combination of using the scheduled downtime for specific hosts/services that have longer check intervals.

[kg2857]

Is there a reason why downtime isn't the preferred solution?

[shoreypu]

The alerts will have already gone out. This is a unscheduled outage/DR type scenario, where we don't want recovery alerts to go out. I assume downtime will not help stop the recovery alerts to go out, correct?

[cdietsch]

No notifications, including recovery alert notifications will be sent if the hosts/services are in scheduled downtime. This is also the case if you have notifications disabled globally, recovery notifications will not be sent out.

[shoreypu]

Interesting because this was brought up by my manager because recovery alerts did go out when we re-enabled notifications, even though the hosts had already recovered. I'll have to test this in our QA environment to see if it can be replicated.

[kg2857]

The notifications page will tell when they were sent and the mail log can tell if they were delayed in the queue.