5.5.0 Receiving Recovery Notifications off of Warnings

[Mortus]

The behavior that I have my alerts set up is that they will send out email notifications on criticals and recoveries. How this functioned prior to my upgrading to 5.5.0 is that if a service/host went into a warning state and then recovered, nothing would happen. It would only notify if it went critical and then recovered.

After upgrading to 5.5.0 I am receiving email notifications of recovery based off of services/hosts in a warning state. Was this an intended change, and if so is there a way to disable it?

[lmiltchev]

How this functioned prior to my upgrading to 5.5.0 is that if a service/host went into a warning state and then recovered, nothing would happen. It would only notify if it went critical and then recovered.

This doesn't sound right... When a host/service recovers from a hard non-ok state, we have a "hard recovery", and the recovery notifications are sent out. See our official documentation on state types here:

https://assets.nagios.com/downloads/nag ... types.html

I don't believe this changed in 5.5.0...

Are you sure that your service was actually in a "hard" warning state, prior to the recovery?

[Mortus]

I have confirmed that the service never went into a Hard state. It was in a Warning state and then recovered. The pages that we receive for this particular instance are done via Service Escalations. The escalation is set to only send pages on Critical and Ok.

[lmiltchev]

You didn't say anything about service escalations in your first post, so we were not looking into escalations. We will try to recreate the issue in-house, but we would need some more information. Can you show us an example of a service escalation that you are using?

[Mortus]

Code: Select all

define serviceescalation {
    # config_name            NOG On Call
    hostgroup_name           Production Database Server,Production Web Servers,Production Windows Servers
    service_description      *
    contact_groups           nog-on_call
    first_notification       1
    last_notification        0
    notification_interval    60
    escalation_period        xi_timeperiod_24x7
    escalation_options       c,r,
}

This is the service escalation set for our on call rotation.

[lmiltchev]

According to this configuration, after the first "regular" notification, the contacts, that are members of the "nog-on_call" contactgroup should be getting "escalated" notifications on CRITICAL and OK (RECOVERY) states.

Serviceescalation - escalation options

This directive is used to define the criteria that determine when this service escalation is used. The escalation is used only if the service is in one of the states specified in this directive. If this directive is not specified in a service escalation, the escalation is considered to be valid during all service states. Valid options are a combination of one or more of the following:
r = escalate on an OK (recovery) state,
w = escalate on a WARNING state,
u = escalate on an UNKNOWN state, and
c = escalate on a CRITICAL state.

Example: If you specify w in this field, the escalation will only be used if the service is in a WARNING state.

Parameter name: escalation_options
Required: no

This hasn't changed in XI 5.5.0. It should have worked the same way prior to the upgrade (and after).

Having said that, there is a Nagios Core issue in 5.5.0 that could be affecting you... It is fixed in Nagios XI 5.5.1. I would recommend upgrading to the latest version of XI to see if this is going to resolve your problem.

This is the bug fix (from the chagelog):

Fixed fix check_interval and retry_interval bug in Core 4.4.1 (Core Patch) -SW,JO

https://www.nagios.com/downloads/nagios-xi/change-log/

[Mortus]

I have upgraded to 5.5.1 but am still having the same issue occur. I have provided an example. All of the notifications here are coming from service escalations. The Warning notifications are intentional, but the ones labeled 'Custom: notify-service-by-pager' are only set for alerting from critical and ok states.

[lmiltchev]

The Warning notifications are intentional, but the ones labeled 'Custom: notify-service-by-pager' are only set for alerting from critical and ok states.

In my opinion, this is exactly what is happening (and what it should happen)... You service recovered and your contactgroup received an escalated recovery notification as per your config.

escalation_options c,r,

r = escalate on an OK (recovery) state,

Am I missing the point?

[Mortus]

Prior to upgrade, we would only receive the recovery notification if the service recovered from a critical state using the escalation feature. This was a preferable case because we use the escalation for our on-call. So receiving pages in the middle of the night for recoveries of objects that weren't critical to begin with isn't ideal.

[lmiltchev]

Prior to upgrade, we would only receive the recovery notification if the service recovered from a critical state using the escalation feature.

This sounds strange - we will need to do some more digging into it. What was the Nagios XI version that you upgraded from?