Page 1 of 2
5.5.0 Receiving Recovery Notifications off of Warnings
Posted: Mon Jul 16, 2018 7:04 am
by Mortus
The behavior that I have my alerts set up is that they will send out email notifications on criticals and recoveries. How this functioned prior to my upgrading to 5.5.0 is that if a service/host went into a warning state and then recovered, nothing would happen. It would only notify if it went critical and then recovered.
After upgrading to 5.5.0 I am receiving email notifications of recovery based off of services/hosts in a warning state. Was this an intended change, and if so is there a way to disable it?
Re: 5.5.0 Receiving Recovery Notifications off of Warnings
Posted: Mon Jul 16, 2018 2:19 pm
by lmiltchev
How this functioned prior to my upgrading to 5.5.0 is that if a service/host went into a warning state and then recovered, nothing would happen. It would only notify if it went critical and then recovered.
This doesn't sound right... When a host/service recovers from a hard non-ok state, we have a "hard recovery", and the recovery notifications are sent out. See our official documentation on state types here:
https://assets.nagios.com/downloads/nag ... types.html
I don't believe this changed in 5.5.0...
Are you sure that your service was actually in a "hard" warning state, prior to the recovery?
Re: 5.5.0 Receiving Recovery Notifications off of Warnings
Posted: Mon Jul 16, 2018 3:05 pm
by Mortus
I have confirmed that the service never went into a Hard state. It was in a Warning state and then recovered. The pages that we receive for this particular instance are done via Service Escalations. The escalation is set to only send pages on Critical and Ok.
Re: 5.5.0 Receiving Recovery Notifications off of Warnings
Posted: Mon Jul 16, 2018 3:14 pm
by lmiltchev
You didn't say anything about service escalations in your first post, so we were not looking into escalations. We will try to recreate the issue in-house, but we would need some more information. Can you show us an example of a service escalation that you are using?
Re: 5.5.0 Receiving Recovery Notifications off of Warnings
Posted: Tue Jul 17, 2018 10:37 am
by Mortus
Code: Select all
define serviceescalation {
# config_name NOG On Call
hostgroup_name Production Database Server,Production Web Servers,Production Windows Servers
service_description *
contact_groups nog-on_call
first_notification 1
last_notification 0
notification_interval 60
escalation_period xi_timeperiod_24x7
escalation_options c,r,
}
This is the service escalation set for our on call rotation.
Re: 5.5.0 Receiving Recovery Notifications off of Warnings
Posted: Tue Jul 17, 2018 11:44 am
by lmiltchev
According to this configuration, after the first "regular" notification, the contacts, that are members of the "nog-on_call" contactgroup should be getting "escalated" notifications on CRITICAL and OK (RECOVERY) states.
Serviceescalation - escalation options
This directive is used to define the criteria that determine when this service escalation is used. The escalation is used only if the service is in one of the states specified in this directive. If this directive is not specified in a service escalation, the escalation is considered to be valid during all service states. Valid options are a combination of one or more of the following:
r = escalate on an OK (recovery) state,
w = escalate on a WARNING state,
u = escalate on an UNKNOWN state, and
c = escalate on a CRITICAL state.
Example: If you specify w in this field, the escalation will only be used if the service is in a WARNING state.
Parameter name: escalation_options
Required: no
This hasn't changed in XI 5.5.0. It should have worked the same way prior to the upgrade (and after).
Having said that, there is a Nagios Core issue in 5.5.0 that could be affecting you... It is fixed in Nagios XI 5.5.1. I would recommend upgrading to the latest version of XI to see if this is going to resolve your problem.
This is the bug fix (from the chagelog):
Fixed fix check_interval and retry_interval bug in Core 4.4.1 (Core Patch) -SW,JO
https://www.nagios.com/downloads/nagios-xi/change-log/
Re: 5.5.0 Receiving Recovery Notifications off of Warnings
Posted: Tue Jul 17, 2018 2:37 pm
by Mortus
I have upgraded to 5.5.1 but am still having the same issue occur. I have provided an example. All of the notifications here are coming from service escalations. The Warning notifications are intentional, but the ones labeled 'Custom: notify-service-by-pager' are only set for alerting from critical and ok states.
Re: 5.5.0 Receiving Recovery Notifications off of Warnings
Posted: Tue Jul 17, 2018 3:45 pm
by lmiltchev
The Warning notifications are intentional, but the ones labeled 'Custom: notify-service-by-pager' are only set for alerting from critical and ok states.
In my opinion, this is exactly what is happening (and what it should happen)... You service recovered and your contactgroup received an escalated recovery notification as per your config.
escalation_options c,r,
r = escalate on an OK (recovery) state,
Am I missing the point?
Re: 5.5.0 Receiving Recovery Notifications off of Warnings
Posted: Wed Jul 18, 2018 7:55 am
by Mortus
Prior to upgrade, we would only receive the recovery notification if the service recovered from a critical state using the escalation feature. This was a preferable case because we use the escalation for our on-call. So receiving pages in the middle of the night for recoveries of objects that weren't critical to begin with isn't ideal.
Re: 5.5.0 Receiving Recovery Notifications off of Warnings
Posted: Wed Jul 18, 2018 10:30 am
by lmiltchev
Prior to upgrade, we would only receive the recovery notification if the service recovered from a critical state using the escalation feature.
This sounds strange - we will need to do some more digging into it. What was the Nagios XI version that you upgraded from?