Receiving notifications for soft states being labeled hard

[GldRush98]

Ok, this has been happening since the XI 5.5 release I feel and we are still seeing this happen today. It's starting to really bug me.

Scenario: I receive an email notification for a service recovery. I never receive a notification for a service problem first, just a random recovery message.
So, I go tot he Service state history in XI and find that the service went down for a single check, then came back up the next minute. However XI is incorrectly labeling the State as HARD. It should be a SOFT state... for both and not sending notifications, yet here I am getting recovery notifications. For example, I got a Recovery notification for this event that happened for one minute. This should not happen:

Capture.PNG

This same scenario is occurring across several different servers and I never saw this happen before 5.5. Have you heard of other servers behaving this way? I can't be the only one seeing this weird notification behavior...

[GldRush98]

A couple more examples of this happening. These 2 examples are from an entirely different XI server (5.5.6) than the first example above (5.5.4).
As you can see though it is the same behavior. The service notification history matches what notifications I'm receiving. Nothing for down, only a recovery notification (but remember, these are SOFT states so I should be getting nothing).

Capture2.PNG

Capture3.PNG

[tgriep]

The issues that you are describing for the recovery notification is a known bug from the newer version of Nagios Core that was released with XI 5.5.x.
Those bugs are currently getting looked at and should be fixed in the next release of XI.
We do not have en exact date for when XI is going to be released at this time.

One question, when the service went down, was the Host check failing as well?

[GldRush98]

Ok, nice to have confirmation on the notification issue...

I just looked in to it and yes, the host shows as unreachable. It looks like the host check is showing as SOFT like it should. Would the host being down SOFT make the child services show down HARD? I didn't think that was typical behavior. A service going down should be SOFT until it reaches the Attempt limit, regardless of the host check, right?

[tgriep]

Your question "Would the host being down SOFT make the child services show down HARD?" is yes, When the service check first goes down, it checks the status of the host and if it see that the host is in a non-ok state, it puts the service on a hard down state right away.

If you look at this link under the hard state section, it describes that.
https://assets.nagios.com/downloads/nag ... types.html

The 4.4.x version of Nagios Core were re-written to match the documentation so that is why you are seeing some differences as the older versions did not match the documentation.

[GldRush98]

OK, yup, this would describe that situation:

When a service check results in a non-OK state and its corresponding host is either DOWN or UNREACHABLE.

Since this is different from the behavior I've known for many years, I thought that this was incorrect, but according to the docs this is functioning correctly. So then my only issue is the recovery notification being sent and that's a known bug right now.
Thanks, this helps clarify things for me!

[tgriep]

Your welcome.