Critical notifications not being sent from XI

[cdienger]

The graph shows the line is right on or maybe a little bit below the CRITICAL threshold. Given that the service isn't configured to send notifications on WARNINGS, this could explain why they didn't get anything. Check the state of this service during that time frame with a State History report under Reports > Available Reports.

[rferebee]

Is this what you're looking for?

[cdienger]

That the one. And it seems to confirm what we saw in the logs - the service was in a WARNING state on the 11th and the service isn't configured to send notifications on WARNING.

[rferebee]

Yes, that I totally agree with, but from May 1st to May 11th the service was Critical and it only sent out one notification even though it's configured to send out one every 1440 minutes to ServerSupportContact group.

That's the issue we need to figure out.

[cdienger]

I think the confusion came in with the introduction of the second issue of the config being overwritten - I got the impression that the behavior prior to the 11th was due to the configs being overwritten by the other machine's config. To make sure we're on the same page now, this is what I see in the data we've collected:

1. Notifications were not sent out between the 11th and 21st because the service was in a WARNING state and the service is not configured to send out notifications for this state.
2. Notifications were not sent out to the ServerSupportContact group on 8th, 9th, and 10th because the service was escalated and notifications were sent to a different group.
3. A notification was sent to the ServerSupportContact group on the 1st but never again.

It is the third point that we are investigating at this point.

[rferebee]

Yes, to #3!

In regards to #2, you're saying that when a service check is escalated the notifications are only sent to the NagiosContact contact group?

[cdienger]

Yes, in an escalation only the NagiosContact group is sent notifications.

https://assets.nagios.com/downloads/nag ... ations.pdf:

Escalations will overwrite ANY notification options on the host or service object when they take affect (this includes the contacts/contactgroups).

To look further into #3, let's get a copy of the audit log capturing the month of May. It can be downloaded under Admin > System Information > Audit Log.

[cdienger]

The audit log looks good...

As a test - Are you able to make a copy of the service, but modify the thresholds so it triggers right away? I'd like to see if it demonstrates the same behavior and logs. If we're able to reproduce it then we can take a closer look in a ticket.

[rferebee]

Do you only want the notification to trigger once? Or, are we also testing the subsequent follow-up alerts?

I just copied the service check and had it trigger immediately by adjusting the Critical threshold. It fired off the notification like it should have to the assigned Contact Groups.

[cdienger]

We'll want to allow it to run a bit to see if it send follow up emails as expected.