Nagios Xi Escalations

[MOHANREDDY]

HI,
we have escalation setup for a service, its sending notification at correct time but not sending Notifications to correct contacts.
The setup is as follows:
Service Notification:
Check Interval: 5mins
Retry Interval: 1 min
Retry attempts: 3
Notification Delay: 30mins
Contacts: contact Admins

Escalation Levels:
Escalation level 1: 4 to 6
Notification interval: 15 mins
Contacts: contact A

Escalation level 2: 7 to 9
Notification interval: 30 mins
Contacts: contact B

Escalation level 3: 10 to 12
Notification interval: 40 mins
Contacts: contact C

Escalation level 1: 13 to 0
Notification interval: 60 mins
Contacts: contact D

The Notifications were received at the following time

Time Contacts Notification #
4:00pm Admins 1(initial alert)
4:35pm 34 mins contact A 2
5:10pm 30 mins Contact A 3
5:30pm 20 mins Contact A 4
5:49pm 20 mins Contact A 5
6:09pm 20mins Contact B 6
6:44pm 35mins Contact B 7
7:19pm 35mins Contact B 8
7:54pm 35mins Contact C 9
8:39pm 45mins Contact C 10
9:23pm 45mins Contact C 11
10:08pm 45mins Contact D 12
11:13pm 65mins Contact D 13
12:18am 65mins Contact D 14
1:22am 65mins Contact D 15
2:27am 65mins Contact D 16
3:32am 65mins Contact D 17
4:36am 65mins Contact D 18


Nagios XI is configured to send level 2 escalation from 7 to 9 to Contact B but from 6 - 8 alert is sent to Contact B.
Nagios XI is configured to send level 3 escalation from 10 to 12 to Contact B but from 9 - 11 alert is sent to Contact C.
Nagios XI is configured to send level 4 escalation from 13 to 0 to Contact B but from 12 - 8 alert is sent to Contact D.

Am i missing something in calculating the notification number while alerting?

Thanks in advance.

[ssax]

There were some bugs related to the notification count and escalations in early XI 5.5, what version are you currently running? You can find it in the bottom left hand side of the web interface.

Please PM me a copy of your profile, you can download it from Admin > System Profile > Download Profile.
- Include the hostname/servicename you were testing with so that I can view the configuration

[MOHANREDDY]

Currently we are using Nagios 5.5.6, i sent you the system profile in PM, please have a look.

Thanks,

[ssax]

That version is impacted, please upgrade to the latest version of XI:

https://support.nagios.com/kb/article.php?id=134

Then when testing, submit an OK result first, that will reset the counter (part of the bug), then test manually:

How to submit passive results for testing:

For Hosts
------------

Go to Home > Service Status:
- Find the Service and click on it
- Click the + tab
- Click the "Submit passive check result" link
- Select the Check Result and type in some text for the Check Output
- Click the Submit button

NOTE: By default, passive_host_checks_are_soft=0 is set in your /usr/local/nagios/etc/nagios.cfg, this differs from services in that when you submit a passive host check result it immeditely goes into a HARD state (and should send a notification if configured to) instead of going into a SOFT state like services do. (notifications are only sent on HARD states).

For Services
----------------

Go to Home > Service Status:
- Find the Service and click on it
- Click the + tab
- Note these two rows:

State Type: Hard
Current Check: 1 of 4

Those columns tell the current State Type and the Current Check number. In order to generate a notification for a service you will need to submit MULTIPLE problem check results (the number that you need to submit is determined by the last number in the Current Check column, that is the max_check_attempts setting). For services, when you submit a passive check result, each result that you submit will be a SOFT state until you submit enough to hit the Max Check Attempts setting that you've defined on the service, only then will the service enter a HARD problem state which will generate the notification (just remember, notifications are only sent on HARD states).

- Click the "Submit passive check result" link
- Select the Check Result and type in some text for the Check Output
- Click the Submit button
- Submit as many as you need, right after another, until the service enters the HARD state so that a notification will be sent

NOTE: When coming from a HARD problem state (whether we are talking about hosts or services) if you submit an OK passive result it should fire off a recovery notification after a single passive result has been submitted.

[MOHANREDDY]

I have upgraded Nagios 5.5.6 to Nagios 5.5.9, I have created escalation on a different service and with the following setup.

Service Notification:
Check Interval: 15 mins
Retry Interval: 1 min
Retry attempts: 1
Notification Delay: 15 mins
Contacts: contact Admins

Escalation Levels:
Escalation level 1: 3 to 3
Notification interval: 10 mins
Contacts: contact A

Escalation level 2: 4 to 4
Notification interval: 10 mins
Contacts: contact B

Escalation level 3: 5 to 5
Notification interval: 15 mins
Contacts: contact C

Escalation level 1:6 to 0
Notification interval: 15 mins
Contacts: contact D

The Notifications were received at the following time

Time Contacts Notification #
11:14 am Admins 1(initial alert)
11:44 am Admins 2
12:14 pm Contact A 3
12:29 pm Contact B 4
12:44 pm Contact C 5
1:14 pm Contact D 6
1:44 pm Contact D 7
2:14 pm Contact D 8

I am supposed to receive 2nd Notification at 11:30 am and also, i received 3rd, 6, 7, 8 notification at wrong interval of 30 mins which i didn't configure. Am i still miss calculating or any issue with software.

Am more confused than Before.

Thanks,

[ssax]

Please PM me a fresh copy of your profile and let me know the new hostname/servicename so that I can look at the proper configs.

Additionally, please send me these reports for this host so that I can compare them:

Reports > State History
- Make sure to limit by timeframe and host (don't limit by service) - select Type: Both, State Type: Both, State: Any

Reports > Notifications
- Make sure to limit the timeframe and host

[MOHANREDDY]

Please see the attached profile, reports in Pm.

Thanks,

[ssax]

The data from your reports and your configuration is not lining up, you must have changed the escalations at some point because they don't match up.

Please submit an OK result on the service you are testing with first, then submit enough results to go through the escalations.

How to submit passive results for testing:

For Hosts
------------

Go to Home > Service Status:
- Find the Service and click on it
- Click the + tab
- Click the "Submit passive check result" link
- Select the Check Result and type in some text for the Check Output
- Click the Submit button

NOTE: By default, passive_host_checks_are_soft=0 is set in your /usr/local/nagios/etc/nagios.cfg, this differs from services in that when you submit a passive host check result it immeditely goes into a HARD state (and should send a notification if configured to) instead of going into a SOFT state like services do. (notifications are only sent on HARD states).

For Services
----------------

Go to Home > Service Status:
- Find the Service and click on it
- Click the + tab
- Note these two rows:

State Type: Hard
Current Check: 1 of 4

Those columns tell the current State Type and the Current Check number. In order to generate a notification for a service you will need to submit MULTIPLE problem check results (the number that you need to submit is determined by the last number in the Current Check column, that is the max_check_attempts setting). For services, when you submit a passive check result, each result that you submit will be a SOFT state until you submit enough to hit the Max Check Attempts setting that you've defined on the service, only then will the service enter a HARD problem state which will generate the notification (just remember, notifications are only sent on HARD states).

- Click the "Submit passive check result" link
- Select the Check Result and type in some text for the Check Output
- Click the Submit button
- Submit as many as you need, right after another, until the service enters the HARD state so that a notification will be sent

NOTE: When coming from a HARD problem state (whether we are talking about hosts or services) if you submit an OK passive result it should fire off a recovery notification after a single passive result has been submitted.

After you've done that, please PM me a FRESH copy of your profile and let me know the new hostname/servicename so that I can look at the proper configs.

Additionally, please send me these reports for this host so that I can compare them:

Reports > State History
- Make sure to limit by timeframe and host (don't limit by service) - select Type: Both, State Type: Both, State: Any

Reports > Notifications
- Make sure to limit the timeframe and host

[MOHANREDDY]

see the details i sent in pm.

Thanks,

[ssax]

I've PM'ed you.