Notifications not being sent to all members of Contact Group

[crnelson]

System Info
Linux Distribution and version?
Distributor ID: CentOS
Release: 5.10
Linux 2.6.18-371.4.1.el5PAE #1 SMP Thu Jan 30 06:51:58 EST 2014 i686 i686 i386 GNU/Linux

32 or 64bit?
32 bit

VMware Image or Manual Install of XI?
VMware Image

Are there specials configurations on your system, ie; is Gnome installed? Are you using a proxy? Are you using SSL?
Yes, using SSL.


ISSUE
A monitored server had a HDD go critical this morning and notifications were sent; however, they were not sent to all contacts.
Facts:
- This particular monitor (Drive G) is applied to a Host Group.
- This monitor is setup to notify a Contact Group.
- All members of said Contact Group were notified, except the one for PagerDuty. This was viewable under Notifications.

Testing:
- Triggering other Drive G notifications for hosts in the same Host Group notifies ALL members of said Contact Group (verified in Notifications).
- Removed the host completely and recreated it.
- Triggered another notification of new host, but no Notifications was shown for the PagerDuty contact. All other contacts were shown in Notifications.
- Validated once more that the PagerDuty contact was a part of the applicable Contact Group.

[slansing]

Can you share the configurations for your host, an associated service you were expecting alerts for, and the contacts which did not receive notifications? "Please blank our their addresses"

You can view your configurations via accessing the CCM and clicking the blue diskette icon next the the configuration's table in the list.

[crnelson]

Selection_001.png

Selection_002.png

Selection_003.png

[crnelson]

Selection_004.png

Selection_005.png

Selection_006.png

[crnelson]

Selection_007.png

As previously mentioned, the contact and contact group works with other hosts, just not "this" one.

[Box293]

Could you also see if there were any entries in /var/log/messages for when the problem occurred (both the original problem and the tests). Paste them here in a code block please.

[crnelson]

Sorry for the slow response. Here's an excerpt from the log during the original failure that prompted my investigation.

Code: Select all

Jun 25 05:21:01 fwapp003 xinetd[3580]: START: nrpe pid=443 from=*.*.*.24
Jun 25 05:21:01 fwapp003 xinetd[3580]: EXIT: nrpe status=0 pid=443 duration=0(sec)
Jun 25 05:21:02 fwapp003 xinetd[3580]: START: nrpe pid=543 from=*.*.*.24
Jun 25 05:21:02 fwapp003 xinetd[3580]: START: nrpe pid=544 from=*.*.*.24
Jun 25 05:21:02 fwapp003 xinetd[3580]: START: nrpe pid=545 from=*.*.*.24
Jun 25 05:21:02 fwapp003 xinetd[3580]: START: nrpe pid=546 from=*.*.*.24
Jun 25 05:21:02 fwapp003 xinetd[3580]: START: nrpe pid=547 from=*.*.*.24
Jun 25 05:21:02 fwapp003 xinetd[3580]: EXIT: nrpe status=0 pid=547 duration=0(sec)
Jun 25 05:21:02 fwapp003 xinetd[3580]: EXIT: nrpe status=0 pid=545 duration=0(sec)
Jun 25 05:21:02 fwapp003 xinetd[3580]: EXIT: nrpe status=0 pid=546 duration=0(sec)
Jun 25 05:21:02 fwapp003 xinetd[3580]: EXIT: nrpe status=0 pid=543 duration=0(sec)
Jun 25 05:21:02 fwapp003 xinetd[3580]: EXIT: nrpe status=0 pid=544 duration=0(sec)
Jun 25 05:21:03 fwapp003 nagios: SERVICE ALERT: *201;Apache Web Server;CRITICAL;SOFT;2;[25.06.2014 05:21:00 SYSTEM] watchdog for monitor is not running. 
Jun 25 05:21:03 fwapp003 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_statehistory SET instance_id='1', state_time=FROM_UNIXTIME(1403698863), state_time_usec='284697', object_id='433', state_change
='1', state='2', state_type='0', current_check_attempt='2', max_check_attempts='5', last_state='2', last_hard_state='0', output='\[25\.06\.2014 05:21:00 SYSTEM\] watchdog for monitor is not running\.', long_outpu
t='\[25\.06\.2014 05:21:00 SYSTEM\] watchdog for monitor is not running\.'' 
Jun 25 05:21:03 fwapp003 ndo2db: mysql_error: 'Table './nagios/nagios_statehistory' is marked as crashed and last (automatic?) repair failed' 
Jun 25 05:21:12 fwapp003 nagios: SERVICE NOTIFICATION: stb*;*008;Drive_G: Disk Usage;CRITICAL;notify-service-by-email;G:\ - total: 75.00 Gb - used: 73.77 Gb (98%) - free 1.23 Gb (2%) 
Jun 25 05:21:12 fwapp003 nagios: SERVICE NOTIFICATION: rmc*;*008;Drive_G: Disk Usage;CRITICAL;notify-service-by-email;G:\ - total: 75.00 Gb - used: 73.77 Gb (98%) - free 1.23 Gb (2%) 
Jun 25 05:21:12 fwapp003 nagios: SERVICE NOTIFICATION: mpr*;*008;Drive_G: Disk Usage;CRITICAL;notify-service-by-email;G:\ - total: 75.00 Gb - used: 73.77 Gb (98%) - free 1.23 Gb (2%) 
Jun 25 05:21:12 fwapp003 nagios: SERVICE NOTIFICATION: eca*;*008;Drive_G: Disk Usage;CRITICAL;notify-service-by-email;G:\ - total: 75.00 Gb - used: 73.77 Gb (98%) - free 1.23 Gb (2%) 
Jun 25 05:21:12 fwapp003 xinetd[3580]: START: nrpe pid=1285 from=*.*.*.10
Jun 25 05:21:12 fwapp003 xinetd[3580]: EXIT: nrpe status=0 pid=1285 duration=0(sec)
Jun 25 05:21:22 fwapp003 xinetd[3580]: START: nrpe pid=1578 from=*.*.*.24
Jun 25 05:21:22 fwapp003 xinetd[3580]: EXIT: nrpe status=0 pid=1578 duration=0(sec)
Jun 25 05:21:44 fwapp003 xinetd[3580]: START: nrpe pid=2631 from=*.*.*.10
Jun 25 05:21:44 fwapp003 xinetd[3580]: EXIT: nrpe status=0 pid=2631 duration=0(sec)
Jun 25 05:21:59 fwapp003 xinetd[3580]: START: nrpe pid=3126 from=*.*.*.24

[crnelson]

Here's an excerpt from a test against another host

The following line is expected because I sent an OK notification. What I wanted to see was the contact trigger, which it did.
"Jun 25 12:15:34 fwapp003 pagerduty_nagios[626]: Nagios event in file /tmp/pagerduty_nagios/pd_1403723733_626.txt REJECTED by the PagerDuty server. Server says: The NOTIFICATIONTYPE field must be present and must be one of: PROBLEM, ACKNOWLEDGEMENT, RECOVERY, NOP. "

Full log with abstracted hostnames, user names, and IPs

Code: Select all

Jun 25 12:15:32 fwapp003 nagios: SERVICE NOTIFICATION: stb*;*011;Drive_G: Disk Usage;CUSTOM (OK);notify-service-by-wv_alerts-with-additional-addresses;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:32 fwapp003 nagios: SERVICE NOTIFICATION: stb*;*011;Drive_G: Disk Usage;CUSTOM (OK);notify-service-by-email;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:32 fwapp003 nagios: SERVICE NOTIFICATION: stb*;*011;Drive_G: Disk Usage;CUSTOM (OK);notify-service-by-email;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:32 fwapp003 nagios: SERVICE NOTIFICATION: st*;*011;Drive_G: Disk Usage;CUSTOM (OK);xi_service_notification_handler;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:33 fwapp003 nagios: SERVICE NOTIFICATION: rmc*;*011;Drive_G: Disk Usage;CUSTOM (OK);notify-service-by-wv_alerts-with-additional-addresses;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:33 fwapp003 nagios: SERVICE NOTIFICATION: rmc*;*011;Drive_G: Disk Usage;CUSTOM (OK);notify-service-by-email;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:33 fwapp003 nagios: SERVICE NOTIFICATION: rmc*;*011;Drive_G: Disk Usage;CUSTOM (OK);notify-service-by-email;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:33 fwapp003 nagios: SERVICE NOTIFICATION: rmc*;*011;Drive_G: Disk Usage;CUSTOM (OK);xi_service_notification_handler;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:33 fwapp003 nagios: SERVICE NOTIFICATION: pagerduty_MSSQL_PROD;*011;Drive_G: Disk Usage;CUSTOM (OK);notify-service-by-pagerduty;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:34 fwapp003 pagerduty_nagios[626]: Nagios event in file /tmp/pagerduty_nagios/pd_1403723733_626.txt REJECTED by the PagerDuty server.  Server says: The NOTIFICATIONTYPE field must be present and must be one of: PROBLEM, ACKNOWLEDGEMENT, RECOVERY, NOP. 
Jun 25 12:15:34 fwapp003 nagios: SERVICE NOTIFICATION: mpr*;*011;Drive_G: Disk Usage;CUSTOM (OK);notify-service-by-wv_alerts-with-additional-addresses;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:34 fwapp003 nagios: SERVICE NOTIFICATION: mpr*;*011;Drive_G: Disk Usage;CUSTOM (OK);notify-service-by-email;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:34 fwapp003 nagios: SERVICE NOTIFICATION: mpr*;*011;Drive_G: Disk Usage;CUSTOM (OK);notify-service-by-email;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:34 fwapp003 nagios: SERVICE NOTIFICATION: mpr*;*011;Drive_G: Disk Usage;CUSTOM (OK);xi_service_notification_handler;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:34 fwapp003 nagios: SERVICE NOTIFICATION: eca*;*011;Drive_G: Disk Usage;CUSTOM (OK);notify-service-by-email;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY 
Jun 25 12:15:35 fwapp003 xinetd[3580]: START: nrpe pid=704 from=*.*.*.10
Jun 25 12:15:35 fwapp003 xinetd[3580]: EXIT: nrpe status=0 pid=704 duration=0(sec)

[Box293]

Code: Select all

Jun 25 12:15:33 fwapp003 nagios: SERVICE NOTIFICATION: pagerduty_MSSQL_PROD;*011;Drive_G: Disk Usage;CUSTOM (OK);notify-service-by-pagerduty;G:\ - total: 250.00 Gb - used: 19.69 Gb (8%) - free 230.31 Gb (92%);crn*;TEST NOTIFY

Jun 25 12:15:34 fwapp003 pagerduty_nagios[626]: Nagios event in file /tmp/pagerduty_nagios/pd_1403723733_626.txt REJECTED by the PagerDuty server.  Server says: The NOTIFICATIONTYPE field must be present and must be one of: PROBLEM, ACKNOWLEDGEMENT, RECOVERY, NOP. 

Thats what we are looking for. It looks like the PagerDuty server is rejecting the message being sent to it.

Check the command notify-service-by-pagerduty and ensure that it is correclty submitting the NOTIFICATIONTYPE field as one of PROBLEM, ACKNOWLEDGEMENT, RECOVERY, NOP.

[crnelson]

Code: Select all

Server says: The NOTIFICATIONTYPE field must be present and must be one of: PROBLEM, ACKNOWLEDGEMENT, RECOVERY, NOP.

This is due to it being a forced notification with the state of OK, which doesn't match any of the parameters PagerDuty is expecting.

The following line is expected because I sent an OK notification. What I wanted to see was the contact trigger, which it did.
"Jun 25 12:15:34 fwapp003 pagerduty_nagios[626]: Nagios event in file /tmp/pagerduty_nagios/pd_1403723733_626.txt REJECTED by the PagerDuty server. Server says: The NOTIFICATIONTYPE field must be present and must be one of: PROBLEM, ACKNOWLEDGEMENT, RECOVERY, NOP. "

Do you see in the log that the notification was NOT sent to PagerDuty when the service went critical? See post Thu Jun 26, 2014 2:29 pm. This is what I'm trying to get to the bottom of.