Notification e-mail not sending to all contactgroup members

[vauran]

Hey Guys,

Working on fixing our Nagios 3.4.1 alerts. Currently, when a host/service critical alert happens, only the first user in our contact group is sent an e-mail. Seeing if you guys can assist with this. I've attached the relevant information for a specific e-mail notification for hostA that *userA* got yesterday but userB and C (which includes me) didn't get.

From contactgroups.cfg:
define contactgroup {
contactgroup_name Email - Group IT Support
alias IT Support Group
members userA, userB, userC
}

From host.cfg:
define host {
host_name hostA
check_command check-host-alive
max_check_attempts 5
check_interval 10
passive_checks_enabled 1
check_period 24x7
obsess_over_host 1
check_freshness 1
freshness_threshold 0
active_checks_enabled 1
event_handler_enabled 1
flap_detection_enabled 1
process_perf_data 1
retain_status_information 1
retain_nonstatus_information 1
notification_interval 5
notification_period 24x7
notifications_enabled 1
failure_prediction_enabled 1
alias hostAliasA
display_name
address 192.168.8.13
notification_options d,u,r,f
stalking_options o,d
contact_groups Pagers,Email - Group IT Support
hostgroups hostGroupA
}

From contacts.cfg (all 3 are set up the same way)
define contact {
contact_name userA
alias userAliasA
email userAEmail
pager
host_notifications_enabled 0
service_notifications_enabled 1
host_notification_period 24x7
service_notification_period 24x7
can_submit_commands 0
retain_status_information 0
retain_nonstatus_information 0
service_notification_options w,c,r
host_notification_commands alert-by-email
service_notification_commands alert-by-email
}

From commands.cfg
define command {
command_name alert-by-email
command_line /usr/bin/printf "%b" "***** Nagios 2.9 *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n\n\nFollow-up Response:\n\n1) Briefly define Root cause\n2) Impact to IT services (as felt by our customer)\n3) Remedy to fix (Immediate/short term)\n4) Remedy to prevent a recurrence (future/long term)\n\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ alert - $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
}


We use sendmail and I am able to email myself (userC) and userB from the command line correctly. Let me know if you need more information.

[Box293]

This stands out in your contact definition:

host_notifications_enabled 0

So no notifications are send to this contact for host notifications.

Also, from /usr/local/nagios/var/objects.cache what are the definitions for the host and contacts ?

[vauran]

Hey,

This is weird because userA is the one who gets the notifications. Here is my config for userC (Me, I don't get anything):

define contact {
contact_name userC
alias userC
email userC@email
pager
host_notifications_enabled 1
service_notifications_enabled 1
host_notification_period 24x7
service_notification_period 24x7
can_submit_commands 0
retain_status_information 0
retain_nonstatus_information 0
service_notification_options w,c,r
host_notification_commands alert-by-email
service_notification_commands alert-by-email

}

From objects.cache
userA(who receives e-mails)
define contact {
contact_name userA
alias userA
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,c,r
host_notification_options n
service_notification_commands alert-by-email
host_notification_commands alert-by-email
email userA@email.com
pager
host_notifications_enabled 0
service_notifications_enabled 1
can_submit_commands 0
retain_status_information 0
retain_nonstatus_information 0
}

and userC(me, who doesn't receive e-mails):
define contact {
contact_name userC
alias userC
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,c,r
host_notification_options n
service_notification_commands alert-by-email
host_notification_commands alert-by-email
email userC@email
pager
host_notifications_enabled 1
service_notifications_enabled 1
can_submit_commands 0
retain_status_information 0
retain_nonstatus_information 0
}

I notice the host_notification_options is set to n (so none), why is that? It says this file is auto-generated.

Host from objects.cache

define host {
host_name hostA
display_name
alias hostA
address hostAIP
check_period 24x7
check_command check-host-alive
contact_groups Pagers,Email - Group IT Support
notification_period 24x7
initial_state o
check_interval 10.000000
retry_interval 1.000000
max_check_attempts 5
active_checks_enabled 1
passive_checks_enabled 1
obsess_over_host 1
event_handler_enabled 1
low_flap_threshold 0.000000
high_flap_threshold 0.000000
flap_detection_enabled 1
flap_detection_options o,d,u
freshness_threshold 0
check_freshness 1
notification_options d,u,r,f
notifications_enabled 1
notification_interval 5.000000
first_notification_delay 0.000000
stalking_options o,d
process_perf_data 1
failure_prediction_enabled 1
retain_status_information 1
retain_nonstatus_information 1
}

Let me know what you think

[tgriep]

Can you run a notification report and post the notification that was not sent out to everyone?
Can you check sendmail's log for any errors?
The log file location is

Code: Select all

/var/log/maillog

[vauran]

Thanks for the quick response. I'm not sure where the notification report is. When I look at Notifications under the web GUI it says "No notifications have been recorded in the current log file", with the file being /usr/local/nagios/var/nagios.log. When I check that log on the server, I just see SERVICE ALERTS and HOST ALERTS, no NOTIFICATIONS.

In regards to the maillog at /var/log/maillog, no errors. I can grep for userA and find where he's been e-mailed (few examples):

Mar 17 05:18:23 phxnagios sendmail[27395]: t2HCINM4027395: to=userA@email.com, ctladdr=nagios (500/500), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30532, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (t2HCINKU027396 Message accepted for delivery)
Mar 17 05:18:23 phxnagios sendmail[27398]: t2HCINKU027396: to=<userA@email.com>, ctladdr=<nagios@companyEmail> (500/500), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=120852, relay=smtp2.company.net. [192.168.8.31], dsn=2.0.0, stat=Sent (t2HCINFG012941 Message accepted for delivery)

There are no entries for userB or userC, though.

[tgriep]

When you run the Notification Report, on the top is an arrow pointing to the left that says "Latest Archive", click that and look for the notifications that were sent on March 17th.

[jdalrymple]

Best guess... this line in objects.cache is beating you up:

Code: Select all

host_notification_options n

Add to your contact definition:

Code: Select all

host_notification_options d,u,r,f

Best guess as to why UserA is getting *something* but you're not, UserA is getting a service notification, not a host notification. By all rights there is NO WAY he/she should get host notifications with those 2 lines:

Code: Select all

host_notification_options n
host_notifications_enabled 0

[vauran]

Hey,

Sorry, inexperience with this is showing. After looking through the set up again, this is definitely a service notification that he got, not a host. Here is the services.cfg for the notification he got:

define service {
host_name hostA
check_command Linux-nrpe_check_disk1
max_check_attempts 5
normal_check_interval 60
retry_interval 5
active_checks_enabled 1
passive_checks_enabled 1
check_period 24x7
parallelize_check 1
obsess_over_service 1
check_freshness 1
freshness_threshold 0
event_handler_enabled 1
flap_detection_enabled 1
process_perf_data 1
retain_status_information 1
retain_nonstatus_information 1
notification_interval 0
notification_period 24x7
notifications_enabled 1
failure_prediction_enabled 1
service_description DISK1
display_name
notification_options u,c,r,f
stalking_options o,w,u,c
contact_groups Email - Group IT Support
servicegroups DISKS
}

The e-mail that he got that I didn't:

***** Nagios 2.9 *****

Notification Type: PROBLEM

Service: DISK1
Host: hostA
Address: 192.168.8.13
State: CRITICAL

Date/Time: Tue Mar 17 05:18:23 PDT 2015

Additional Info:

DISK CRITICAL - free space: / 5644 MB (10% inode=96%):


Follow-up Response:

1) Briefly define Root cause
2) Impact to IT services (as felt by our customer)
3) Remedy to fix (Immediate/short term)
4) Remedy to prevent a recurrence (future/long term)

I have since modified contacts.cfg for myself, my new contacts.cfg setting for userC:
define contact {
contact_name userC
alias userC
email userC@email.com
pager
host_notifications_enabled 1
service_notifications_enabled 1
host_notification_period 24x7
service_notification_period 24x7
can_submit_commands 0
retain_status_information 0
retain_nonstatus_information 0
host_notification_options d,u,r
service_notification_options w,c,r
host_notification_commands alert-by-email
service_notification_commands alert-by-email

}

Objects.cache for userC (me)
define contact {
contact_name userC
alias userC
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,c,r
host_notification_options d,u,r
service_notification_commands alert-by-email
host_notification_commands alert-by-email
email userC@email.com
pager
host_notifications_enabled 1
service_notifications_enabled 1
can_submit_commands 0
retain_status_information 0
retain_nonstatus_information 0
}


objects.cache for the service
define service {
host_name hostA
service_description DISK1
display_name
check_period 24x7
check_command Linux-nrpe_check_disk1
contact_groups Email - Group IT Support
notification_period 24x7
initial_state o
check_interval 60.000000
retry_interval 5.000000
max_check_attempts 5
is_volatile 0
parallelize_check 1
active_checks_enabled 1
passive_checks_enabled 1
obsess_over_service 1
event_handler_enabled 1
low_flap_threshold 0.000000
high_flap_threshold 0.000000
flap_detection_enabled 1
flap_detection_options o,w,u,c
freshness_threshold 0
check_freshness 1
notification_options u,c,r,f
notifications_enabled 1
notification_interval 0.000000
first_notification_delay 0.000000
stalking_options o,u,w,c
process_perf_data 1
failure_prediction_enabled 1
retain_status_information 1
retain_nonstatus_information 1
}


So the e-mail he got and I didn't was a service notification, not a host. At the time my contact.cfg section did have service notifications enabled and w,c,r as the options. So the issue seems to be that userA gets service notifications and userB and C don't. Sorry for the confusion!

When I look at our nagios gui (nagios.domain.com) I choose Notifications from the left, under Reports. When the page comes up, I don't see an arrow pointing to the left.

[jdalrymple]

Did anything change from the time when it was broken (UserA got his/her mail but UserC did not receive mail) until now?

From what I can make out - the current configuration would work. Even if nothing changed I would still suggest testing the configuration at a minimum by submitting a custom service notification or preferably by simulating a CRITICAL state again.

[vauran]

Hey,

Nothing has changed for services information. I just now added debug_level=32 and specified my debug_file to see if that will give me anymore information. I am unfamiliar with submitting customer service notifications. Is there an easy way to simulate a critical state? The servers our nagios monitors are production servers so doing anything crazy with them would be a no-go.