Page 1 of 1

Unwanted Notifications (because of Inheritance)?

Posted: Tue Oct 23, 2018 8:37 am
by wschwarz
Problem:
This host appears to be using the contact group "call-everyone" when it should be using the contact group "email-network".

Background:
A year ago everything that went down triggered a call to everyone. To make that better I attempted to create different contact groups for different team members. I created the switch_host_noncritical group which is for switches that should not call, only email. I also created the email-network contact group. Yesterday the whole team was called in the middle of the night because this switch was down. I then added "contact_groups email-network" to the host definition (it was previously only in the switch_host_noncritical definition) and restarted nagios but the switch went down and triggered a call to everyone again last night. What am I missing?

Relevant Definition:
define host{
use switch_host_noncritical
host_name buildingASA
alias BuildingASA
address 10.87.104.1
hostgroups switches
notes Network
contact_groups email-network
parents BigRouter
notification_period daytime
}

define host {
name switch_host_noncritical
max_check_attempts 6
check_command check-host-alive
passive_checks_enabled 1
check_period 24x7
obsess_over_host 1
check_freshness 0
event_handler_enabled 1
flap_detection_enabled 1
process_perf_data 0
retain_status_information 1
retain_nonstatus_information 1
contact_groups email-network
notification_interval 0
notification_period 24x7
notification_options d,r,f
notifications_enabled 1
icon_image switch40.gif
statusmap_image switch40.gd2
register 0
}

define contactgroup{
contactgroup_name call-everyone
alias IT
members systemsguy1,systemsguy2,bossguy1,applicationsguy1,applicationsguy2,helpdeskguy1,ims4k
}

define contactgroup{
contactgroup_name email-network
alias Email-Network
members networkguy1,networkguy2
}

define contactgroup{
contactgroup_name call-network
alias Call-Network
members networkguy1,networkguy2,ims4k_network
}

define contact{
contact_name ims4k
alias IMS 4000
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,c,r,f
host_notification_options d,r,u,f
service_notification_commands notify-service-by-ims
host_notification_commands notify-host-by-ims
}

define contact{
contact_name ims4k_network
alias IMS 4000 Network
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,c,r,f
host_notification_options d,r,u,f
service_notification_commands notify-service-by-ims-network
host_notification_commands notify-host-by-ims-network
}

define contact{
contact_name networkguy1
alias Network Guy
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,c,r,f
host_notification_options d,r,u,f
service_notification_commands notify-service-by-nagiosEmail
host_notification_commands notify-host-by-nagiosEmail
email [email protected]
}

define command{
command_name notify-host-by-ims
command_line /usr/local/ims/ims_notify.sh 1 $NOTIFICATIONTYPE$ $HOSTSTATE$
}

define command{
command_name notify-service-by-ims
command_line /usr/local/ims/ims_notify.sh 1 $NOTIFICATIONTYPE$ $SERVICESTATE$
}

define command{
command_name notify-host-by-ims-network
command_line /usr/local/ims/ims_notify.sh 4 $NOTIFICATIONTYPE$ $HOSTSTATE$
}

define command{
command_name notify-service-by-ims-network
command_line /usr/local/ims/ims_notify.sh 4 $NOTIFICATIONTYPE$ $SERVICESTATE$
}

define command{
command_name notify-service-by-nagiosEmail
command_line /usr/local/nagiosEmail/nagiosEmail.pl "$CONTACTEMAIL$" [email protected] "$SERVICESTATE$" "$HOSTALIAS$" "$HOSTADDRESS$" "$NOTIFICATIONTYPE$" "$LONGDATETIME$" "$HOSTNOTES$" "$SERVICEOUTPUT$" "$SERVICEDESC$"
}

define command{
command_name notify-host-by-nagiosEmail
command_line /usr/local/nagiosEmail/nagiosEmail.pl "$CONTACTEMAIL$" [email protected] "$HOSTSTATE$" "$HOSTALIAS$" "$HOSTADDRESS$" "$NOTIFICATIONTYPE$" "$LONGDATETIME$" "$HOSTNOTES$" "$HOSTOUTPUT$"
}

Re: Unwanted Notifications (because of Inheritance)?

Posted: Tue Oct 23, 2018 4:08 pm
by tgriep
I would look at the Service Checks for that host and verify that they do not have the Contact Group set to them as that could be where the Notifications are coming from.

Re: Unwanted Notifications (because of Inheritance)?

Posted: Wed Oct 24, 2018 10:15 am
by wschwarz
Aha, that's it. Thanks for the assist!

Here are the relevant service definitions in case others are affected and would benefit from more information:

define service{
use generic-service
host_name !controller7
hostgroup_name switches
service_description Uptime
is_volatile 1
check_command check_switch_snmp!sysUpTime.0 -t 600 -c 10: -w 60000:
#contact_groups call-everyone
active_checks_enabled 1
passive_checks_enabled 1
check_interval 5
retry_check_interval 1
active_checks_enabled 1
passive_checks_enabled 1
freshness_threshold 900
notification_interval 0
}

define service{
name generic-service ; The 'name' of this service template
active_checks_enabled 1 ; Active service checks are enabled
passive_checks_enabled 1 ; Passive service checks are enabled/accepted
parallelize_check 1 ; Active service checks should be parallelized (disabling this can lead to major performance problems)
obsess_over_service 1 ; We should obsess over this service (if necessary)
check_freshness 0 ; Default is to NOT check service 'freshness'
notifications_enabled 1 ; Service notifications are enabled
event_handler_enabled 1 ; Service event handler is enabled
flap_detection_enabled 1 ; Flap detection is enabled
process_perf_data 1 ; Process performance data
retain_status_information 1 ; Retain status information across program restarts
retain_nonstatus_information 1 ; Retain non-status information across program restarts
is_volatile 0 ; The service is not volatile
check_period 24x7 ; The service can be checked at any time of the day
max_check_attempts 3 ; Re-check the service up to 3 times in order to determine its final (hard) state
check_interval 10 ; Check the service every 10 minutes under normal conditions
retry_interval 2 ; Re-check the service every two minutes until a hard state can be determined
contact_groups call-everyone ; Notifications get sent out to everyone in the 'admins' group
notification_options w,u,c,r ; Send notifications about warning, unknown, critical, and recovery events
notification_interval 60 ; Re-notify about service problems every hour
notification_period 24x7 ; Notifications can be sent out at any time
register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL SERVICE, JUST A TEMPLATE!
}

The fix for me is to specify "contact_groups email-network" in the Uptime service definition.

Re: Unwanted Notifications (because of Inheritance)?

Posted: Wed Oct 24, 2018 3:59 pm
by tgriep
Your Welcome. Glad to help. I'll close and lock the post as solved but feel free to open a new post in the future if needed.