XI2014R1.4 - Notifications problem

[srrhd]

Hi all,

Since we have upgraded our 2 XI servers in 2014 version, we got a problem on the notifications. When a notification has to be send to a contactgroup, only the first contact is being notified but as many times as there are contacts in the group.
Let's says ADAM, EVE, HARRY, SAM and ZACH are in the contactgroup SUPPORT, when a notification is sent to SUPPORT, only ADAM receives it but he gets 5. If i remove HARRY, SAM and ZACH, then ADAM gets 2 notifications.
We are using nagios_send_host_mail.pl and nagios_send_service_mail.pl which require enable_environment_macros set to 1 in nagios.cfg. I know that there was a bug with this in Nagios Core 4.0.0 but it is said solved since 4.0.3.

Just in case: each contact has its own email adress, and the maillog confirms that all the notifications are send to only one contact. I also tryied to delete and reconfigure the contacts with no results.

Do you have any clue on this weird behavior ? I can't find out if it's an XI bug or if i am missing something obvious.

Thanks in advance for your support.

[slansing]

Do you have any escalations applied to this host, or it's services that you are testing with? Also, can you attach those custom notification scripts? Also, any chance you can share the configuration file of one of these services? Configure > CCM > Services > "service name" click the blue diskette icon next to it's name and post the output here. Also, are you set up to use sendmail, or smtp? Thanks!

[srrhd]

We have no escalations. It doesn't concerned just one or several services, but all hosts and services on our two XI servers.

Please find below a config file:

###############################################################################
#
# Service configuration file
#
# Created by: Nagios Core Config Manager 2.2.0
# Date: 2014-09-17 08:51:38
# Version: Nagios 3.x config file
#
# --- DO NOT EDIT THIS FILE BY HAND ---
# Nagios CCM will overwrite all manual settings during the next update if you
# would like to edit files manually, place them in the 'static' directory or
# import your configs into the CCM by placing them in the 'import' directory.
#
###############################################################################

define service {
host_name ALVARISTAR
service_description CPU Usage
use xiwizard_windowsserver_nsclient_service
check_command check_xi_service_nsclient!NagiosXI!CPULOAD!-l 5,80,90
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contact_groups Astreinte DATA
_xiwizard windowsserver
register 1
}

define service {
host_name ALVARISTAR
service_description Drive C: Disk Usage
use xiwizard_windowsserver_nsclient_service
check_command check_xi_service_nsclient!NagiosXI!USEDDISKSPACE!-l C -w 90 -c 95
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
first_notification_delay 0
notification_period xi_timeperiod_24x7
notification_options w,u,c,r,f,s
notifications_enabled 1
contact_groups Astreinte DATA
_xiwizard windowsserver
register 1
}

define service {
host_name ALVARISTAR
service_description Memory Usage
use xiwizard_windowsserver_nsclient_service
check_command check_xi_service_nsclient!NagiosXI!MEMUSE!-w 82 -c 85
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
first_notification_delay 0
notification_period xi_timeperiod_24x7
notification_options w,u,c,r,f,s
notifications_enabled 1
contacts null
contact_groups Astreinte DATA
_xiwizard windowsserver
register 1
}

define service {
host_name ALVARISTAR
service_description Ping
use xiwizard_windowsserver_ping_service
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contact_groups Astreinte DATA
_xiwizard windowsserver
register 1
}

define service {
host_name ALVARISTAR
service_description Uptime
use xiwizard_windowsserver_nsclient_service
check_command check_xi_service_nsclient!NagiosXI!UPTIME
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contact_groups Astreinte DATA
_xiwizard windowsserver
register 1
}

###############################################################################
#
# Service configuration file
#
# END OF FILE
#
###############################################################################

As mail agent, we use postfix.

Find attached the two notifications scripts and if you want more informations on them, they are Frank4DD's one, you can find them at :
http://nagios.frank4dd.com/howto/nagios ... ations.htm

But my guess that the scripts works as they use to work all NagiosXI release until the 2014 one and as they work for notifying the first contact of the contactgroup. It seems to be the use of a variable that retains the first value. In addition the notifications commands are defined as below:

define command {
command_name notify-host-by-email-HTML-GRAPH
command_line $USER1$/nagios_send_host_mail.pl -f graph -u -p "Supervision SRRHD" -l fr
}

define command {
command_name notify-service-by-email_HTML-GRAPH
command_line $USER1$/nagios_send_service_mail.pl -f graph -u -p "Supervision SRRHD" -l fr
}

Regards,

Aurélien.

[Box293]

Can you also post the definition for the contact_groups "Astreinte DATA".

[srrhd]

Below the definitions of the contactgroup, the first contact and the last contact

define contactgroup {
contactgroup_name Astreinte DATA
alias Astreinte DATA
members AS_Mail,NBL_mail,Portable_Astreinte,SAM_Mail,SRI_Mail
}

define contact {
contact_name AS_Mail
alias Xxxxxxxx XXXXXXXXX
host_notifications_enabled 1
service_notifications_enabled 1
host_notification_period 24x7
service_notification_period 24x7
host_notification_options d,u,r,s,
service_notification_options w,u,c,r,s,
host_notification_commands notify-host-by-email-HTML-GRAPH
service_notification_commands notify-service-by-email_HTML-GRAPH
email [email protected]
host_notifications_enabled 1
service_notifications_enabled 1
}

define contact {
contact_name SRI_Mail
alias Yyyyyyyyyyyy YYYYYYYY
host_notifications_enabled 1
service_notifications_enabled 1
host_notification_period 24x7
service_notification_period 24x7
host_notification_options d,u,r,s,
service_notification_options w,u,c,r,s,
host_notification_commands notify-host-by-email-HTML-GRAPH
service_notification_commands notify-service-by-email_HTML-GRAPH
email [email protected]
host_notifications_enabled 1
service_notifications_enabled 1
}

[Box293]

The only thing I can identify in the contacts you've posted is that host_notifications_enabled and service_notifications_enabled are defined twice in each contact.

Seeing as you are using Frank4DD's notification scripts, I think it would be worthwhile if you spun up a Nagios XI VM from the Nagios Library. It shouldn't take too long to configure it with a couple of contacts and test these scripts on a different system.

My thinking behind this is that IF there is some sort of bug with XI 2014 and Frank4DD's notification scripts, they will show up here. Otherwise we know it's something specific to your system.

[srrhd]

Sorry for the delay i was on a training last week.

I modified the contacts configuration so that the variables aren't duplicated but the problem persist. I guess the update as force the notification variable to ON instead of SKIP as those variables are defined in the misc settings to have them as environment variables.

We already have two systems that are impacted, one physical server and one VM. Both get the problem appeared with the 2014 update. Unfortunately our team is reduced and will be until the end of the year. I am affraid we won't have time to set up a test server before a while.
If you can do the test on your side we would greatly appreciate it.

[abrist]

Sorry for the delay i was on a training last week.

No problem.

I modified the contacts configuration so that the variables aren't duplicated but the problem persist. I guess the update as force the notification variable to ON instead of SKIP as those variables are defined in the misc settings to have them as environment variables.

Those free variables get set when you create a user as a monitoring contact. If you do not have notifications handlers set, or have disabled notifications, this should not effect you.

If you can do the test on your side we would greatly appreciate it.

We can test many things, but in general, custom notification scripts must be tested by the client in their environment. When you get time to set up the necessary testing env, we will gladly help you by testing through a remote session.

[srrhd]

I finally get time to set up a test server, and i didn't succeed to reproduce the problem. There must be a problem on our server or configuration. You can close this thread.