Some email notifications not being sent

[umunix]

Hi Folks,

I was hoping you might be able to provide me with some advice with an issue I'm having with email notifications:

I have a service that sends out an alert to 3 contact groups. Most of the contacts in these groups use the XI notification handler. However, there is one non-XI contact in one of the groups that uses the notify-service-by-email notification handler.

Alerts sent to the one contact using the notify-service-by-email handler work fine -- However, most of the contacts using the XI handler seem to fail. It looks like the handler is called (the notification appears in XI and syslog), but I don't see matching messages in /var/log/maillog.

I'm a little puzzled because this problem only seems to exist with this one particular service. When I look at the configuration for this service against other working services, I can't find any obvious problems (although there very well could be one). I'm hoping for some thoughts on what else I might be able to check.

Kind regards,
Richard

[rkennedy]

Could you post here what the name of the host / service are, and PM over your /usr/local/nagios/var/objects.cache for us to review?

This should be what we need to help identify where the culprit is.

EDIT: objects received.

[umunix]

Sure thing. The name of the host is 'acrux' and the name of the service is 'HTTPS 9443 - Banner Integration for eProcurement'.

I'll PM you my objects.cache now.

[dwhitfield]

For the purposes of squashing bugs, what version of XI are you using?

[rkennedy]

There are quite a few contacts involved between these 3 contact groups - just to make sure, are they all created as a user that are utilizing xi_host/service_notification_handler?

Code: Select all

define service {
	host_name	acrux
	contact_groups	cg-unix,cg-integration,cg-application-maintenance

Things look fine between your settings otherwise. Usually if xi_host/service_notification_handler aren't working it's related to a crash SQL table, or a bug that has been seen in 5.3.0/5.3.1. (hence @dwhitfield asking for your version)

Could you also PM over a profile (Admin -> System Profile -> Download profile), and a screenshot of your 'Manage Email Settings' page?

[umunix]

Thanks for your responses guys.

For the purposes of squashing bugs, what version of XI are you using?

Sorry, I didn't mean to ignore you. I somehow didn't notice the question. As a matter of fact, we recently (Oct. 20) upgraded from 5.2.9 to 5.3.1. I'm not positive whether the problem existed before then. Anecdotally, we didn't notice any problems until after the upgrade.

There are quite a few contacts involved between these 3 contact groups - just to make sure, are they all created as a user that are utilizing xi_host/service_notification_handler?

Code: Select all

define service {
	host_name	acrux
	contact_groups	cg-unix,cg-integration,cg-application-maintenance

Things look fine between your settings otherwise. Usually if xi_host/service_notification_handler aren't working it's related to a crash SQL table, or a bug that has been seen in 5.3.0/5.3.1. (hence @dwhitfield asking for your version)

Could you also PM over a profile (Admin -> System Profile -> Download profile), and a screenshot of your 'Manage Email Settings' page?

I can verify that for all of the contacts using the xi_host/service_notification handlers, they are were all created as XI users. I'll PM you over our profile and a screenshot of our 'manage email settings'.

I made a couple other observations about our problem which hopefully help: We've noticed this problem doesn't appear to be limited to a specific check. From what I can tell, for each alert that is sent for a host or service to multiple contacts, only one (the first?) contact receives the email. /usr/local/nagiosxi/tmp/phpmailer.log only shows one sent email line for each alert that should go to multiple recipients. I'm also seeing "ERROR: Could not find user_id for contact '' in eventman.log. Just thought I'd mention this if in the hopes these observations align with the bug you guys have already discovered in 5.3.0/5.3.1.

Best,
Richard

[rkennedy]

Thanks for your responses guys.

For the purposes of squashing bugs, what version of XI are you using?

Sorry, I didn't mean to ignore you. I somehow didn't notice the question. As a matter of fact, we recently (Oct. 20) upgraded from 5.2.9 to 5.3.1. I'm not positive whether the problem existed before then. Anecdotally, we didn't notice any problems until after the upgrade.

There are quite a few contacts involved between these 3 contact groups - just to make sure, are they all created as a user that are utilizing xi_host/service_notification_handler?

Code: Select all

define service {
	host_name	acrux
	contact_groups	cg-unix,cg-integration,cg-application-maintenance

Things look fine between your settings otherwise. Usually if xi_host/service_notification_handler aren't working it's related to a crash SQL table, or a bug that has been seen in 5.3.0/5.3.1. (hence @dwhitfield asking for your version)

Could you also PM over a profile (Admin -> System Profile -> Download profile), and a screenshot of your 'Manage Email Settings' page?

I can verify that for all of the contacts using the xi_host/service_notification handlers, they are were all created as XI users. I'll PM you over our profile and a screenshot of our 'manage email settings'.

I made a couple other observations about our problem which hopefully help: We've noticed this problem doesn't appear to be limited to a specific check. From what I can tell, for each alert that is sent for a host or service to multiple contacts, only one (the first?) contact receives the email. /usr/local/nagiosxi/tmp/phpmailer.log only shows one sent email line for each alert that should go to multiple recipients. I'm also seeing "ERROR: Could not find user_id for contact '' in eventman.log. Just thought I'd mention this if in the hopes these observations align with the bug you guys have already discovered in 5.3.0/5.3.1.

Best,
Richard

I received your profile and things look good in here as well after a quick glance of your log files. Could you try upgrading to 5.3.2 which was just released today, to see if that solves things once and for all? This will rule out the complications between 5.3.0/5.3.1.

If the upgrade to 5.3.2 doesn't work, could you please run the following and post the output? echo 'use nagiosxi; select `username` from `xi_users`;' | mysql -uroot -pnagiosxi I'll use this to line up what we're seeing in your configuration vs SQL based on the log from eventman.log.

[umunix]

I upgraded to 5.3.2 but unfortunately the problem still persists. I'll PM you the output of the xi_users table as you asked. As far as I can tell though, all the notifications using the xi_service/host_notification_handler have xi users with the same contact name. Although, I am still seeing "ERROR: Could not find user_id for contact '' when doing a tail -f on /usr/local/nagiosxi/var/e*.

Code: Select all

*** GLOBAL HANDLER...
Array
(
    [event_id] => 1772896
    [event_source] => 2
    [event_type] => 2
    [event_time] => 2016-11-01 19:31:43
    [event_meta] => 
    [logging_enabled] => 1
)
ERROR: Could not find user_id for contact ''
PROCESS EVENT: ID=1772897, SOURCE=2, TYPE=2, TIME=2016-11-01 19:31:43

It's not my handler arguments, is it?

Code: Select all

xi_service_notification_handler	/usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_notification.php --notification-type=service --contact="$CONTACTNAME$" --contactemail="$CONTACTEMAIL$" --type=$NOTIFICATIONTYPE$ --escalated="$NOTIFICATIONISESCALATED$" --author="$NOTIFICATIONAUTHOR$" --comments="$NOTIFICATIONCOMMENT$" --host="$HOSTNAME$" --hostaddress="$HOSTADDRESS$" --hostalias="$HOSTALIAS$" --hostdisplayname="$HOSTDISPLAYNAME$" --service="$SERVICEDESC$" --hoststate=$HOSTSTATE$ --hoststateid=$HOSTSTATEID$ --servicestate=$SERVICESTATE$ --servicestateid=$SERVICESTATEID$ --lastservicestate=$LASTSERVICESTATE$ --lastservicestateid=$LASTSERVICESTATEID$ --servicestatetype=$SERVICESTATETYPE$ --currentattempt=$SERVICEATTEMPT$ --maxattempts=$MAXSERVICEATTEMPTS$ --serviceeventid=$SERVICEEVENTID$ --serviceproblemid=$SERVICEPROBLEMID$ --serviceoutput="$SERVICEOUTPUT$" --longserviceoutput="$LONGSERVICEOUTPUT$" --datetime="$LONGDATETIME$"

	xi_host_notification_handler	/usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_notification.php --notification-type=host --contact="$CONTACTNAME$" --contactemail="$CONTACTEMAIL$" --type=$NOTIFICATIONTYPE$ --escalated="$NOTIFICATIONISESCALATED$" --author="$NOTIFICATIONAUTHOR$" --comments="$NOTIFICATIONCOMMENT$" --host="$HOSTNAME$" --hostaddress="$HOSTADDRESS$" --hostalias="$HOSTALIAS$" --hostdisplayname="$HOSTDISPLAYNAME$" --hoststate=$HOSTSTATE$ --hoststateid=$HOSTSTATEID$ --lasthoststate=$LASTHOSTSTATE$ --lasthoststateid=$LASTHOSTSTATEID$ --hoststatetype=$HOSTSTATETYPE$ --currentattempt=$HOSTATTEMPT$ --maxattempts=$MAXHOSTATTEMPTS$ --hosteventid=$HOSTEVENTID$ --hostproblemid=$HOSTPROBLEMID$ --hostoutput="$HOSTOUTPUT$" --longhostoutput="$LONGHOSTOUTPUT$" --datetime="$LONGDATETIME$"

I also tried removing our iSMS plugin, in the event the third party plugin was causing problems for the notifications. That didn't seem to help either.

[tgriep]

Can you login to the XI server as root, run the following commands and post the /tmp/meta.txt file?

Code: Select all

echo "select * from xi_meta order by meta_id desc limit 50;" | psql nagiosxi nagiosxi >/tmp/meta.txt
echo "select * from xi_meta order by meta_id desc limit 50;" | mysql -uroot -pnagiosxi nagiosxi >>/tmp/meta.txt

Also, run the following and post the output here.

Code: Select all

ls -l /usr/local/nagiosxi/cron/eventman.php
ls -l /usr/local/nagiosxi/cron/event_handler.php
ls -l /usr/local/nagiosxi/html/includes/utils-events.inc.php
ls -l /usr/local/nagiosxi/scripts/handle_nagioscore.inc.php

This will help us to see if the files on your system were upgraded correctly.

[umunix]

Here you are, good sir:

Code: Select all

-rwxr-xr-x 1 nagios nagios 4325 Nov  1 17:13 /usr/local/nagiosxi/cron/eventman.php
-rwxr-xr-x 1 nagios nagios 3714 Nov  1 17:13 /usr/local/nagiosxi/cron/event_handler.php
-rwxr-x--- 1 nagios nagios 21760 Nov  1 17:13 /usr/local/nagiosxi/html/includes/utils-events.inc.php
-rw-r----- 1 nagios nagios 3316 Nov  1 17:13 /usr/local/nagiosxi/scripts/handle_nagioscore.inc.php