Problem email alerts to certain users

[rez1sog]

I am not sure what is going on but we seem to be having a problem sending alerts to certain users. I am not sure why.

One of the users that has issues is: pvangundy

When I watch the nagios.log file, I see the following:

Code: Select all

[1349874430] SERVICE NOTIFICATION: rhardesty;OPDBPROD;SSH Server;FLAPPINGSTOP (OK);xi_service_notification_handler;openssh-daemon (pid  2519) is running...
[1349874430] SERVICE NOTIFICATION: pvangundy;OPDBPROD;SSH Server;FLAPPINGSTOP (OK);xi_service_notification_handler;openssh-daemon (pid  2519) is running...
[1349874430] SERVICE NOTIFICATION: mdunn;OPDBPROD;SSH Server;FLAPPINGSTOP (OK);xi_service_notification_handler;openssh-daemon (pid  2519) is running...
[1349874430] SERVICE NOTIFICATION: jgranara;OPDBPROD;SSH Server;FLAPPINGSTOP (OK);xi_service_notification_handler;openssh-daemon (pid  2519) is running...
[1349874430] SERVICE NOTIFICATION: jglatiotis;OPDBPROD;SSH Server;FLAPPINGSTOP (OK);xi_service_notification_handler;openssh-daemon (pid  2519) is running...
[1349874431] SERVICE NOTIFICATION: elevis;OPDBPROD;SSH Server;FLAPPINGSTOP (OK);xi_service_notification_handler;openssh-daemon (pid  2519) is running...

Everyone gets the email except vangundy. I check the Contact settings via the Admin Gui in XI and it looks the same as rhardesty. When I look at the actual contacts.cfg file on the server it looks like:

Code: Select all

define contact {
        contact_name                            pvangundy
        alias                                   Paul VanGundy
        host_notification_period                pvangundy_notification_times
        service_notification_period             pvangundy_notification_times
        host_notification_options               d,u,r,f,s
        service_notification_options            w,u,c,r,f,s
        email                                   [email protected]
        use                                     xi_contact_generic
        }

This is one from a working contact....

Code: Select all

define contact {
        contact_name                            rhardesty
        alias                                   Robert Hardesty
        host_notification_period                rhardesty_notification_times
        service_notification_period             rhardesty_notification_times
        host_notification_options               d,u,r,f,s
        service_notification_options            w,u,c,r,f,s
        email                                   [email protected]
        use                                     xi_contact_generic
        }

They look the same to me. The *_notification_times are the same as well (24 hr).

I can see the email for rhardesty going out using sendmail:

Code: Select all

Oct 10 09:12:00 nagiosxi sendmail[2279]: q9ADC0u5002279: [email protected], [email protected] (500/100), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=31751, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (q9ADC0Zn002282 Message accepted for delivery)
Oct 10 09:12:00 nagiosxi sendmail[2284]: q9ADC0Zn002282: to=<[email protected]>, delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=121940, relay=mail.rez1.com. [10.20.20.18], dsn=2.0.0, stat=Sent (Ok: queued as 91581282370)

But nothing shows up for pvangundy in the maillog file.

They are all in the same contact group.

I have about three email address that do this. I am looking for a way to fix this that is more then just "update the software".

Any ideas?

Here is the system info:

Code: Select all

Nagios XI Installation Profile
Download Profile	
System:
nagiosxi 2.6.18-308.4.1.el5 x86_64
CentOS release 5.6 (Final)
Gnome is not installed
Apache Information
PHP Version: 5.1.6
Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1
Server Name: nagiosxi
Server Address: 10.20.20.74
Server Port: 80
Date/Time
PHP Timezone: America/New_York
PHP Time: Wed, 10 Oct 2012 09:26:16 -0400
System Time: Wed, 10 Oct 2012 09:26:16 -0400
Nagios XI Data
nagios (pid 22286) is running...
NPCD running (pid 4046).
ndo2db (pid 4181) is running...
CPU Load 15: 2.84
Total Hosts: 237
Total Services: 2754

Ping Test localhost
Running:

/bin/ping -c 3 localhost 2>&1 

PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.027 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.024 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.042 ms

--- localhost ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.024/0.031/0.042/0.007 ms

[CGraham]

Where I'd start....

#1. Can you show us the pvangundy_notification_times vs. rhardesty_notification_times
#2. Change just the rhardesty email address to [email protected] and fail something to see if he receives the alarm.

[rez1sog]

Thanks. Those were good suggestions. The two *notifications* are the exact same other than the name. I even swapped them to no avail between me and him.

Interesting, when I did change the vangundy email address to me, it worked. I had also changed my address to him and it did not. I thought it was something in Sendmail, but I do not even see any error with the vangundy address in it. So, I am confused.

What is happening between the nagios.log entry and the actual email being request being sent to Sendmail?

[mguthrie]

Could this email address simply be catching all of the emails in a SPAM filter?

[rez1sog]

I thought that too, but I can send him an email just mail using the UNIX "mail" command just fine. He can get email via Sendmail from the server, just not from Nagios. There is just no entry in Sendmail to show a failure.

Is there a Spam filter in Nagios somewhere I have not heard about?

[CGraham]

Sounds like "a" spam filter, but maybe you could use strace to follow the sendmail pid and grep for the email address.

strace -f -p [sendmail's pid] | grep "vangundy"

I don't use sendmail so I'm not sure what you're going to get with that. You can also look at making logging more verbose...

EDIT: You replied already. Ok. Try running the EXACT command that nagios is going to run from your service email config on the command line. Send to yours then his. Running from the console might show the error.

[scottwilkerson]

Also, are you sure the user has their notifications enabled? Click username in top right of interface and choose notification preferences.

[CGraham]

It shouldn't be the notifications being disabled because when he changes just the email address it starts working and when he puts the vangundy email on a working contact it stops working. Quite strange.

[rez1sog]

I check if notifications were enabled and they are. How would I send the alert on the command line using nagios? For something like this:

[1349893195] SERVICE NOTIFICATION: pvangundy;EDIFTPPROD;Memory Usage;OK;xi_service_notification_handler;OK - 1736 / 2003 MB (86%) Free Memory, Used: 267 MB, Shared: 0 MB, B

What would be the paths and syntax to use?

[CGraham]

Go into your CCM and click "Commands" on the left. Search for email and find your "notify-service-by-email" command. Mine looks like this:

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTNAME$\nHostname: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nHost Info: $HOSTNOTES$\n\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nNotification Comments: $NOTIFICATIONCOMMENT$\n\nAdditional Info: $SERVICEOUTPUT$\n$SERVICENOTES$" | /bin/mail -s "Service $NOTIFICATIONTYPE$: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$" $CONTACTEMAIL$

The variables won't work, so remove the "$". And edit the contact email at the end with the emails we're testing. Here's mine removing the dollar signs and changing the email address at the end.

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: NOTIFICATIONTYPE\n\nService: SERVICEDESC\nHost: HOSTNAME\nHostname: HOSTALIAS\nAddress: HOSTADDRESS\nHost Info: HOSTNOTES\n\nState: SERVICESTATE\n\nDate/Time: LONGDATETIME\n\nNotification Comments: NOTIFICATIONCOMMENT\n\nAdditional Info: SERVICEOUTPUT\nSERVICENOTES" | /bin/mail -s "Service NOTIFICATIONTYPE: HOSTALIAS/SERVICEDESC is SERVICESTATE" [email protected]

You should be able to run that from the command line and get a notification or an error message....