Page 1 of 1

Email Notifications not being received

Posted: Thu Aug 01, 2013 5:57 am
by StuartHurst
Hi all
Hoping that my Obi Wan is here as I'm pulling my hair out here.

I have Nagios 3.4.1 running Ubuntu Server 13.04. I have postfix service installed. (I have changed the email address below to a generic one but the real one works)

I originally set the commands.cfg email to be as below as I was testing everything, but now I need this to go to different people depending on the fault. I have also tried using contact_groups too.
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/mailx -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" [email protected]
this sent out email perfectly fine, but now we're nearly ready to go live, I've created the contacts.cfg with the appropriate information
define contact{
contact_name ita
alias analysts
host_notifications_enabled 1
service_notifications_enabled 1
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,u,c,r
host_notification_options d,r,u
service_notification_commands notify-service-by-email
host_notification_commands notify-host-by-email
email [email protected]
}
In the service check, I use
contacts ita
in the definition

In the nagios log, I only ever see
[1375353854] HOST ALERT: BLC_Switch;DOWN;HARD;1;Unreachable
[1375353854] HOST NOTIFICATION: root;BLC_Switch;DOWN;notify-host-by-email;Unreachable
I've been looking and checked that all the macro names have a $ at each end. I'm absolutely stuck as looking on the web and here I see that some remove the end $, some put it in " or '.

Please - help me Obi Wan, you're my only hope.

Re: Email Notifications not being received

Posted: Thu Aug 01, 2013 10:55 am
by abrist
You should replace your email in the command with:

Code: Select all

$CONTACTEMAIL$
So that the command reads:

Code: Select all

command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/mailx -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
This way, each contact's email will replace the $CONTACTEMAIL$ macro.

Re: Email Notifications not being received

Posted: Thu Aug 01, 2013 11:06 am
by StuartHurst
Yes - I should have posted up my current commands.cfg for the notification - it is as below..
# 'notify-host-by-email' command definition
define command{
command_name notify-host-by-email
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
}

# 'notify-service-by-email' command definition
define command{
command_name notify-service-by-email
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /usr/bin/mail -s "** $NOTIFICATION$ TYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
}

I am not sure if the problem is actually here or somewhere else - looking at the nagios.log file, I see the following
[1375372857] HOST ALERT: Cr1Sw9;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
[1375372857] SERVICE ALERT: Cr1Sw9;PING;CRITICAL;HARD;1;PING CRITICAL - Packet loss = 100%
[1375372857] PASSIVE HOST CHECK: BLC_Switch;1;100
[1375372857] HOST ALERT: BLC_Switch;DOWN;HARD;1;100
[1375372857] HOST NOTIFICATION: technicians;BLC_Switch;DOWN;notify-host-by-email;100
[1375372857] SERVICE ALERT: Cr1Sw8;PING;CRITICAL;HARD;1;PING CRITICAL - Packet loss = 100%
[1375372867] HOST ALERT: Cr1Sw9;UP;SOFT;2;PING OK - Packet loss = 0%, RTA = 0.58 ms
[1375372867] HOST ALERT: Cr1Sw8;UP;SOFT;2;PING OK - Packet loss = 0%, RTA = 0.88 ms
[1375372867] SERVICE ALERT: Cr1Sw9;PING;OK;HARD;1;PING OK - Packet loss = 0%, RTA = 0.78 ms
I believe that the bolded area is where the notification is setup and should be sent to Nagios to send out via postfix but we get nothing.
As the system would receive email from the hardcoded email address, there isn't an issue with exchange (and I've set it up with a separate IP and set that to open relay from Nagios IP only anyway.

Re: Email Notifications not being received

Posted: Thu Aug 01, 2013 11:09 am
by StuartHurst
This is mad - I've just re-submitted a passive check and blow me - it's not only gone and emailed me... I wonder if the email queue is full????

Re: Email Notifications not being received

Posted: Thu Aug 01, 2013 12:08 pm
by abrist
You can check the queue with:

Code: Select all

mailq
To find out more information about the mailing of notifications:

Code: Select all

tail -25 /var/log/maillog

Re: Email Notifications not being received

Posted: Thu Aug 01, 2013 3:57 pm
by StuartHurst
cheers - I will check tomorrow

Re: Email Notifications not being received

Posted: Thu Aug 01, 2013 4:05 pm
by lmiltchev
Sure - let us know when you have the output.

Re: Email Notifications not being received

Posted: Fri Aug 02, 2013 2:52 am
by StuartHurst
Interesting as I'm using Exchange 2013 on Server 2012 that isn't live yet.... It has 8Gb RAM and 4 CPU's. Just going to take a look at it now.

-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
303477C1 559 Fri Aug 2 08:42:33 "nagios@nagios"@hambleton.gov.uk
(host 10.1.2.239[10.1.2.239] said: 452 4.3.1 Insufficient system resources (in reply to end of DATA command))
[email protected]

238715BC 554 Fri Aug 2 08:42:33 "nagios@nagios"@hambleton.gov.uk
(host 10.1.2.239[10.1.2.239] said: 452 4.3.1 Insufficient system resources (in reply to end of DATA command))
[email protected]

2C1765DF 559 Fri Aug 2 08:42:33 "nagios@nagios"@hambleton.gov.uk
(host 10.1.2.239[10.1.2.239] said: 452 4.3.1 Insufficient system resources (in reply to end of DATA command))
[email protected]

26AE15C2 554 Fri Aug 2 08:42:33 "nagios@nagios"@hambleton.gov.uk
(host 10.1.2.239[10.1.2.239] said: 452 4.3.1 Insufficient system resources (in reply to end of DATA command))
[email protected]

Re: Email Notifications not being received

Posted: Fri Aug 02, 2013 4:00 am
by StuartHurst
Restarted the server, allocated more memory, and all is well. how annoying that this didn't come up when the email address was hardcoded in.

Re: Email Notifications not being received

Posted: Fri Aug 02, 2013 9:39 am
by abrist
StuartHurst wrote:Restarted the server, allocated more memory, and all is well. how annoying that this didn't come up when the email address was hardcoded in.
That is odd. At least the error message was verbose enough that once you got the error it was easily remedied. Enjoy the weekend, cheers!