Nagios Sending Notification Emails to $@hostname.localdomain

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
Locked
rommelcid
Posts: 4
Joined: Sat May 23, 2015 3:19 am

Nagios Sending Notification Emails to $@hostname.localdomain

Post by rommelcid »

Hi,

This has really been bugging me for quite a while and I am already pulling my hair out. For the life of me, I don't understand why everytime Nagios sends email alerts, it also sends alerts to $@hostname.localdomain. As a result, the nagios user is receiving a lot of emails of non-delivery reports. I don't think this is a postfix error as well, since I have already tried and succeeded sending emails as the nagios user to our designated corporate email accounts using mailx without fail.

We are using Nagios 4.0.8. Please help me troubleshoot this.
jdalrymple
Skynet Drone
Posts: 2620
Joined: Wed Feb 11, 2015 1:56 pm

Re: Nagios Sending Notification Emails to $@hostname.localdo

Post by jdalrymple »

These are the somewhat canned default notification commands:

notify-host-by-email

Code: Select all

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
notify-service-by-email

Code: Select all

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
Note that the macro $CONTACTEMAIL$ is the winner in this case, that comes from your contact definition. What do you have it defined as?

Honestly, I suspect what you're actually getting is bounce/undeliverable messages. We'd have to see the appropriate parts of the maillog to be certain.
rommelcid
Posts: 4
Joined: Sat May 23, 2015 3:19 am

Re: Nagios Sending Notification Emails to $@hostname.localdo

Post by rommelcid »

Thank you very much for your reply jdalrymple. Here's a portion of /var/log/maillog:

Code: Select all

Jun  9 06:32:28 wms postfix/qmgr[20515]: 28303411A3: from=<nagios@wms.localdomain>, size=709, nrcpt=1 (queue active)
Jun  9 06:32:28 wms postfix/pickup[16532]: 28AB0411A6: uid=2223 from=<nagios>
Jun  9 06:32:28 wms postfix/cleanup[23156]: 28AB0411A6: message-id=<20150609103228.28AB0411A6@wms.localdomain>
Jun  9 06:32:28 wms postfix/qmgr[20515]: 28AB0411A6: from=<nagios@wms.localdomain>, size=717, nrcpt=1 (queue active)
Jun  9 06:32:28 wms postfix/local[23160]: 277904119C: to=<$@wms.localdomain>, orig_to=<$>, relay=local, delay=0.1, delays=0.06/0.02/0/0.02, dsn=5.1.1, status=bounced (unknown user: "$")
Jun  9 06:32:28 wms postfix/cleanup[23156]: 327A0411A8: message-id=<20150609103228.327A0411A8@wms.localdomain>
Jun  9 06:32:28 wms postfix/qmgr[20515]: 327A0411A8: from=<>, size=2454, nrcpt=1 (queue active)
Jun  9 06:32:28 wms postfix/bounce[23163]: 277904119C: sender non-delivery notification: 327A0411A8
Jun  9 06:32:28 wms postfix/qmgr[20515]: 277904119C: removed
Jun  9 06:32:28 wms postfix/local[23162]: 28AB0411A6: to=<root@localhost.localdomain>, orig_to=<root@localhost>, relay=local, delay=0.09, delays=0.04/0.03/0/0.02, dsn=2.0.0, status=sent (delivered to mailbox)
Jun  9 06:32:28 wms postfix/local[23160]: 327A0411A8: to=<nagios@wms.localdomain>, relay=local, delay=0.02, delays=0/0/0/0.02, dsn=2.0.0, status=sent (delivered to mailbox)
Jun  9 06:32:28 wms postfix/qmgr[20515]: 327A0411A8: removed
Jun  9 06:32:28 wms postfix/qmgr[20515]: 28AB0411A6: removed
What follows below are logs of emails sent by Nagios to non-local recipients. From what I can glean from the logs, Nagios always tries to send notifications first to $@wms.localdomain, followed by everyone else.

I really hope you can help me on this one. We are using Nagios 4.0.8. utilizing notify-host-by-email and notify-service-by-email that came with it. I have already rebuilt the system. I have also disabled dovecot. I have also updated NagiosQL to the latest version. I've gone over all the configuration files and simplified them. After all that, we still have this issue, and I'm starting to think Nagios does this by design. What's weird is that nagios logs does not show it to be sending emails to $@wms.local:

Code: Select all

[1433886622] SERVICE NOTIFICATION: eilon;bitcoingg.com;HTTP;CRITICAL;notify-service-by-email;connect to address www.bitcoingg.com and port 80: Connection timed out
[1433886622] SERVICE NOTIFICATION: root;bitcoingg.com;HTTP;CRITICAL;notify-service-by-email;connect to address www.bitcoingg.com and port 80: Connection timed out
[1433886622] SERVICE NOTIFICATION: tech;bitcoingg.com;HTTP;CRITICAL;notify-service-by-email;connect to address www.bitcoingg.com and port 80: Connection timed out
[1433886647] Auto-save of retention data completed successfully.
[1433886808] SERVICE NOTIFICATION: eilon;solari88.com;HTTP;CRITICAL;notify-service-by-email;connect to address www.solari88.com and port 80: Connection timed out
[1433886808] SERVICE NOTIFICATION: root;solari88.com;HTTP;CRITICAL;notify-service-by-email;connect to address www.solari88.com and port 80: Connection timed out
[1433886808] SERVICE NOTIFICATION: tech;solari88.com;HTTP;CRITICAL;notify-service-by-email;connect to address www.solari88.com and port 80: Connection timed out
[1433886855] SERVICE NOTIFICATION: eilon;bestbitcoinroulette.com;HTTP;CRITICAL;notify-service-by-email;connect to address www.bestbitcoinroulette.com and port 80: Connection timed out
[1433886855] SERVICE NOTIFICATION: root;bestbitcoinroulette.com;HTTP;CRITICAL;notify-service-by-email;connect to address www.bestbitcoinroulette.com and port 80: Connection timed out
[1433886855] SERVICE NOTIFICATION: tech;bestbitcoinroulette.com;HTTP;CRITICAL;notify-service-by-email;connect to address www.bestbitcoinroulette.com and port 80: Connection timed out
[1433886865] SERVICE NOTIFICATION: eilon;pineoptions.com;HTTP;CRITICAL;notify-service-by-email;connect to address www.pineoptions.com and port 80: Connection timed out
[1433886865] SERVICE NOTIFICATION: root;pineoptions.com;HTTP;CRITICAL;notify-service-by-email;connect to address www.pineoptions.com and port 80: Connection timed out
[1433886865] SERVICE NOTIFICATION: tech;pineoptions.com;HTTP;CRITICAL;notify-service-by-email;connect to address www.pineoptions.com and port 80: Connection timed out
[1433887126] SERVICE ALERT: bitcoingg.com;HTTP;OK;HARD;2;HTTP OK: Status line output matched "200" - 8191 bytes in 86.301 second response time
[1433887126] SERVICE NOTIFICATION: eilon;bitcoingg.com;HTTP;OK;notify-service-by-email;HTTP OK: Status line output matched "200" - 8191 bytes in 86.301 second response time
[1433887126] SERVICE NOTIFICATION: root;bitcoingg.com;HTTP;OK;notify-service-by-email;HTTP OK: Status line output matched "200" - 8191 bytes in 86.301 second response time
[1433887126] SERVICE NOTIFICATION: tech;bitcoingg.com;HTTP;OK;notify-service-by-email;HTTP OK: Status line output matched "200" - 8191 bytes in 86.301 second response time
Thanks.
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: Nagios Sending Notification Emails to $@hostname.localdo

Post by tgriep »

Have you checked the notify-host-by-email and notify-service-by-email commands to see if there is an extra $ that shouldn't be there?
Can you post your commands and contact files here so we can review them?
Be sure to check out our Knowledgebase for helpful articles and solutions!
rommelcid
Posts: 4
Joined: Sat May 23, 2015 3:19 am

Re: Nagios Sending Notification Emails to $@hostname.localdo

Post by rommelcid »

Thank you very much for your repy, Tagrief. As per your request, here are the command definitions we use on the server for email notifications:

Code: Select all

define command {
	command_name                   	notify-host-by-email
	command_line                   	/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
	register                       	1
}

define command {
	command_name                   	notify-service-by-email
	command_line                   	/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
	register                       	1
}
Here are also the contact definitions. I modified them for security's sake:

Code: Select all

define contact {
	contact_name                   	eilon
	alias                          	Eilon Arad
	host_notifications_enabled     	1
	service_notifications_enabled  	1
	email                          	eilon@example.com.ph
	use                            	generic-contact
	register                       	1
	}	

define contact {
	contact_name                   	root
	alias                          	Nagios Admin
	email                          	root@localhost
	use                            	generic-contact
	register                       	1
	}	

define contact {
	contact_name                   	tech
	alias                          	Leadya Tech Support
	host_notifications_enabled     	1
	service_notifications_enabled  	1
	email                          	tech@example.com.ph
	use                            	generic-contact
	register                       	1
	}	
Please note that all configuration files were created by NagiosQL after they were imported from Nagios when the former was installed, and they all work as expected except for the notifications thing.

Thanks guys!
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:

Re: Nagios Sending Notification Emails to $@hostname.localdo

Post by Box293 »

Can you please post the template definition for "generic-contact".
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
rommelcid
Posts: 4
Joined: Sat May 23, 2015 3:19 am

Re: Nagios Sending Notification Emails to $@hostname.localdo

Post by rommelcid »

Here's the contact definition for the generic-contact template:

Code: Select all


define contact {
	name                           	generic-contact
	host_notification_period       	24x7
	service_notification_period    	24x7
	host_notification_options      	d,u,r,f,s
	service_notification_options   	w,u,c,r,f,s
	host_notification_commands     	notify-host-by-email
	service_notification_commands  	notify-service-by-email
	register                       	0
}

jdalrymple
Skynet Drone
Posts: 2620
Joined: Wed Feb 11, 2015 1:56 pm

Re: Nagios Sending Notification Emails to $@hostname.localdo

Post by jdalrymple »

Can you remove the contact named root from your service definition(s) and force a notification? I suspect whatever is going on is outside the hands of nagios and is actually some mangling that your local mailer is doing for one reason or another. Maybe not - but that would help us determine for sure.
spaceopdin
Posts: 1
Joined: Thu Sep 10, 2015 6:48 am

Re: Nagios Sending Notification Emails to $@hostname.localdo

Post by spaceopdin »

Hi
I'm have the same issue and have not been able to find a solutions, my configs were also generated with nagiosql.

Was a solutions to this ever found?
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: Nagios Sending Notification Emails to $@hostname.localdo

Post by tmcdonald »

We're not able to provide assistance for NagioSQL, so if none of the advice offered thus far seems relevant to your system then as @jdalrymple mentioned it seems likely to be outside of the nagios system.
Former Nagios employee
Locked