Duplicate alert emails.

[vinothsethuram]

** PROBLEM Host Alert: http://www.xyz.com is DOWN **


***** Nagios *****

Notification Type: PROBLEM
Host: http://www.xyz.com
State: DOWN
Address: 100.00.00.000
Info:

Date/Time: Fri Nov 22 11:15:40 EST 2013




** PROBLEM Host Alert: http://www.xyz.com is DOWN **



***** Nagios *****

Notification Type: PROBLEM
Host: http://www.xyz.com
State: DOWN
Address: 100.00.00.00
Info:

Date/Time: Fri Nov 22 11:15:34 EST 2013



Here is the command used

Code: Select all

define command{
        command_name    notify-host-by-email
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $LONGHOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
        }

[slansing]

Hmm.. would it be possible for you to set up a random email address that does not use your exchange/relay to receive notifications then send a test notification through that contact. Is this happening with all of your notifications?

[vinothsethuram]

Yes, its happening for all the contacts and for most of the notifications.

[slansing]

Can you run a tail on your mail log and force a notification from one of the hosts/services that is duplicating notifications? You should be able to force a notification from the details page of that object.

Code: Select all

tail -f /var/log/maillog

Does it actually show two emails being sent out, roughly 10-20 seconds apart? If possible, can you show us this output if that is true? "Block out any sensitive information."

[sreinhardt]

After looking through your templates I see many of them with the admins group added, which is generally a blanket group for most nagios contacts. Are the contacts that are receiving multiple messages also in the admins group or possibly added to a host\service group that is also inherited on those objects?

[vinothsethuram]

Here is the log... Log shows only one count, but I received 2 emails...



nagios sendmail[22737]: rB23Huc8022713: to=<[email protected]>, ctladdr=<[email protected]> (2147400000/2147400000), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=122164, relay=fraexsmtp1.abcdams.com. [10.1.4.22], dsn=2.0.0, stat=Sent ( <[email protected]> Queued mail for delivery)
Dec 1 22:17:57 nagios sendmail[22735]: rB23HucT022711: to=<[email protected]>, ctladdr=<[email protected]> (2147400000/2147400000), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=122150, relay=fraexsmtp1.abcdams.com. [10.1.4.22], dsn=2.0.0, stat=Sent ( <[email protected]> Queued mail for delivery)
Dec 1 22:17:57 nagios sendmail[22734]: rB23Hupw022709: to=<[email protected]>, ctladdr=<[email protected]> (2147400000/2147400000), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=122156, relay=fraexsmtp2.abcdams.com. [10.3.32.143], dsn=2.0.0, stat=Sent ( <[email protected]> Queued mail for delivery)
Dec 1 22:17:57 nagios sendmail[22738]: rB23HurO022707: to=<[email protected]>, ctladdr=<[email protected]> (2147400000/2147400000), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=122160, relay=fraexsmtp2.abcdams.com. [10.3.32.143], dsn=2.0.0, stat=Sent ( <[email protected]> Queued mail for delivery)
Dec 1 22:17:57 nagios sendmail[22742]: rB23Hvlt022716: to=<[email protected]>, ctladdr=<[email protected]> (2147400000/2147400000), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=122142, relay=fraexsmtp2.abcdams.com. [10.3.32.143], dsn=2.0.0, stat=Sent ( <[email protected]> Queued mail for delivery)
Dec 1 22:17:57 nagios sendmail[22733]: rB23Huam022708: to=<[email protected]>, ctladdr=<[email protected]> (2147400000/2147400000), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=122144, relay=fraexsmtp2.abcdams.com. [10.3.32.143], dsn=2.0.0, stat=Sent ( <[email protected]> Queued mail for delivery)
Dec 1 22:17:57 nagios sendmail[22740]: rB23Hu2Q022712: to=<[email protected]>, ctladdr=<[email protected]> (2147400000/2147400000), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=122148, relay=fraexsmtp1.abcdams.com. [10.1.4.22], dsn=2.0.0, stat=Sent ( <[email protected]> Queued mail for delivery)
Dec 1 22:17:57 nagios sendmail[22743]: rB23HuCS022714: to=<[email protected]>, ctladdr=<[email protected]> (2147400000/2147400000), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=122176, relay=fraexsmtp2.abcdams.com. [10.3.32.143], dsn=2.0.0, stat=Sent ( <[email protected]> Queued mail for delivery)
Dec 1 22:17:57 nagios sendmail[22744]: rB23Hv95022717: to=<[email protected]>, ctladdr=<[email protected]> (2147400000/2147400000), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=122144, relay=fraexsmtp1.abcdams.com. [10.1.4.22], dsn=2.0.0, stat=Sent ( <[email protected]> Queued mail for delivery)
Dec 1 22:17:57 nagios sendmail[22746]: rB23HvMP022719: to=<[email protected]>, ctladdr=<[email protected]> (2147400000/2147400000), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=122154, relay=fraexsmtp1.abcdams.com. [10.1.4.22], dsn=2.0.0, stat=Sent ( <[email protected]> Queued mail for delivery)

[abrist]

If the maillog only logged 1 outgoing message per alert, but you received, I would assume your relay is duplicating them. Have you checked your mail relay logs?

[vinothsethuram]

How to check mail relay logs? Please help

[slansing]

This would be something internal to your environment, we have no idea how this is set up on your end but your mail admin would have a good idea.

Now... that being said these are all from the same second, so it looks like you have a ton of notification traffic, which makes it hard for us to see if a duplicate was created without a larger tail grab.

[lftsy]

Good afternoon,
It might be related
I am trying to reopen bug report because I have it closed by mistake "0000388: multiple checks on a same service after reception of an external command PROCESS_SERVICE_CHECK_RESULT."
=> http://tracker.nagios.org/view.php?id=388#bugnotes
If a check is duplicated at runtime and send a notification each time, it could explain your problem