using contacts causing double email alert

[tmcdonald]

Thanks, as ever, for the help millisa.

[hata_ph]

Hi milisa,

Yes I do have multiple email addresses put in itdba because i want the alert to include multiple emails in the TO list. Using contact_groups do not have that option.
Reading your suggestion, i did try to use single email for itdba but using contacts itdba still show duplicate emails...
I do agree with you about the issue is cause by contact inheritance but i am just not sure where it inherite from...

About the mail.log, it show 2 seperate mail entry is because first i am test using contacts itdba and the second one is using contact_groups itdba

Code: Select all

Sep 17 08:42:09 SIT-NAGIOS postfix/pickup[19373]: 8E979600B3: uid=106 from=<nagios>
Sep 17 08:42:09 SIT-NAGIOS postfix/cleanup[4639]: 8E979600B3: message-id=<20140917004209.8E979600B3@SIT-NAGIOS>
Sep 17 08:42:09 SIT-NAGIOS postfix/qmgr[976]: 8E979600B3: from=<nagios@SIT-NAGIOS>, size=649, nrcpt=2 (queue active)
Sep 17 08:42:09 SIT-NAGIOS postfix/smtp[4645]: 668066008D: to=<itnetworksupport@xxx.com>, relay=sit-smtp01.xxx.com[10.16.0.142]:25, delay=0.47, delays=0.11/0.07/0.02/0.28, dsn=2.6.0, status=sent (250 2.6.0  <20140917004209.668066008D@SIT-NAGIOS> Queued mail for delivery)
Sep 17 08:42:09 SIT-NAGIOS postfix/qmgr[976]: 668066008D: removed
Sep 17 08:42:09 SIT-NAGIOS postfix/smtp[4648]: 8E979600B3: to=<itdba@xxx.com>, relay=sit-smtp01.xxx.com[10.16.0.142]:25, delay=0.37, delays=0.01/0.04/0.09/0.22, dsn=2.6.0, status=sent (250 2.6.0  <20140917004209.8E979600B3@SIT-NAGIOS> Queued mail for delivery)
Sep 17 08:42:09 SIT-NAGIOS postfix/smtp[4648]: 8E979600B3: to=<itnetworksupport@xxx.com>, relay=sit-smtp01.xxx.com[10.16.0.142]:25, delay=0.37, delays=0.01/0.04/0.09/0.22, dsn=2.6.0, status=sent (250 2.6.0  <20140917004209.8E979600B3@SIT-NAGIOS> Queued mail for delivery)
Sep 17 08:42:09 SIT-NAGIOS postfix/qmgr[976]: 8E979600B3: removed

[millisa]

I think it is a bad idea to use the multiple email addresses for a single contact like that; it just complicates things.
As for the 2 service notifications, I'd really want to see a matching config to the nagios log. The last one you supplied that was complete enough didn't match up with the log supplied.

Please do this:
Setup three service definitions (not the templates) with the following:
Test1

Code: Select all

   contacts         itdba
   contact_groups         null

Test2

Code: Select all

  contacts  null
  contact_groups itnetworksupport

Test3

Code: Select all

And the third with the config you are trying to make work (state what you expect that alert to do).
If you are wanting to notify a group, put the contact_groups setting here.  Just make it clear who you are intending to receive the alerts.

Save the config. Restart nagios after checking the config. Do not change the config any further from this point.

Cause each of the three services to generate an alert. If you click on the service in the nagios web GUI, one of the service commands is to 'Send custom service notification', choose that one. Click the 'forced' checkbox'. Put a comment in the field and commit it. That will cause a service notification without needing to actually simulate a critical or warning condition. Note the time you send each. For each test, note which email addresses receive the notice. Give about a minute between each test just so its easier to pick them out from each other in the logs.

Submit the nagios.log showing the nagios restart and up to the point that the 3rd test has been made.
Submit the current config files (when editing the contact email address, use the format of XXX1@xxx.com so that we can see where you have multiple addresses). Do not edit anything other than those email addresses and don't change the configs until someone gets back to you.

I'll be at the keyboard another 3-4 hours and if you get the results in by then I'll get you a response before I call it a day.

[hata_ph]

i have attached the config and log files with only the domain name changed.
Pls take note that I have disable service notification globally and enable notification for only 3 services with different contacts/contacts_group emails.
For the 3 contacts list (root, itdba and itnetworksupport), i have modify it to use a single email address instead of multiple one..

Code: Select all

# default services template
define service{
	name				generic-service
	use				generic-service-template
	#contacts			root
	contacts			null
	contact_groups			admins
}

# services template for DBA team
define service{
	name				generic-service-dba
	use				generic-service-template
	contacts			itdba
	contact_groups			null
	#contact_groups			db-admins
}

# services template for CIM team
define service{
	name				generic-service-cim
	use				generic-service-template
	contacts			itcim
	#contact_groups			cim-admins
}

You can observe the nagios.log that for 6800SRETDB (using generic-service-dba) and SIT-NAS2 (using generic-service) I only receive 1 notification while 2 for SRMSHOST08A (using generic-service-cim).

Code: Select all

[1411110016] SERVICE FLAPPING ALERT: SIT-WSUS;CPU Load;STARTED; Service appears to have started flapping (30.5% change >= 20.0% threshold)
[1411110186] SERVICE NOTIFICATION: itdba;6800SRETDB;F Drive;CRITICAL;notify-service-by-email;SNMP CRITICAL - F:\ Label:New Volume  Serial Number 46939174 at 96% with 10,190 of 286,081 MB free
[1411110236] SERVICE NOTIFICATION: itnetworksupport;SIT-NAS2;F Drive;CRITICAL;notify-service-by-email;SNMP CRITICAL - F:\ Label:Data  Serial Number d46ec860 at 97% with 8,741 of 419,258 MB free
[1411110356] SERVICE ALERT: SIT-HUBCAS01;Physical Memory;WARNING;SOFT;1;SNMP WARNING - Physical Memory at 85% with 1,178 of 8,186 MB free
[1411110416] SERVICE ALERT: S-CEMSDB02;Physical Memory;CRITICAL;SOFT;1;SNMP CRITICAL - Physical Memory at 97% with 188 of 8,191 MB free
[1411110416] SERVICE ALERT: SIT-HUBCAS01;Physical Memory;WARNING;SOFT;2;SNMP WARNING - Physical Memory at 85% with 1,178 of 8,186 MB free
[1411110476] SERVICE ALERT: S-CEMSDB02;Physical Memory;CRITICAL;SOFT;2;SNMP CRITICAL - Physical Memory at 97% with 188 of 8,191 MB free
[1411110476] SERVICE ALERT: SIT-HUBCAS01;Physical Memory;OK;SOFT;3;SNMP OK - Physical Memory at 84% with 1,237 of 8,186 MB free
[1411110536] SERVICE ALERT: S-CEMSDB02;Physical Memory;CRITICAL;SOFT;3;SNMP CRITICAL - Physical Memory at 97% with 188 of 8,191 MB free
[1411110536] SERVICE ALERT: SINFOHUB;D Drive;CRITICAL;HARD;6;SNMP CRITICAL - D:\ Label:New Volume  Serial Number 78af2747 at 90% with 24,382 of 256,000 MB free
[1411110536] SERVICE FLAPPING ALERT: SINFOHUB;D Drive;STARTED; Service appears to have started flapping (22.0% change >= 20.0% threshold)
[1411110566] SERVICE ALERT: SIT-MBX01;Physical Memory;WARNING;HARD;6;SNMP WARNING - Physical Memory at 94% with 879 of 16,378 MB free
[1411110576] SERVICE NOTIFICATION: itnetworksupport;SRMSHOST08A;C Drive;CRITICAL;notify-service-by-email;SNMP CRITICAL - C:\ Label:  Serial Number 30ae4373 at 82% with 5,220 of 30,482 MB free
[1411110576] SERVICE NOTIFICATION: itcim;SRMSHOST08A;C Drive;CRITICAL;notify-service-by-email;SNMP CRITICAL - C:\ Label:  Serial Number 30ae4373 at 82% with 5,220 of 30,482 MB free
[1411110596] SERVICE ALERT: S-CEMSDB02;Physical Memory;CRITICAL;SOFT;4;SNMP CRITICAL - Physical Memory at 98% with 104 of 8,191 MB free
[1411110656] SERVICE ALERT: S-CEMSDB02;Physical Memory;CRITICAL;SOFT;5;SNMP CRITICAL - Physical Memory at 98% with 104 of 8,191 MB free
[1411110716] SERVICE ALERT: S-CEMSDB02;Physical Memory;CRITICAL;HARD;6;SNMP CRITICAL - Physical Memory at 97% with 197 of 8,191 MB free
[1411110786] SERVICE NOTIFICATION: itdba;6800SRETDB;F Drive;CRITICAL;notify-service-by-email;SNMP CRITICAL - F:\ Label:New Volume  Serial Number 46939174 at 96% with 10,190 of 286,081 MB free
[1411110836] SERVICE NOTIFICATION: itnetworksupport;SIT-NAS2;F Drive;CRITICAL;notify-service-by-email;SNMP CRITICAL - F:\ Label:Data  Serial Number d46ec860 at 97% with 8,741 of 419,258 MB free
[1411110866] SERVICE ALERT: SIT-MBX01;Physical Memory;CRITICAL;HARD;6;SNMP CRITICAL - Physical Memory at 95% with 768 of 16,378 MB free

[millisa]

SRMHOST08A notification for C drive full

Code: Select all

[1411110576] SERVICE NOTIFICATION: itnetworksupport;SRMSHOST08A;C Drive;CRITICAL;notify-service-by-email;SNMP CRITICAL - C:\ Label:  Serial Number 30ae4373 at 82% with 5,220 of 30,482 MB free
[1411110576] SERVICE NOTIFICATION: itcim;SRMSHOST08A;C Drive;CRITICAL;notify-service-by-email;SNMP CRITICAL - C:\ Label:  Serial Number 30ae4373 at 82% with 5,220 of 30,482 MB free

1411110576 = Fri, 19 Sep 2014 07:09:36 GMT, 15:09 localtime. Corresponding maillogs:

Code: Select all

Sep 19 15:09:36 SIT-NAGIOS postfix/pickup[10150]: 5609F60062: uid=106 from=<nagios>
Sep 19 15:09:36 SIT-NAGIOS postfix/cleanup[29168]: 5609F60062: message-id=<20140919070936.5609F60062@SIT-NAGIOS>
Sep 19 15:09:36 SIT-NAGIOS postfix/qmgr[976]: 5609F60062: from=<nagios@SIT-NAGIOS>, size=619, nrcpt=1 (queue active)
Sep 19 15:09:36 SIT-NAGIOS postfix/smtp[29172]: 5609F60062: to=<itnetworksupport@xxx.com>, relay=sit-smtp01.xxx.com[10.16.0.142]:25, delay=0.42, delays=0.06/0.03/0.01/0.32, dsn=2.6.0, status=sent (250 2.6.0  <20140919070936.5609F60062@SIT-NAGIOS> Queued mail for delivery)
Sep 19 15:09:36 SIT-NAGIOS postfix/qmgr[976]: 5609F60062: removed
------
Sep 19 15:09:36 SIT-NAGIOS postfix/pickup[10150]: 6CE616008B: uid=106 from=<nagios>
Sep 19 15:09:36 SIT-NAGIOS postfix/cleanup[29168]: 6CE616008B: message-id=<20140919070936.6CE616008B@SIT-NAGIOS>
Sep 19 15:09:36 SIT-NAGIOS postfix/qmgr[976]: 6CE616008B: from=<nagios@SIT-NAGIOS>, size=619, nrcpt=1 (queue active)
Sep 19 15:09:36 SIT-NAGIOS postfix/smtp[29177]: 6CE616008B: to=<itnetworksupport@xxx.com>, relay=sit-smtp01.xxx.com[10.16.0.142]:25, delay=0.35, delays=0.01/0.03/0.02/0.29, dsn=2.6.0, status=sent (250 2.6.0  <20140919070936.6CE616008B@SIT-NAGIOS> Queued mail for delivery)
Sep 19 15:09:36 SIT-NAGIOS postfix/qmgr[976]: 6CE616008B: removed

It sends both mails to itnetworksupport@xxx, (contact itcim has the just itnetworksupport@xxx.com for it email., contact itnetworksupport also has just itnetworksupport@xxx.com).


Starting with the service definition:

Code: Select all

define service{
	use			generic-service-cim
	host_name		SRMSHOST08A
	service_description	C Drive
	check_command		check_snmp_disk!public!C!70%!80%
	notifications_enabled	1
}

No specific contacts defined here, just the template generic-service-cim

Template generic-service-cim:

Code: Select all

define service{
	name				generic-service-cim
	use				generic-service-template
	contacts			itcim
	#contact_groups			cim-admins
}

Has contacts itcim. This accounts for the one of the two notifications. It is missing the register 0 line. It includes the template generic-service-template:

Code: Select all

define service{
        name                            generic-service-template ; The 'name' of this service template
        active_checks_enabled           1       ; Active service checks are enabled
        passive_checks_enabled          1       ; Passive service checks are enabled/accepted
        parallelize_check               1       ; Active service checks should be parallelized (disabling this can lead to major performance problems)
        obsess_over_service             1       ; We should obsess over this service (if necessary)
        check_freshness                 0       ; Default is to NOT check service 'freshness'
        notifications_enabled           0       ; Service notifications are enabled
        event_handler_enabled           1       ; Service event handler is enabled
        flap_detection_enabled          1       ; Flap detection is enabled
        failure_prediction_enabled      1       ; Failure prediction is enabled
        process_perf_data               1       ; Process performance data
        retain_status_information       1       ; Retain status information across program restarts
        retain_nonstatus_information    1       ; Retain non-status information across program restarts
		notification_interval           10		; Only send notifications on status change by default.
		is_volatile                     0
		check_period                    24x7
		normal_check_interval           5
		retry_check_interval            1
		max_check_attempts              6
		notification_period             24x7
		notification_options            w,u,c,r
        register                        0       ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL SERVICE, JUST A TEMPLATE!
        }

This is where the chain ends and does not account for a second mail for the C drive service alert. If it was set to use 'generic-service' instead of 'generic-service-template' that would explain things. But if this config hasn't been modified what is causing the notification to itnetworksupport on the current D drive check isn't in this config.

Do you have a serviceescalation defined somewhere in your config? If you do, post it (or them if there are more than one). It would be something that would cause an additional notification and would be outside the inheritance issue. That config above does not account for a second sending to itnetwork support.


As for SIT-NAS2 - all the service checks are empty of contact and contact group settings and all include the generic-service template. The generic service template has a contacts of null and a contact_groups of admins. Only admins are going to receive these alerts. The admins group consists of itnetworksupport. So, for each alert of a SIT-NAS2 service, you will receive 1 email to itnetworksupport due to its members in the admins group. This matches with the logs. Everything matches up with what I'd expect here.

For 6800SRETDB, you define neither contact or contact group in the end service. All services include the template generic-service-dba. generic-service-dba containts a contact_groups null line, and a contacts of itdba. This means regardless of whats set further up the chain, the 6800SRETDB services will only notify the itdba contact (which currently has its email set to itnetworksupport@XXX). This matches with the single notification showing in the logs. Everything matches up with what I'd expect here.

[sreinhardt]

Thanks for the awesome post again millissa. OP please let us know how the above instructions work for you.

[hata_ph]

hi milisa,

No service escalation defined in my setup. And I only modify the email domain on the attached files.
I attached my whole /etc/nagios3 directory for further inspection minus the sensitive data...

[abrist]

OP, do you have other responses regarding the rest of millisa's inquiries?

[hata_ph]

OP, do you have other responses regarding the rest of millisa's inquiries?

HI abrist,

As i already mention i did not configure service escalation on my nagios.

[Nagios Support]

hata_ph,

We will have to do some more digging into this and will get back to you within the next 24 hours.