Page 2 of 3

Re: Not getting Email alerts

Posted: Fri Dec 07, 2018 12:16 pm
by shamrozkadiwal
lmiltchev wrote:@shamrozkadiwal, can you zip up the /usr/local/nagios directory and PM the zip file to me or any other member of the Nagios Support team? Thank you!
Please check your inbox

Re: Not getting Email alerts

Posted: Fri Dec 07, 2018 12:40 pm
by lmiltchev
I sent you a PM. Thanks!

Re: Not getting Email alerts

Posted: Fri Dec 07, 2018 2:55 pm
by shamrozkadiwal
benjaminsmith wrote:Hello shamrozkadiwal,

Let's change the debug level settings to capture notification data, and then put adxp011a into a down state and then back up (recovery state). We will then take a look at the log file for this host and review the state changes and whether or not notifications were sent.

1. Open up the nagios configuration file /usr/local/nagios/etc/nagios.cfg, and set debug_level=32, and then re-start Nagios.

2. Put the adxpo11a into a down state and then a recovery.

2. Post or PM the nagios log data or the contents of the log file at /usr/local/nagios/var
nagios.log
. You'll want to search the log data for host notification entries for adxp011a. The perl statement at the end will convert the unix timestamp into human readable text. Are notifications beings sent? If so, to the correct contacts?

Code: Select all

cat nagios.log | grep adxp011a  |  perl -pe 's/(\d+)/localtime($1)/e'
Let me know what you find out.

Main Configuration File Options
https://assets.nagios.com/downloads/nag ... gmain.html
I noticed that debug_log was 0 so I changed it to 32 as you said. I have attached whole nagios log file

Re: Not getting Email alerts

Posted: Fri Dec 07, 2018 5:39 pm
by npolovenko
@shamrozkadiwal, Looks like the archive you sent to @lmiltchev doesn't have all the information we needed. Your configs and objects.cache file is in a custom directory /usr/nagios. Can you zip that whole folder and send it to me in a PM?

Re: Not getting Email alerts

Posted: Mon Dec 10, 2018 10:20 am
by shamrozkadiwal
npolovenko wrote:@shamrozkadiwal, Looks like the archive you sent to @lmiltchev doesn't have all the information we needed. Your configs and objects.cache file is in a custom directory /usr/nagios. Can you zip that whole folder and send it to me in a PM?
I have sent you PM anther tar.gz file

Re: Not getting Email alerts

Posted: Mon Dec 10, 2018 4:20 pm
by benjaminsmith
Hello @shamrozkadiwal,

In your service escalations, you've configured the disk-contacts group to receive email alerts for W,C,R and the infrastructure-pager contact group for C,R.

Code: Select all

## Service Escalations
define serviceescalation {
        hostgroup_name          data-servers,processing-servers,!no_data_drive
        host_name               aft001,eris,ftp01,ftp02,!app403,!app012
        service_description     Disk Space Data
        contact_groups          disk-contacts
        first_notification      1
        last_notification       0
        notification_interval   180
        escalation_options      w,c,r
}
define serviceescalation {
        hostgroup_name          data-servers,!no_data_drive
        service_description     Disk Space Data
        contact_groups          infrastructure-pager
        first_notification      1
        last_notification       0
        notification_interval   120
        escalation_options      c,r
}
One thing I noticed is that your the configuration settings for the member in disk-contacts, production and setup, have the service notification options set to none (n) and the host notification options set to unreachable (u).

Code: Select all

define contact {
	contact_name	production
	alias	Production
	host_notification_period	never
	service_notification_options	n
	host_notification_options	u
	service_notification_commands	notify-by-email
	host_notification_commands	host-notify-by-email
	email	production@blank.com
	host_notifications_enabled	1
	service_notifications_enabled	1
	can_submit_commands	1
	retain_status_information	1
	retain_nonstatus_information	1
	}

	define contact {
	contact_name	setup
	alias	Setup
	host_notification_period	never
	service_notification_options	n
	host_notification_options	u
	service_notification_commands	notify-by-email
	host_notification_commands	host-notify-by-email
	email	setup@blank.com
	host_notifications_enabled	1
	service_notifications_enabled	1
	can_submit_commands	1
	retain_status_information	1
	retain_nonstatus_information	1
	}
While the members of infrastructure-pager are as so:

Code: Select all

	define contact {
	contact_name	bhswar
	alias	Bhagwat Swarup
	service_notification_period	24x7
	host_notification_period	24x7
	service_notification_options	c,r
	host_notification_options	d,r
	service_notification_commands	notify-by-email,notify-by-pager
	host_notification_commands	host-notify-by-email,host-notify-by-pager
	email	bhagwat.swarup2@blank.com
	pager	5127016714@blank.net
	host_notifications_enabled	1
	service_notifications_enabled	1
	can_submit_commands	1
	retain_status_information	1
	retain_nonstatus_information	1
	}

define contact {
	contact_name	bhswar-email
	alias	Bhagwat Swarup - Email Only
	service_notification_period	24x7
	host_notification_period	24x7
	service_notification_options	w,u,c,r
	host_notification_options	d,u,r
	service_notification_commands	notify-by-email
	host_notification_commands	host-notify-by-email
	email	bhagwat.swarup2@blank.com
	host_notifications_enabled	1
	service_notifications_enabled	1
	can_submit_commands	1
	retain_status_information	1
	retain_nonstatus_information	1
	}
Please try re-setting the notification options, W,C,U, for the disk-contacts group and put adxp011a into a down state, and then review the log file as mentioned earlier to verify notifications were sent.

Re: Not getting Email alerts

Posted: Wed Dec 12, 2018 11:40 am
by shamrozkadiwal
@Benjaminsmith, I didn't understand where exactly or file I need to make a change.
In your service escalations, you've configured the disk-contacts group to receive email alerts for W,C,R and the infrastructure-pager contact group for C,R.
We have purposely set W,C,R just for email notification and C,R for infrastructure-pager. We don't want W on the infrastructure-pager.

Re: Not getting Email alerts

Posted: Wed Dec 12, 2018 11:50 am
by shamrozkadiwal
Also wondering, why do we receive the EMAIL notification for adx800, but not for adxp011a when we have the similar configuration for both :shock:

Re: Not getting Email alerts

Posted: Thu Dec 13, 2018 3:43 pm
by benjaminsmith
Hello @shamrozkadiwal,
Also wondering, why do we receive the EMAIL notification for adx800, but not for adxp011a when we have the similar configuration for both
I'm trying to figure that out as well. For Nagios Core, you are running a large system and there are many layers to the configuration files. Currently, adxp011a is not set to notify on a warning state. You can see that in the nagios.debug log:

Code: Select all

[1544453107.040894] [032.0] [pid=1583] ** Service Notification Attempt ** Host: 'adxp011a', Service: 'Disk Space Data', Type: 0, Options: 0, Current State: 1, Last Notification: Wed Dec 31 18:00:00 1969
[1544453107.040915] [032.1] [pid=1583] We shouldn't notify about WARNING states for this service.
[1544453107.040919] [032.0] [pid=1583] Notification viability test failed.  No notification will be sent out.
If you look at the service definition for Disk Space Data on adxp011a you'll see the notifications options are only for critical and recovery. The debug logs that I have do not show this host going into a state that would trigger notifications.

Code: Select all

define service {
	host_name	adxp011a
	service_description	Disk Space Data
	check_period	no-sun
	check_command	check_nt_disk-large!D!90!95
	contact_groups	infrastructure-pager
	notification_period	24x7
	initial_state	o
	check_interval	20.000000
	retry_interval	3.000000
	max_check_attempts	5
	is_volatile	0
	parallelize_check	1
	active_checks_enabled	1
	passive_checks_enabled	1
	obsess_over_service	1
	event_handler_enabled	1
	low_flap_threshold	0.000000
	high_flap_threshold	0.000000
	flap_detection_enabled	1
	flap_detection_options	o,w,u,c
	freshness_threshold	0
	check_freshness	0
	notification_options	c,r
	notifications_enabled	1
	notification_interval	1440.000000
	first_notification_delay	0.000000
	stalking_options	n
	process_perf_data	1
	failure_prediction_enabled	1
	retain_status_information	1
	retain_nonstatus_information	1
	}
Keep the debug settings as before, then update your service definition for adxp011a, put the host in a warning state by adjusting the check variables, restart and then examine the debug log to see if the notifications were sent

Re: Not getting Email alerts

Posted: Thu Dec 20, 2018 11:47 am
by shamrozkadiwal
Please keep this thread open. I am also debugging from my end too