Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Notifications for services not going out.

https://support.nagios.com/forum/viewtopic.php?t=36908

Page 2 of 3

Re: Notifications for services not going out.

Posted: Thu Feb 04, 2016 2:51 pm
by [email protected]
Correct all host notifications are being update but not emailing. Actually now the services notifications are working! Sort of it missed the last alert(in the notifications list) but did email it to me.

Will attach.

Here's the email
[email protected]
11:41 AM (10 minutes ago)

to reta
***** Nagios Monitor XI Alert *****

Notification Type: PROBLEM
Host: saucelabs.dev.meetearnest.com
State: DOWN
Address: 10.0.7.182
Info: CRITICAL - Socket timeout after 10 seconds

Date/Time: Thu Feb 4 11:41:18 PST 2016

Re: Notifications for services not going out.

Posted: Thu Feb 04, 2016 3:07 pm
by rkennedy
Warning: Host 'admin.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'agiloft.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'client-payment-configuration.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'loan-payment.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'localhost' has no default contacts or contactgroups defined!
Warning: Host 'looker.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'mobile.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'node-predict.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'production-nodejs-app1' has no default contacts or contactgroups defined!
Warning: Host 'production-nodejs-app2' has no default contacts or contactgroups defined!
Warning: Host 'production-nodejs-app3' has no default contacts or contactgroups defined!
Warning: Host 'production-nodejs-app5' has no default contacts or contactgroups defined!
Warning: Host 'production-nodejs-app7' has no default contacts or contactgroups defined!
Warning: Host 'saucelabs.dev.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'vault.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'vault.staging.meetearnest.com' has no default contacts or contactgroups defined!
It looks like the only host with a contact defined is production-node-js-app6.

Code: Select all

define host {
	host_name			production-node-js-app6
	use				generic-host
	alias				production-node-js-app6
	display_name			Production API
	address				10.0.7.233
	check_command			check_http!!!!!!!!
	max_check_attempts		3
	check_interval			2
	retry_interval			1
	active_checks_enabled		1
	passive_checks_enabled		1
	check_period			24x7
	process_perf_data		1
	retain_status_information	1
	retain_nonstatus_information	1
	contacts			reta
	notification_interval		5
	notification_period		24x7
	notification_options		d,u,r,
	register			1
	}
Can you add the 'reta' contact to all of the remaining hosts?

Re: Notifications for services not going out.

Posted: Thu Feb 04, 2016 3:53 pm
by [email protected]
let me send u the updated profile

Re: Notifications for services not going out.

Posted: Thu Feb 04, 2016 5:19 pm
by rkennedy
Can you rerun the verification, and post the output?

Code: Select all

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Re: Notifications for services not going out.

Posted: Thu Feb 04, 2016 6:32 pm
by [email protected]
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.1.1
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-19-2015
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Warning: Service 'Admin Service' on host 'admin.internal.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'Agiloft Service' on host 'agiloft.internal.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'Node Client Payment Configuration Serice' on host 'client-payment-configuration.internal.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'Node Loan Payment' on host 'loan-payment.internal.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'Looker Service' on host 'looker.internal.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'Node Mobile Service' on host 'mobile.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'Node Predict Service' on host 'node-predict.internal.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'SauceLabs SC Process' on host 'saucelabs.dev.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Checked 61 services.
Warning: Host 'admin.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'agiloft.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'client-payment-configuration.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'loan-payment.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'localhost' has no default contacts or contactgroups defined!
Warning: Host 'looker.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'mobile.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'node-predict.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'production-nodejs-app4' has no default contacts or contactgroups defined!
Warning: Host 'production-nodejs-app8' has no default contacts or contactgroups defined!
Warning: Host 'production-nodejs-app9' has no default contacts or contactgroups defined!
Warning: Host 'production-task-runner-1' has no default contacts or contactgroups defined!
Warning: Host 'production-task-runner-2' has no default contacts or contactgroups defined!
Warning: Host 'production-task-runner-3' has no default contacts or contactgroups defined!
Warning: Host 'production-task-runner-5' has no default contacts or contactgroups defined!
Warning: Host 'production-task-runner-sched4' has no default contacts or contactgroups defined!
Warning: Host 'saucelabs.dev.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'vault.internal.meetearnest.com' has no default contacts or contactgroups defined!
Warning: Host 'vault.staging.meetearnest.com' has no default contacts or contactgroups defined!
Checked 19 hosts.
Checked 5 host groups.
Checked 0 service groups.
Checked 3 contacts.
Checked 3 contact groups.
Checked 128 commands.
Checked 9 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 19 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 9 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 27
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
[root@ip-172-16-3-76 var]#


These were defined

Re: Notifications for services not going out.

Posted: Thu Feb 04, 2016 6:38 pm
by [email protected]
I updated using tool:

now

hecking objects...
Warning: Service 'Admin Service' on host 'admin.internal.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'Agiloft Service' on host 'agiloft.internal.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'Node Client Payment Configuration Serice' on host 'client-payment-configuration.internal.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'Node Loan Payment' on host 'loan-payment.internal.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'Looker Service' on host 'looker.internal.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'Node Mobile Service' on host 'mobile.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'Node Predict Service' on host 'node-predict.internal.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Warning: Service 'SauceLabs SC Process' on host 'saucelabs.dev.meetearnest.com' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Checked 61 services.
Checked 19 hosts.
Checked 5 host groups.
Checked 0 service groups.
Checked 3 contacts.
Checked 3 contact groups.
Checked 128 commands.
Checked 9 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 19 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 9 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 8
Total Errors: 0

Re: Notifications for services not going out.

Posted: Thu Feb 04, 2016 7:27 pm
by [email protected]
Now just email is not going out, I see the notifications logged in Nagios.

Re: Notifications for services not going out.

Posted: Thu Feb 04, 2016 8:37 pm
by [email protected]
1454636055] SERVICE NOTIFICATION: reta;saucelabs.dev.meetearnest.com;Disk Usage;CRITICAL;xi_service_notification_handler;3

[1454636098] SERVICE ALERT: saucelabs.dev.meetearnest.com;Disk Usage;OK;HARD;3;DISK OK - free space: / 1065 MB (13% inode=43%):
[1454636098] SERVICE NOTIFICATION: nagiosadmin;saucelabs.dev.meetearnest.com;Disk Usage;OK;xi_service_notification_handler;DISK OK - free space: / 1065 MB (13% inode=43%):
[1454636098] SERVICE NOTIFICATION: reta;saucelabs.dev.meetearnest.com;Disk Usage;OK;xi_service_notification_handler;DISK OK - free space: / 1065 MB (13% inode=43%):

Seeing no email

Re: Notifications for services not going out.

Posted: Thu Feb 04, 2016 9:08 pm
by Box293
Run this command in an ssh session on your XI host:

Code: Select all

tail -f /usr/local/nagiosxi/var/eventman.log
Now go to the host or service object and from the Advanced tab force a notification to be sent. Please send us the full output from the ssh session.

Re: Notifications for services not going out.

Posted: Fri Feb 05, 2016 1:26 pm
by [email protected]
Nagios Log :

[1454696660] SERVICE ALERT: saucelabs.dev.meetearnest.com;Disk Usage;CRITICAL;SOFT;1;crit 1
[1454696667] SERVICE ALERT: saucelabs.dev.meetearnest.com;Disk Usage;CRITICAL;SOFT;2;2
[1454696673] SERVICE ALERT: saucelabs.dev.meetearnest.com;Disk Usage;CRITICAL;HARD;3;3
[1454696673] SERVICE NOTIFICATION: nagiosadmin;saucelabs.dev.meetearnest.com;Disk Usage;CRITICAL;xi_service_notification_handler;3
[1454696673] SERVICE NOTIFICATION: reta;saucelabs.dev.meetearnest.com;Disk Usage;CRITICAL;xi_service_notification_handler;3
[1454696705] SERVICE ALERT: saucelabs.dev.meetearnest.com;Disk Usage;OK;HARD;3;DISK OK - free space: / 1047 MB (13% inode=43%):

EVENT LOG:

User has mobile text notifications disabled...

PROCESS EVENT: ID=5369, SOURCE=2, TYPE=2, TIME=2016-02-05 10:24:34
*** GLOBAL HANDLER...
Array
(
[event_id] => 5369
[event_source] => 2
[event_type] => 2
[event_time] => 2016-02-05 10:24:34
[event_meta] => Array
(
[notification-type] => service
[contact] => reta
[contactemail] => [email protected]
[type] => PROBLEM
[escalated] => 0
[author] =>
[comments] =>
[host] => saucelabs.dev.meetearnest.com
[hostaddress] => 10.0.7.182
[hostalias] => Sauce Labs
[hostdisplayname] => Sauce Labs
[service] => Disk Usage
[hoststate] => UP
[hoststateid] => 0
[servicestate] => CRITICAL
[servicestateid] => 2
[lastservicestate] => CRITICAL
[lastservicestateid] => 2
[servicestatetype] => HARD
[currentattempt] => 3
[maxattempts] => 3
[serviceeventid] => 1020
[serviceproblemid] => 409
[serviceoutput] => 3
[longserviceoutput] =>
[datetime] => Fri Feb 5 10:24:33 PST 2016
)

[logging_enabled] => 1
)
*** GLOBAL HANDLER (snmptrapsender)...
Array
(
[event_id] => 5369
[event_source] => 2
[event_type] => 2
[event_time] => 2016-02-05 10:24:34
[event_meta] => Array
(
[notification-type] => service
[contact] => reta
[contactemail] => [email protected]
[type] => PROBLEM
[escalated] => 0
[author] =>
[comments] =>
[host] => saucelabs.dev.meetearnest.com
[hostaddress] => 10.0.7.182
[hostalias] => Sauce Labs
[hostdisplayname] => Sauce Labs
[service] => Disk Usage
[hoststate] => UP
[hoststateid] => 0
[servicestate] => CRITICAL
[servicestateid] => 2
[lastservicestate] => CRITICAL
[lastservicestateid] => 2
[servicestatetype] => HARD
[currentattempt] => 3
[maxattempts] => 3
[serviceeventid] => 1020
[serviceproblemid] => 409
[serviceoutput] => 3
[longserviceoutput] =>
[datetime] => Fri Feb 5 10:24:33 PST 2016
)

[logging_enabled] => 1
)
Got XI user id for contact 'reta': 2
An email notification will be sent...

Email Notification Data:

Array
(
[from] => Nagios XI <root@localhost>
[to] => [email protected]
[subject] => PROBLEM Service Alert - saucelabs.dev.meetearnest.com/Disk Usage is CRITICAL
[high_priority] => 0
[message] => ***** Nagios XI Alert *****

Nagios has detected a problem with this service.

Notification Type: PROBLEM

Service: Disk Usage
Host: saucelabs.dev.meetearnest.com
Address: 10.0.7.182
State: CRITICAL
Info:
3
Date/Time: 2016-02-05 10:24:35

Respond: http://172.16.3.76/nagiosxi/rr.php?uid= ... 2a8d18604d
Nagios URL: http://172.16.3.76/nagiosxi/

)


User has mobile text notifications disabled...

PROCESS EVENT: ID=5370, SOURCE=2, TYPE=1, TIME=2016-02-05 10:24:34
*** GLOBAL HANDLER...
Array
(
[event_id] => 5370
[event_source] => 2
[event_type] => 1
[event_time] => 2016-02-05 10:24:34
[event_meta] => Array
(
[handler-type] => service
[host] => saucelabs.dev.meetearnest.com
[service] => Disk Usage
[hostaddress] => 10.0.7.182
[hoststate] => UP
[hoststateid] => 0
[hosteventid] => 1016
[hostproblemid] => 0
[servicestate] => CRITICAL
[servicestateid] => 2
[lastservicestate] => CRITICAL
[lastservicestateid] => 2
[servicestatetype] => HARD
[currentattempt] => 3
[maxattempts] => 3
[serviceeventid] => 1020
[serviceproblemid] => 409
[serviceoutput] => 3
[longserviceoutput] =>
[servicedowntime] => 0
)

[logging_enabled] => 1
)
*** GLOBAL HANDLER (snmptrapsender)...
Array
(
[event_id] => 5370
[event_source] => 2
[event_type] => 1
[event_time] => 2016-02-05 10:24:34
[event_meta] => Array
(
[handler-type] => service
[host] => saucelabs.dev.meetearnest.com
[service] => Disk Usage
[hostaddress] => 10.0.7.182
[hoststate] => UP
[hoststateid] => 0
[hosteventid] => 1016
[hostproblemid] => 0
[servicestate] => CRITICAL
[servicestateid] => 2
[lastservicestate] => CRITICAL
[lastservicestateid] => 2
[servicestatetype] => HARD
[currentattempt] => 3
[maxattempts] => 3
[serviceeventid] => 1020
[serviceproblemid] => 409
[serviceoutput] => 3
[longserviceoutput] =>
[servicedowntime] => 0
)

[logging_enabled] => 1
)
SNMP TRAP SENDER NOT CONFIGURED!
.........tail: eventman.log: file truncated
.
PROCESSED 5 EVENTS
tail: eventman.log: file truncated
..
PROCESSED 3 EVENTS
All times are UTC-05:00
Page 2 of 3

Powered by phpBB® Forum Software © phpBB Limited