Part 2: No Notifications for Specific Service Check

[mmartin]

Well, after I got the Notifications working for my one specific Service yesterday, today they are not working anymore for this particular Service...

Original Post "No Notifications for Specific Service Check" --> http://support.nagios.com/forum/viewtop ... =7&t=30919

I tried checking the same thing I did yesterday to get it working but it didn't seem to fix it. In case you didn't check out the link above from my post yesterday, the disk space was full on the blosxom partition and clearing out all the files in "/opt/blosxom/posts" seemed to fix the issue. But today the notifications are not working for this service anymore and I can't figure out why?

I know Notifications for other Services are working because I've been getting them all morning... The Service is a Passive service check which gets sent to Nagios using NSCA.

In the output below from "/var/log/messages" this is what Nagios logged after sending an "OK" status when the service was already in a "CRITICAL" state. And no Notifications were sent for it.

Passive Service Check Result from: /var/log/messages

Code: Select all

Jan 16 11:19:35 nagios nsca[1193]: Connection from 127.0.0.1 port 35024
Jan 16 11:19:35 nagios nsca[1193]: Handling the connection...
Jan 16 11:19:35 nagios nsca[1193]: SERVICE CHECK -> Host Name: 'Test-Nagios', Service Description: 'Test_Check-In', Return Code: '0', Output: 'OK: This is a TEST of the OK STATUS...'
Jan 16 11:19:35 nagios nsca[1193]: End of connection...
Jan 16 11:19:35 nagios nagios: EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;Test-Nagios;Test_Check-In;0;OK: This is a TEST of the OK STATUS...
Jan 16 11:19:44 nagios nagios: PASSIVE SERVICE CHECK: Test-Nagios;Test_Check-In;0;OK: This is a TEST of the OK STATUS...
Jan 16 11:19:44 nagios nagios: SERVICE ALERT: Test-Nagios;Test_Check-In;OK;SOFT;2;OK: This is a TEST of the OK STATUS...

I ended up changing up the configuration a bit from my previous post in the link above, but that didn't seem to help either.
Here is the Current Configuration for my Service:

Code: Select all

### Host Definition:
define host {
	host_name                  Test-Nagios
	alias                      Test-Nagios
	use                        pnpHost
	address                    127.0.0.1
	check_command              check-host-alive
	max_check_attempts         3
	check_period               24x7
	contact_groups             ME_ONLY
	notification_interval      120
	notification_period        24x7
	notification_options       d,u,r,f
	parents                    JWP4510R-HQ
}

### Service Template:
define service {
	name                       Test_Checks
	use                        generic-service,pnpService
	host_name                  Test-Nagios
	max_check_attempts         3
	normal_check_interval      3
	retry_check_interval       1
	notification_interval      30
	notification_period        24x7
	notification_options       w,u,c,r
	check_period               24x7
	contact_groups             ME_ONLY
	register                   0			; DONT REGISTER THIS DEFINITION - ITS A TEMPLATE
}

### Service Definition:
define service {
	service_description        Test_Check-In
	use                        Test_Checks
	contact_groups             ME_ONLY
	active_checks_enabled      0
	passive_checks_enabled     1
	flap_detection_enabled     0
	check_command              Test_Check-In!0	; Dummy Command since this is a Passive check (*receive only)
}

### Command Definition:
define command {
	command_name               Test_Check-In
	command_line               $USER1$/Test_RMC/Test_nagios
}

I also tried stopping nagios, deleting Nagios' "objects.cache", and then restarting Nagios, but that didn't help either...

Any other ideas for what I can try, any suggestions would be greatly appreciated.

Thanks in Advance,
Matt

[Box293]

Try setting the debug level on and then restart Nagios.

Code: Select all

sed -i 's/.*debug_level=.*/debug_level=-1/g' /usr/local/nagios/etc/nagios.cfg
service nagios restart

Then watch the log:
tail -f /usr/local/nagios/var/nagios.debug

Force a HARD state change (HARD Critical to OK).

While watching the debug log, what do you see happening?

This will turn debugging off:

Code: Select all

sed -i 's/.*debug_level=.*/debug_level=0/g' /usr/local/nagios/etc/nagios.cfg
service nagios restart

[mmartin]

Hey, thanks for the reply.

Ok cool, I'll give that a shot and see if I can see anything.

Thanks again for the reply, much appreciated!

Thanks,
Matt

[mmartin]

Ok, I think I may have figured out my issue.

So I had been under the assumption that when a Service changes from one state to another, no matter if it was a SOFT state change or a HARD state change that you are supposed to get a notification sent to you...

But after reading a bit more on the 2 states, I realized that doesn't seem to be the case... With this quote below from http://nagios.sourceforge.net/docs/3_0/statetypes.html I decided to force a HARD state change and see what happens, and what do ya know I got a notification...!

There are two state types in Nagios - SOFT states and HARD states. These state types are a crucial part of the monitoring logic, as they are used to determine when event handlers are executed and when notifications are initially sent out.

So if I want to get an email whenever the service changes states, no matter how many times it checked the service, I think I can just change the line below in my configuration from '3' to '1' and I should get a notification on the first state change of the service... I'm going to give it a test but I assume it will work that way..?

Code: Select all

### FROM THIS:
	max_check_attempts		3

### TO THIS:
	max_check_attempts		1

Thanks again for the help!

Thanks,
Matt

[Box293]

I think I can just change the line below in my configuration from '3' to '1' and I should get a notification on the first state change of the service

Exactly right.

[tgriep]

Glad that you solved it. I'll close the post for you.