Critical alarms not showing up in service view.

[jdelivuk]

Hello everyone,

I have been having an issue with nagios alarms. It seems to mostly be effecting one service check for snmp drive space on windows servers. The alarm will trigger and if I go to a notifications report it will show me it's critical or warning, but on the service overview it doesn't show anything and we will never receive an email notification. I have tried searching but the only thing I'm coming up with is misconfigured email. Our checks usually operate fine, but this one is goofy and it even seems like nagios knows what it's doing. I recently upgraded to 3.3.1 just to make sure this wasn't addressed in a software update.

Thanks for your help
//John

[agriffin]

You mentioned that you can see the notifications fine in the notification log, but are there any state changes in the event log?

[jdelivuk]

Thanks for the reply agriffin.

I am seeing it fail and recover in the event log, where as notifications I only see it fail and drop. It occurred around 4 AM which is when we have to reboot our BO servers. We don't setup a special window in monitoring for this, do you think we're overloading it with all of the status changes? It's about 9 hosts and around 90 service checks that bounce states in 5 minutes.

[agriffin]

90 checks in 5 minutes isn't really a whole lot, so it's probably not that. You probably haven't configured Nagios to receive notifications for recoveries. What do your notification_options look like for these services or their templates? If there isn't a letter 'r' there that would explain it.

[jdelivuk]

I use the same template for majority of my checks. It has notification_options set to w,u,c,r.The only check is drive space that just stops sending w/o any flapping notification. I'm starting to wonder if there is something wrong with the way the service is configured. I'll dig a little deeper and let you know if I find anything

[wormfishin]

To clarify, the notifications page is saying it was sent out? My first thought was maybe it was due to soft vs hard state.

But if it's showing on the notifications page I don't think that'd be it. Can you review the maillog file to see if the email was sent out from your server to the relay?

[jdelivuk]

[04-26-2012 03:32:17] SERVICE NOTIFICATION: nagiosadmin;P-SSI-3S5-1;G:\Drive Space;WARNING;notify-service-by-email;WARNING : Percent Used : 99.4%, Total : 273.4 GiB, Used : 271.9 GiB, Free : 1.4 GiB
[04-26-2012 03:32:17] SERVICE ALERT: P-SSI-3S5-1;G:\Drive Space;WARNING;HARD;3;WARNING : Percent Used : 99.4%, Total : 273.4 GiB, Used : 271.9 GiB, Free : 1.4 GiB
[04-26-2012 03:30:18] SERVICE ALERT: P-SSI-3S5-1;G:\Drive Space;WARNING;SOFT;2;WARNING : Percent Used : 99.4%, Total : 273.4 GiB, Used : 271.9 GiB, Free : 1.4 GiB
[04-26-2012 03:28:17] SERVICE ALERT: P-SSI-3S5-1;G:\Drive Space;WARNING;SOFT;1;WARNING : Percent Used : 99.2%, Total : 273.4 GiB, Used : 271.4 GiB, Free : 1.9 GiB

So here is an excerpt from my log. I'm I'm starting to think it's flapping shutting off the notifications. It looks like it's going down and recovering 2 - 3 times when this occurs. I'm going to disable flap detection to see if that helps on these services.

[jdelivuk]

Actually just found this

damn it was hiding well....

[04-26-2012 04:02:17] SERVICE FLAPPING ALERT: P-SSI-3S5-1;G:\Drive Space;STARTED; Service appears to have started flapping (23.2% change >= 20.0% threshold)

[agriffin]

Good to see you got it figured out.