Page 1 of 2

Sending events when downtime ends

Posted: Mon Jul 14, 2014 2:28 am
by Fred Kroeger
Is there a way to trigger sending any active events when a scheduled downtime ends?
The scenario I have is that we want to suppress sending any events during a scheduled downtime but want an alert generated if the host/service is "not OK" when the downtime ends.

regards... Fred

Re: Sending events when downtime ends

Posted: Mon Jul 14, 2014 11:28 am
by slansing
This should happen by default, when nagios runs the next scheduled check on your hosts that were in downtime it should pick the critical up and push out a notification just as it normally would, is this not happening in your environment? It happens this way because in downtime your hosts will undoubtedly be in a critical or unreachable state unless they are not actually down, thus, if you were to send a notification immediately after leaving downtime and they were still coming back up you would essentially get a false notification.

Re: Sending events when downtime ends

Posted: Mon Jul 14, 2014 7:26 pm
by Fred Kroeger
Thanks - I need to do some further testing then as I am relying on a Global event handler instead of notifications. Does the Global event handler get triggered again as well as the notifications when it comes out of downtime?
I am using the check_dummy command to generate the critical.
My testing sequence is.
1) Put Host in scheduled downtime
2) Change the check_dummy parameter from 0 to 2 (Check interval 5min)
3) Apply config

In the event log I can see:
- Scheduled downtime starting
- Critical alert for Host
- Critical alert processed by Global event handler
- Scheduled downtime stopping

There are no further messages to indicate that the Global Event handler has received another event after that.
BTW - the event handler ignores any event that is in downtime, which is why I need to send another event if it is still active when it comes out of downtime.

Fred

Re: Sending events when downtime ends

Posted: Tue Jul 15, 2014 12:02 pm
by sreinhardt
Ya the event handler would not do this, simply because it is only going to get called when a state change occurs or if a host\service has isvolatile set making each return a state change. Unfortunately, I don't think simply setting a contact with notifications will truly do what you want without duplicating traps that are sent.

Re: Sending events when downtime ends

Posted: Fri Jul 18, 2014 2:57 am
by Fred Kroeger
Thanks Spenser
Would it be possible to setup the scheduled downtime start/end routines to trigger the event handler?
ie: When Downtime starts and ends, it send the current state.

The event handler I am using interfaces into HP's SM9 to generate tickets. So we now have a situation where we could have a critical event but nobody knows about it as we don't use the notifications. How does the Nagios IM handle this scenario?

reagrsd... Fred

Re: Sending events when downtime ends

Posted: Fri Jul 18, 2014 10:26 am
by abrist
If you were using the notification handlers, you could send a notification when downtime ends, parse the email and take action.
You could pass $HOSTDOWNTIME$ or $SERVICEDOWNTIME$ macros to your event handler script to check the objects downtime depth.

IM handles this through an integrated XI component, with considerable logic for these types of issues. I will have to look into the php more before I have a more specific answer.

Re: Sending events when downtime ends

Posted: Mon Jul 21, 2014 2:13 am
by Fred Kroeger
The event handler that HP provides works exactly as it should, in that it is passing the downtime status along with the state so that the end process can determine whether to take any action or not.
I don't think any condiderable logic is required for this - all I need is the event handler to be triggered when downtime ends. That way the current state will always be sent.

regards... Fred

Re: Sending events when downtime ends

Posted: Mon Jul 21, 2014 4:11 pm
by sreinhardt
The only way that I could think to do this, would be to create a contact specifically for it, use your event handler as the host and service notification command probably with some minor changes to how you call it, and finally set this contact to only ever notify on downtime actions. Unfortunately a pure event handler will not work, as they are only executed on state change, and downtime does not cause a state change the same as a check result will.

As abrist mentioned, there is a very large amount of logic with the IM component for XI. Essentially it collects and processes just about every actionable event within your nagios system, determines how it should act for both system and does so. Neither event handlers nor notifications have near this same level of logic in any form of a controllable or callable manor, that I know of. If it's not protected you are more than welcome to take a peek at the IM component and see if there is a hook or two you could implement with the snmptrap component to work better for you.

Re: Sending events when downtime ends

Posted: Thu Sep 25, 2014 12:44 am
by Fred Kroeger
I've had a bit more of a think about this and perhaps I can define my requirements a bit better.

Essentially, I would like Nagios to alert on any Non-OK Host or Service state that is active when a Host comes out of Downtime.

Because the notification was suppressed during the downtime, the Admins have no way of knowing if there was any alert apart from having to log in to Nagios to view the current status of the monitors for that host.

Is that possible? ..... Fred

Re: Sending events when downtime ends

Posted: Thu Sep 25, 2014 11:14 am
by Box293
After downtime ends this will happen automatically on the next notification_interval of the host or service object (as long as the object was not acknowledged).