Volatile service, notification when the check output change

[rebug]

Hi,

I have some services that I need to be inform when the check output change on the same state (critical)
I read that volatile service can be convinient for this purpose.

My goal is simple:

I have a check testing disk health on my hosts. When a disk fail, the state is critical mentioning which disk is failing.
I want to aknowlegde this until I change the disk, but I need to be alerted if the output change (typically if an another disk fail)
The perfect solution would be that when the output change, the acknowledgment is deleted and I'm able to detect that.
I'm not sure this is possible. Right now I'm just trying to be alerted that the output changed but even with the "is volatile" option set to On I don't get anything.

Thank's you.

[ssax]

Yes, you would want to use volatile services for this. The problem is the acknowledging, once you acknowledge it the notifications will be suppressed (even with is_volatile being set and the output changing) because acknowledgments are only removed when the state changes or if an OK is received (depending on the ack settings). There isn't currently a way around that given the existing functionality.

It could be done but you would essentially need to code it into the plugin to check if the state doesn't change but the output does and that the service is currently acknowledged, before returning the result it would remove the ack:

https://assets.nagios.com/downloads/nag ... and_id=117

[rebug]

Thank you.

I made a event handler script that check the last output with the current output and if they are different break the acknowledgement and enable notification for this service.

[ssax]

I initially thought of that but I think because the event handler runs after the check, this would occur:

- Disk C Failure
- Event Handler Runs
- Acknowledged
- Drive D Failure
- Event Handler Runs removes Acknowledgement
- No drive D notification because the event handler runs after
- Drive D Failure
- Now you'd get the notification

Without is_volatile set on the service it also wouldn't run the standard event handler on either of those Drive D failures because event handlers only run on state changes (or in SOFT states/recoveries) without is_volatile being set.

Code: Select all

When Are Event Handlers Executed?

Event handlers are executed when a service or host:

    Is in a SOFT problem state
    Initially goes into a HARD problem state
    Initially recovers from a SOFT or HARD problem state 

Taken from here:

https://assets.nagios.com/downloads/nag ... dlers.html

[rebug]

I don't see what do you mean by that.
With is_volatile set and event handler enabled, this occur (tested).

Initial check of disk: OK
Another check : Critical "1 disk failed" -> event handler launch my script and log the output.
Acknowledgement.
Another check : Critical "1 disk failed" -> my script detect no change in the output, nothing happen
Another check : Critical "2 disk failed" -> output change detection, sending ENABLE_SVC_NOTIFICATIONS and REMOVE_SVC_ACKNOWLEDGEMENT on this service.

[ssax]

That's what will happen, the problem is this:

Initial check of disk: OK
Another check : Critical "1 disk failed" -> event handler launch my script and log the output.
Acknowledgement.
Another check : Critical "1 disk failed" -> my script detect no change in the output, nothing happen
Another check : Critical "2 disk failed" -> output change detection, sending ENABLE_SVC_NOTIFICATIONS and REMOVE_SVC_ACKNOWLEDGEMENT on this service. *** You should not receive this notification because of when the event handler runs ***

Another check would need to come in for you to be notified about it.

[rebug]

I see.

I can use SEND_CUSTOM_SVC_NOTIFICATION to immediately send a notification.

[ssax]

That should meet your needs then, just have the event handler do that right after removing the ack. Let us know when we're okay to lock this up and mark it as resolved.

[rebug]

That should meet your needs then, just have the event handler do that right after removing the ack. Let us know when we're okay to lock this up and mark it as resolved.

You can mark as resolved

[scottwilkerson]

That should meet your needs then, just have the event handler do that right after removing the ack. Let us know when we're okay to lock this up and mark it as resolved.

You can mark as resolved

Locking thread