eventhandler executions and notifications

[pixelgrunt]

I've been working with Nagios for years.. I've always used it for passive monitoring of hosts and services, and things have gone so well that I've implemented Nagios Core at my past 3 workplaces with widespread approval and acceptance from mgmt and peers.

Now, I'm getting the itch (as well as mgmt prodding) to take it a step further. One of the ways we're looking to expand is with proactive eventhandlers. I have successfully configured a basic eventhandler to invoke a remote httpd service restart through NRPE when a problem is detected. The problem is that no one is notified. From reading the documentation, it seems that this was the intent. However, my fellow admins, mgmt, and developers not only want the service to be restarted as quickly as possible, but they also want to know that it happened. I know, we're a needy bunch

So, I've tried passing nagios macros such as $NOTIFICATIONRECIPIENTS$, $CONTACTEMAIL$, as well as others along to my basic event handler with the intent of sending out an email notification similar to the /bin/mail command used by notify-host-by-email or notify-service-by-email. It seems that these macros aren't populated when the eventhandler is called, and so those variables are empty when the eventhandler is executed.

So, am I going down the right path here?

Is everyone else content with not knowing when a service is bounced by an eventhandler?

Taking a step back, is there some other means by which I can remotely bounce a service when needed AND get a notification email sent to a contactgroup informing them that the action occurred?

Thanks in advance, and I hope to be able to help out on this forum in the future.

[jsmurphy]

Yeah a lot of macros aren't accessible to the event handler, you can see them here: http://nagios.sourceforge.net/docs/3_0/macrolist.html

If you want notifications, maybe the best way to do this would be to configure the notification handler to be your eventhandler? Create a wrapper script that does both? Might be worth while using the $SERVICENOTIFICATIONNUMBER$ and $HOSTNOTIFICATIONNUMBER$ to prevent it from executing the recovery job more than once?

I think the reason why most people are content not getting notifications for eventhandlers is that they are executing the eventhandler so that they don't get a notification. There's no reason however why combining the notification with the eventhandler with the notification handler shouldn't work for you?

[pixelgrunt]

Thanks for the help! I was looking at the macro reference page that you listed, but I clearly wasn't reading all of it because I didn't notice the big column that indicated whether the macro was available to eventhandlers.

I like your idea of going further upstream and potentially modifying the notification handler. I think I may create a duplicate, name it accordingly, modify it to notify and restart the remote service, and use it for the few hosts/services where it's needed.

If/when I get it working, I'll posts relevant configs and info.

Thanks again!

[jsmurphy]

Glad I could help!

[pixelgrunt]

It's nice to have a new road to trod, even if it hasn't taken me anywhere yet.

I attempted to merge my eventhandler script with the default 'notify-service-by-email' command in a bash script wrapper, and then define it with the 'service_notification_commands' in a contact template. I thought this would have made the macros listed in the service notifications column of the macrolist (http://nagios.sourceforge.net/docs/3_0/macrolist.html) available.

It didn't.

I had my script output the arguments used to invoke it to a logfile in /tmp, and here is the tally of what was available vs. what was not:

Available: $HOSTALIAS$, $HOSTADDRESS$, $SERVICESTATE$, $SERVICESTATETYPE$, $SERVICEDESC$, $SERVICEATTEMPT$, $LONGDATETIME$, and $SERVICEOUTPUT$.

Unavailable: $NOTIFICATIONTYPE$, $CONTACTEMAIL$, $SERVICEACKAUTHOR$, $NOTIFICATIONAUTHOR$, $SERVICEACKCOMMENT$, and $NOTIFICATIONCOMMENT$.

You may have noticed that this is a superset of the macros used for the default 'notify-service-by-email' command. I assumed that once I defined my new bash script with a command object in a .cfg file, the macros would be available. After all, they work for notify-service-by-email. Looking at the macrolist from the docs, it seems that the subset of macros available and those that didn't work fall in line nicely with the service check column, instead of the service notification column that I was expecting.

I'm really at a loss why so many macros aren't available to my script, even though they are available to the /usr/bin/mail command invoked in the default notify-service-by-email command. The $CONTACTEMAIL$ macro is particularly important to my initial goal of sending out notifications when a service is automatically restarted by NRPE.

Am I heading down the right path?

Thanks again for taking the time to point me in the right direction.

[jsmurphy]

That's... weird... can I see the command definition you've created and how you've applied it to a service? There's no reason I can think of why any of that stuff would be unavailable to a notification command.