Page 1 of 2
Mail notification (Ping service) attach traceroute
Posted: Mon Dec 08, 2014 6:16 am
by trazzdk
Hi guys,
We would like to have a traceroute attached to our Service: PING notifications.
So if we get a Ping warning / critical e-mail notification, a traceroute is attached as well.
Any ideas how to do this ?
/Trazz
Re: Mail notification (Ping service) attach traceroute
Posted: Mon Dec 08, 2014 9:41 pm
by Box293
There are two ways you could go about this.
1) Create an Event Handler that will do the traceroute when the ping check fails ... I've not done this before so it could get complicated.
2) Create a wrapper script that executes the ping check. Inside the wrapper script, if the ping check failed then it could perform the traceroute and append this information to the status information of the ping check. This information would then appear as part of the service detail AND would get sent with the notification.
Re: Mail notification (Ping service) attach traceroute
Posted: Tue Dec 09, 2014 6:24 am
by trazzdk
Hi box293.
Thanks for your fast reply.
I have now spend some hours trying with the script "plugin_command_wrapper" found on nagios_exchange.
It feels like I am onto something, but is missing the last bit.
I can get the plugin to return the traceroute and PING CRITICAL
/usr/local/nagios/libexec/plugin_command_wrapper.sh -c "/usr/bin/traceroute 8.8.8.9 -m8" -p "/usr/local/nagios/libexec/check_ping -H 8.8.8.9 -w 3000.0,80% -c 5000.0,100% -p 5" -s"c"
traceroute to 8.8.8.9 (8.8.8.9), 8 hops max, 60 byte packets
1 172.16.16.1 (172.16.16.1) 0.611 ms 0.741 ms 0.883 ms
2 94.18.233.161 (94.18.233.161) 13.792 ms 13.813 ms 13.810 ms
3 ams-ix.zensystems.net (80.249.209.220) 18.716 ms 18.805 ms 18.804 ms
4 core1.ams.net.google.com (80.249.208.247) 15.929 ms 15.798 ms 16.423 ms
5 209.85.248.92 (209.85.248.92) 16.343 ms 16.797 ms 16.968 ms
6 72.14.238.69 (72.14.238.69) 19.410 ms 209.85.253.247 (209.85.253.247) 14.583 ms 209.85.253.249 (209.85.253.249) 14.711 ms
7 209.85.254.213 (209.85.254.213) 18.183 ms 216.239.48.143 (216.239.48.143) 17.935 ms 216.239.46.167 (216.239.46.167) 18.143 ms
8 * * *
PING CRITICAL - Packet loss = 100%|rta=5000.000000ms;3000.000000;5000.000000;0.000000 pl=100%;80;100;0
However I can't get it to notify the above output in an email.
Maybe I have messed too much around in the configuration files..
Re: Mail notification (Ping service) attach traceroute
Posted: Tue Dec 09, 2014 11:47 am
by emislivec
trazzdk wrote:However I can't get it to notify the above output in an email.
Are you able to get any of that output in a notification? You should see the first line with the default notification commands in Core.
To get the rest, you will need to use the LONGHOSTOUTPUT and LONGSERVICEOUTPUT macros in your notification command. HOSTOUTPUT/SERVICEOUTPUT will only contain the first line.
If you want, you can move the check_ping output to the first line before the traceroute like so:
Code: Select all
PING CRITICAL - Packet loss = 100%|rta=5000.000000ms;3000.000000;5000.000000;0.000000 pl=100%;80;100;0
traceroute to 8.8.8.9 (8.8.8.9), 8 hops max, 60 byte packets
1 172.16.16.1 (172.16.16.1) 0.611 ms 0.741 ms 0.883 ms
2 94.18.233.161 (94.18.233.161) 13.792 ms 13.813 ms 13.810 ms
...
This will make it the output look more like check_ping, and you have the check_ping output in the short output, and the traceroute as the long output.
Re: Mail notification (Ping service) attach traceroute
Posted: Tue Dec 09, 2014 1:58 pm
by eloyd
Box was on track with the event handler. SO MUCH EASIER this way. You could even do it with a special contact who's notification method is a script that does the traceroute and emails it.
Re: Mail notification (Ping service) attach traceroute
Posted: Tue Dec 09, 2014 5:15 pm
by Box293
I was looking at something similar just yesterday in relation to traceroute.
I believe you need two entries in sudoers:
NAGIOSXI ALL = NOPASSWD:/bin/traceroute
NAGIOSXI ALL = NOPASSWD:/bin/su
I found that when testing the traceroute command as the nagios user, -I gave this message:
su nagios
/bin/traceroute -I some.domain.com
The specified type of tracerouting is allowed for superuser only
Try running the wrapper script as the user nagios as this is the account core is running as.
su nagios
Re: Mail notification (Ping service) attach traceroute
Posted: Wed Dec 10, 2014 9:47 am
by trazzdk
Thank you all for your replies.
I think, when I have time to look into this again, I will try to go for the event handler solution

Re: Mail notification (Ping service) attach traceroute
Posted: Wed Dec 10, 2014 10:11 am
by eloyd
Edit: My spacing is off on my example script, but you get the idea.
Okay, so here's what I would do.
Event handlers are great, but you have to take into account which execution pass, whether it's hard or soft, and the state to take action. That's fine, but you can also do it with a custom notification command.
Make a new command called "notify-pinger" that executes a script called notify-pinger.sh (use your own stuff here, this is just an example). Create a new contact called "pinger" that has the "service_notification_commands" set to the "notify-pinger" command you just created and set the email address to the person/people/mailing list that you want to notify (probably the same as whatever you're notifying now). Change the service to also notify pinger when it goes critical (leave all your other notifcations the same). This sets the stage for the magic below:
Code: Select all
define command{
command_name notify-pinger
command_line $USER1$/notify-pinger.sh "$NOTIFICATIONTYPE$" "$SERVICEDESC$" "$HOSTNAME$" "$SERVICESTATE$" "$SERVICEOUTPUT$" "$CONTACTEMAIL$"
}
define contact{
contact_name pinger
service_notification_period 24x7
service_notification_options w,c,u,r
service_notification_commands notify-pinger
email EMAIL@HOSTNAME
}
# Make sure you add the following contacts line to whatever service you're doing this for:
define service{
...
contacts pinger
...
}
This gives you a new contact that gets notified when all your other contact(s) get notified and sets the script used to notify them to something special. Let's go create that something special:
Code: Select all
vi /usr/local/nagios/libexec/notify-pinger.sh
#!/bin/sh
# Our parameters are "$NOTIFICATIONTYPE$" "$SERVICEDESC$" "$HOSTNAME$" "$SERVICESTATE$" "$SERVICEOUTPUT$" "$CONTACTEMAIL$"
# ^---$1 ^---$2 ^---$3 ^---$4 ^---$5 ^---$6
# Only wait 2 seconds and only do 1 query per hop (change these if you want):
output=`traceroute $3 -w 2 -q 1`
# The following is stolen from the Nagios core service notification command, altered to use our parameters as passed in from Nagios:
/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: ${1}\n\nService: ${2}\nHost: ${3}\nState: ${4}\n\nTraceroute:\n\${output}\n" | /bin/mail -s "** ${1} Service Alert: ${3}/${2} is ${4} **" ${6{
Bam. Instant traceroute as part of the ping notification, but done as a second notification using a custom script. I may have made a typo in there somewhere because I did it all off the cuff, but you get the idea.
Re: Mail notification (Ping service) attach traceroute
Posted: Wed Dec 10, 2014 6:00 pm
by tmcdonald
Eric, I knew there was a reason we kept you around
@trazzdk, let us know how that works out for you.
Re: Mail notification (Ping service) attach traceroute
Posted: Wed Dec 17, 2014 4:58 am
by trazzdk
Hi guys.
Once again thanks for your valuable input.
I ended up doing it globally simple by changing notify-host-by-email:
in /usr/local/nagios/etc/objects/commands.cfg
Code: Select all
# The New notify-host-by-email - To add Traceroute.
define command{
command_name notify-host-by-email
command_line $USER1$/notify-pinger.sh "$NOTIFICATIONTYPE$" "$LONGDATETIME$" "$HOSTNAME$" "$CONTACTEMAIL$" "$HOSTADDRESS$" "$HOSTALIAS$" "$HOSTSTATE$" "$HOSTOUTPUT$"
}
in /usr/local/nagios/libexec/notify-pinger.sh
Code: Select all
#!/bin/sh
# Parameters are "$NOTIFICATIONTYPE$" "$LONGDATETIME$" "$HOSTNAME$" "$CONTACTEMAIL$" "$HOSTADDRESS$" "$HOSTALIAS$" "$HOSTSTATE$" "$HOSTOUTPUT$"
# ^---$1 ^---$2 ^---$3 ^---$4 ^---$5 ^---$6 ^---$7 ^---$8
TR=`mtr --report $8`
/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: ${1}\n\nHost: ${3}\nAddress: ${5}\nState: ${7}\n\nDate/Time:${2}\n\nAdditional Info:\n${8}\n\nTraceroute:\n$TR" | /usr/bin/sendemail -f [email protected] -s 172.16.16.5:25 -u ** Host Alert: ${3} is ${7}** -t ${4}