Mail notification (Ping service) attach traceroute

[trazzdk]

Hi guys,

We would like to have a traceroute attached to our Service: PING notifications.

So if we get a Ping warning / critical e-mail notification, a traceroute is attached as well.

Any ideas how to do this ?

/Trazz

[Box293]

There are two ways you could go about this.

1) Create an Event Handler that will do the traceroute when the ping check fails ... I've not done this before so it could get complicated.

2) Create a wrapper script that executes the ping check. Inside the wrapper script, if the ping check failed then it could perform the traceroute and append this information to the status information of the ping check. This information would then appear as part of the service detail AND would get sent with the notification.

[trazzdk]

Hi box293.


Thanks for your fast reply.

I have now spend some hours trying with the script "plugin_command_wrapper" found on nagios_exchange.

It feels like I am onto something, but is missing the last bit.

I can get the plugin to return the traceroute and PING CRITICAL

/usr/local/nagios/libexec/plugin_command_wrapper.sh -c "/usr/bin/traceroute 8.8.8.9 -m8" -p "/usr/local/nagios/libexec/check_ping -H 8.8.8.9 -w 3000.0,80% -c 5000.0,100% -p 5" -s"c"
traceroute to 8.8.8.9 (8.8.8.9), 8 hops max, 60 byte packets
1 172.16.16.1 (172.16.16.1) 0.611 ms 0.741 ms 0.883 ms
2 94.18.233.161 (94.18.233.161) 13.792 ms 13.813 ms 13.810 ms
3 ams-ix.zensystems.net (80.249.209.220) 18.716 ms 18.805 ms 18.804 ms
4 core1.ams.net.google.com (80.249.208.247) 15.929 ms 15.798 ms 16.423 ms
5 209.85.248.92 (209.85.248.92) 16.343 ms 16.797 ms 16.968 ms
6 72.14.238.69 (72.14.238.69) 19.410 ms 209.85.253.247 (209.85.253.247) 14.583 ms 209.85.253.249 (209.85.253.249) 14.711 ms
7 209.85.254.213 (209.85.254.213) 18.183 ms 216.239.48.143 (216.239.48.143) 17.935 ms 216.239.46.167 (216.239.46.167) 18.143 ms
8 * * *
PING CRITICAL - Packet loss = 100%|rta=5000.000000ms;3000.000000;5000.000000;0.000000 pl=100%;80;100;0


However I can't get it to notify the above output in an email.

Maybe I have messed too much around in the configuration files..

[emislivec]

However I can't get it to notify the above output in an email.

Are you able to get any of that output in a notification? You should see the first line with the default notification commands in Core.

To get the rest, you will need to use the LONGHOSTOUTPUT and LONGSERVICEOUTPUT macros in your notification command. HOSTOUTPUT/SERVICEOUTPUT will only contain the first line.

If you want, you can move the check_ping output to the first line before the traceroute like so:

Code: Select all

PING CRITICAL - Packet loss = 100%|rta=5000.000000ms;3000.000000;5000.000000;0.000000 pl=100%;80;100;0
traceroute to 8.8.8.9 (8.8.8.9), 8 hops max, 60 byte packets
1 172.16.16.1 (172.16.16.1) 0.611 ms 0.741 ms 0.883 ms
2 94.18.233.161 (94.18.233.161) 13.792 ms 13.813 ms 13.810 ms
...

This will make it the output look more like check_ping, and you have the check_ping output in the short output, and the traceroute as the long output.

[eloyd]

Box was on track with the event handler. SO MUCH EASIER this way. You could even do it with a special contact who's notification method is a script that does the traceroute and emails it.

[Box293]

I was looking at something similar just yesterday in relation to traceroute.

I believe you need two entries in sudoers:

NAGIOSXI ALL = NOPASSWD:/bin/traceroute
NAGIOSXI ALL = NOPASSWD:/bin/su

I found that when testing the traceroute command as the nagios user, -I gave this message:

su nagios
/bin/traceroute -I some.domain.com

The specified type of tracerouting is allowed for superuser only


Try running the wrapper script as the user nagios as this is the account core is running as.
su nagios

[trazzdk]

Thank you all for your replies.

I think, when I have time to look into this again, I will try to go for the event handler solution

[eloyd]

Edit: My spacing is off on my example script, but you get the idea.

Okay, so here's what I would do.

Event handlers are great, but you have to take into account which execution pass, whether it's hard or soft, and the state to take action. That's fine, but you can also do it with a custom notification command.

Make a new command called "notify-pinger" that executes a script called notify-pinger.sh (use your own stuff here, this is just an example). Create a new contact called "pinger" that has the "service_notification_commands" set to the "notify-pinger" command you just created and set the email address to the person/people/mailing list that you want to notify (probably the same as whatever you're notifying now). Change the service to also notify pinger when it goes critical (leave all your other notifcations the same). This sets the stage for the magic below:

Code: Select all

define command{
  command_name  notify-pinger
  command_line  $USER1$/notify-pinger.sh "$NOTIFICATIONTYPE$" "$SERVICEDESC$" "$HOSTNAME$" "$SERVICESTATE$" "$SERVICEOUTPUT$" "$CONTACTEMAIL$"
}

define contact{
  contact_name  pinger
  service_notification_period  24x7
  service_notification_options  w,c,u,r
  service_notification_commands  notify-pinger
  email  EMAIL@HOSTNAME
}

# Make sure you add the following contacts line to whatever service you're doing this for:
define service{
  ...
  contacts  pinger
  ...
}

This gives you a new contact that gets notified when all your other contact(s) get notified and sets the script used to notify them to something special. Let's go create that something special:

Code: Select all

vi /usr/local/nagios/libexec/notify-pinger.sh

#!/bin/sh
# Our parameters are "$NOTIFICATIONTYPE$" "$SERVICEDESC$" "$HOSTNAME$" "$SERVICESTATE$" "$SERVICEOUTPUT$" "$CONTACTEMAIL$"
#                                 ^---$1                           ^---$2                  ^---$3              ^---$4                    ^---$5                      ^---$6
# Only wait 2 seconds and only do 1 query per hop (change these if you want):
output=`traceroute $3 -w 2 -q 1`
# The following is stolen from the Nagios core service notification command, altered to use our parameters as passed in from Nagios:
/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: ${1}\n\nService: ${2}\nHost: ${3}\nState: ${4}\n\nTraceroute:\n\${output}\n" | /bin/mail -s "** ${1} Service Alert: ${3}/${2} is ${4} **" ${6{

Bam. Instant traceroute as part of the ping notification, but done as a second notification using a custom script. I may have made a typo in there somewhere because I did it all off the cuff, but you get the idea.

[tmcdonald]

Eric, I knew there was a reason we kept you around

@trazzdk, let us know how that works out for you.

[trazzdk]

Hi guys.

Once again thanks for your valuable input.

I ended up doing it globally simple by changing notify-host-by-email:

in /usr/local/nagios/etc/objects/commands.cfg

Code: Select all

# The New notify-host-by-email - To add Traceroute.

define command{
  command_name  notify-host-by-email
  command_line  $USER1$/notify-pinger.sh "$NOTIFICATIONTYPE$" "$LONGDATETIME$" "$HOSTNAME$"  "$CONTACTEMAIL$" "$HOSTADDRESS$" "$HOSTALIAS$" "$HOSTSTATE$" "$HOSTOUTPUT$"
}

in /usr/local/nagios/libexec/notify-pinger.sh

Code: Select all

#!/bin/sh
# Parameters are "$NOTIFICATIONTYPE$" "$LONGDATETIME$" "$HOSTNAME$" "$CONTACTEMAIL$" "$HOSTADDRESS$" "$HOSTALIAS$" "$HOSTSTATE$" "$HOSTOUTPUT$"
#                 ^---$1               ^---$2            ^---$3       ^---$4          ^---$5          ^---$6        ^---$7         ^---$8

TR=`mtr --report $8`

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: ${1}\n\nHost: ${3}\nAddress: ${5}\nState: ${7}\n\nDate/Time:${2}\n\nAdditional Info:\n${8}\n\nTraceroute:\n$TR" | /usr/bin/sendemail -f nagios@secomea.com -s 172.16.16.5:25 -u ** Host Alert: ${3} is ${7}** -t ${4}