New Notification Troubleshooting

[skynardo]

Hello all, I am a newb to Nagios. I have installed Core 4.0.0-beta1 on RHEL6 and have been kicking the tires for about a week. I used the quickstart install and am monitoring my local Nagios host and one remote linux box via nrpe. Email notifications seem to be working but I was wanting to test defining a new notification that would call a script on my Nagios machine, pass some command line arguments related to the event which would forward the alert to our 3rd party event console. I have been playing around with the command_line but it seems it does not attempt to fire my script as I have it defined. If I change the command line to do a /bin/echo "command_line as shown below"> local_file.out, the output in the local file contains what I expected an in fact when I chmod+x the local file and execute it as user nagios, it runs without error and sends the event to the Event console. Below is an example of the command_line I am trying to define in my commands.cfg file. Does anyone see anything obviously wrong here?

command_line /bin/forward_event -a \"$SERVICEDESC$\" -m \"$NOTIFICATIONTYPE$ Service: $SERVICEDESC$ on Host: $HOSTALIAS$ State: $SERVICESTATE$\" -i \"$SERVICEOUTPUT$\" -r $SERVICESTATE$ -p send-service-to-console -P \"$SERVICEDESC$\" -h $HOSTALIAS$ -e GROUP_TEST

[jsmurphy]

I don't think you need to escape the quotes... because I think you literally want to quote the text correct? not parse the quote as text to the script... because what you are doing now will create an awful lot of arguments.

Assuming you can run the above command definition from the command line just fine, my second wild stab in the dark would be double check the permissions. I know NRPE tries to protect you from yourself by preventing you from executing in /bin but I have no idea if Nagios tries to protect you in the same manner. It might be worthwhile moving the executable to a different directory unless one of the Nagios guys can confirm this isn't the case.

[abrist]

I would try it from /usr/bin as that is where the default core notifications run printf from. Is forward_event a script? if so, adding some echo to file commands to it may help the troubleshooting.

[skynardo]

Thanks for your responses.
I need to escape the quotes around "macros" that contain spaces so that the quotes are passed to my external script to contain those strings as a single argument. When I change the command to "

command_line /bin/echo "/mydir/bin/send_event -a \"$SERVICEDESC$\" -m \"$NOTIFICATIONTYPE$ Service: $SERVICEDESC$ on Host: $HOSTALIAS$ State: $SERVICESTATE$\" -i \"$SERVICEOUTPUT$\" -r $SERVICESTATE$ -p send-service-event -P \"$SERVICEDESC$\" -h $HOSTALIAS$ -e GROUP_TEST">/logs/send-service-event.out

The notification triggers and produces a send-service-event.out file as shown below:
/mydir/bin/send_event -a "Postgres_Size" -m "PROBLEM Service: Postgres_Size on Host: localhost State: CRITICAL" -i "POSTGRES_DATABASE_SIZE CRITICAL: DB postgres (host:127.0.0.1) postgres:6632568 (6477 kB) template1: 6632568 (6477 kB) template0: 6513156 (6361 kB)" -r CRITICAL -p send-service-event -P "Postgres_Size" -h localhost -e GROUP_TEST

I can actually make this file executeable and run it as the nagios user and it sends the event just fine. But if I change the command_line back to remove the /bin/echo and >/logs/send-service-event.out, it seems not to run at all.

Notification commands aren't governed by the nagios.cmd file for external commands are they? I assumed that was for commands being executed from the UI.

[abrist]

Can you add a line to your script that echos to a file in /tmp, so that you can verify if it is being run at all? If it is not running, check permissions by getting a listing of its directory and posting the output here.

[sreinhardt]

That makes more sense as to why you need the quotes, especially if you are echoing. Have you tried using echo to pipe your arguments to the script in question? Something like:

Code: Select all

command_line /bin/echo "-a \"$SERVICEDESC$\" -m \"$NOTIFICATIONTYPE$ Service: $SERVICEDESC$ on Host: $HOSTALIAS$ State: $SERVICESTATE$\" -i \"$SERVICEOUTPUT$\" -r $SERVICESTATE$ -p send-service-event -P \"$SERVICEDESC$\" -h $HOSTALIAS$ -e GROUP_TEST" | /mydir/bin/send_event

It certainly seems like your current echo command is sending it out correctly, it just doesn't want to send it from the directory you are using. As mentioned previously, this certainly could be a restriction by nagios core, in addition to /usr/bin, you might try the location of your other plugins as a location for this file. I would also agree with abrist, if you can modify the send_event code to echo or touch a file in /tmp and be sure it is actually being run.

[skynardo]

It appears the send_event code is not being run, regardless of where I put it. I have tried /bin, /usr/bin as well as my nagios libexec directory. I tried testing by calling /send_event -V >/tmp/test_send_event.out (this will generate a useage statement) but it does not produce the file when run from Nagios as a notification. If I run the same command as an "at" job from the nagios user, it creates the expected file without an issue. I should mention that send_event is a perl script but was compiled using perl2exe for portability.

file nagios/libexec/send_event
nagios/libexec/send_event: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.0.0, stripped

Below is dir and file listing of current testing location which is my plugins directory

[root@devut512 techserv]# ls -ld nagios
drwxr-xr-x 4 nagios nagios 4096 Aug 9 08:21 nagios
[root@devut512 techserv]# ls -lrt nagios/libexec/send_event
-rwxr-xr-x 1 nagios nagios 1141239 Aug 19 11:43 nagios/libexec/send_event

I also tried to echo the the arguments and pipe them to send_event with same results, it appears not to run or attempt to run the script at all.

[skynardo]

I have not been able to resolve this issue, and have no more time to work on it for now. Since my troubleshooting step of echoing the cmnd_line with instantiated variables to a local file works just fine, I wrote a small shell script that runs from cron every minute to check for such files, execute them then remove them and this seems to serve the desired purpose for now.

If I find a better resolution down the road I will update this post.

Thanks

[abrist]

Fair enough, we will leave the thread open.