Warning: Attempting to execute the command errors...

[ucemike]

Having some issues with service notifications (not host) being sent. I am running nagios v3.2.3 on RHEL 6.1.

Okay, here is the /var/log/nagios.log message:

[1321474179] SERVICE NOTIFICATION: admin-test;somehost.here.com;CPU load;WARNING;notify-service-by-email-test;WARN - 15min Load 7.99 at 1 CPUs (warning at 5.00)

[1321474179] Warning: Attempting to execute the command "/bin/echo "***** Nagios *****\n\nNotification Type: PROBLEM\n\nService: CPU load\nHost: somehost.here.com\nAddress: 1.2.3.4\nState: WARNING\n\nDate/Time: Wed Nov 16 20:09:39 GMT 2011\n\nAdditional Info:\n\nWARN - 15min Load 7.99 at 1 CPUs (warning at 5.00)\n** PROBLEM Service Alert: somehost.here.com/CPU load is WARNING ** some@email.com\n-------------------\n" >> /var/log/nagios/service-test.log" resulted in a return code of 127. Make sure the script or binary you are trying to execute actually exists...

As you can see it's just trying to echo the output and it fails. I added the admin-test contact because we're seeing this exact same error with the email notification using printf. The error however is random. It doesn't occur 100% of the time. The above is the full text of the error with the exception of host/ip being altered. This also only seems to happen on service problems, NOT hosts.

Here is the commands.cfg entry for admin-test, notify-service-by-email-test:

command_line /bin/echo "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ ** $CONTACTEMAIL$\n------------\n" >> /var/log/nagios/host-test.log

Here is the nagiosadmin/notify-service-by-email commands.cfg entry:

command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$

Here is the same thing only with the printf email notification:

[1321474179] SERVICE NOTIFICATION: nagiosadmin;somehost.here.com;CPU load;WARNING;notify-service-by-email;WARN - 15min Load 7.99 at 1 CPUs (warning at 5.00)

[1321474179] Warning: Attempting to execute the command "/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: PROBLEM\n\nService: CPU load\nHost: somehost.here.com\nAddress: 1.2.3.4\nState: WARNING\n\nDate/Time: Wed Nov 16 20:09:39 GMT 2011\n\nAdditional Info:\n\nWARN - 15min Load 7.99 at 1 CPUs (warning at 5.00)\n" | /bin/mail -s "** PROBLEM Service Alert: somehost.here.com/CPU load is WARNING **" some@email.com" resulted in a return code of 127. Make sure the script or binary you are trying to execute actually exists...

As with the other error all I've done is masked the host/ip.

I initially thought some character was causing the problem but as far as I can tell the above service alert has worked on other hosts with similar names/characters. The fact it fails on a /bin/echo is stumping me completely. If anyone has suggestions or questions regarding the setup please let me know.

[ucemike]

One thing I neglected to mention, copy/pasting the command line it says failed works fine.

[jsmurphy]

Are you able to post your host notification command definition as well? At first I thought it looked like a permissions issue... but for it to be affecting one and not the other is kind of strange. The host notification definition might help us work out what we're missing!

[ucemike]

Sure thing:

command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$

I am actually leaning back towards some character in the service text info. Although, as I said, if I copy/paste the command in the log it said failed and run it as user "nagios" it works fine.

[jsmurphy]

Hmmm, I can't see anything obviously wrong and I think this may have me stumped as well especially considering it works sometimes but not others... if you believe it's due to special characters, which characters? Do you have allow nasty meta characters enabled in the nagios config?

[ucemike]

Hmmm, I can't see anything obviously wrong and I think this may have me stumped as well especially considering it works sometimes but not others... if you believe it's due to special characters, which characters? Do you have allow nasty meta characters enabled in the nagios config?

Well, I'm just guessing it's characters. We have no special ones as far as I know. I did a once over on the alerts and removed the ()'s and []'s just to see if somehow that was causing something but didn't resolve anything.

I've been eyeballing the logs but tomorrow im going to do some grep/parsing to get better specifics on this. Right now it's looking like specific services are not getting notices but others are. It's not specific to a host (filesystem alerts for the host will go out but cpu alerts will not).

As far as nasty meta characters enabled in nagios config...nope.

[jsmurphy]

Very strange, very strange... if you find something I'd be interested to know what caused it.

[ucemike]

Holidays slowed me down on this but...

Still trying to nail this down. I even wrote my own perl script to handle this. The perl script doesn't use mailx (uses a smtp mod) and so far it seems to be reacting the same (only the nagios logs don't report an error, it just doesn't go out).

More work tomorrow to try and debug this down to the cause.

[ucemike]

Okay, so here is what I did to get around the problem. Created a perl script to connect to an outbound smtp server and deliver the message (using Net::SMTP). I still am unable to figure out why postfix (postfix-2.6.6-2.1.el6_0.x86_64) is not taking for delivery. It is not a random thing as it seemed to be at first. Some hosts would send out CPU alerts but the same alert on another host would fail.

Regardless the log results in mail and nagios just do not make it clear what the problem is. For now I am using the perl script to handle the deliveries but I am still going to try and figure out what is actually going on.

[jsmurphy]

I'm glad you've found a work around, may even be worth updating/re-installing postfix... at this stage anything is worth a shot I suppose.