Custom Nagios check works from command line, but not Nagios

[steved]

Hello all,

I've created a basic script that grabs our AWS reserved instance dates and does a simple calculation to determine if the oldest instance will expire in the next 5 days.

I can run the script from a shell without issue, but when Nagios runs it as a check some values are not populated:

Script:

Code: Select all

#!/bin/bash

nowdate=`date +%s`
command=`/opt/ec2-api-tools/bin/ec2-describe-reserved-instances -F state=active | awk '{print $10}' | awk -F T '{print $1}' | sed '/^$/d' | sort`
reservations=($(echo $command))
reserveddate=$(echo $reservations | xargs date +"%s" -d)
diffdate=$(($nowdate - $reserveddate))
remaininghours=$(((31536000 + $reserveddate - $nowdate) / 3600))

if [ $remaininghours -le 120 ]
   then
      echo "A reserved instance will expire in $remaininghours hours. Please renew via EC2 console."
      # debug
      echo nowdate is $nowdate
      echo reserveddate is $reserveddate
      echo diffdate is $diffdate
      echo remaininghours is $remaininghours
      # /debug
      exit 1
   else
      echo "All reservations are current."
      # debug
      echo nowdate is $nowdate
      echo reserveddate is $reserveddate
      echo diffdate is $diffdate
      echo remaininghours is $remaininghours
      # /debug
      exit 0
fi

Output in a shell:

Code: Select all

A reserved instance will expire in 37 hours. Please renew via EC2 console.
nowdate is 1381243634
reserveddate is 1349841600
diffdate is 31402034
remaininghours is 37

Output from Nagios:

Code: Select all

A reserved instance will expire in -374918 hours. Please renew via EC2 console.
nowdate is 1381243625
reserveddate is
diffdate is
remaininghours is -374918

So it looks like when I try to convert $reserveddate the variable isn't being passed to the shell properly, or maybe not passed back? I'm not entirely sure why this is happening.

Any help is appreciated.

[tmcdonald]

I think the issue might be earlier on than that. What are the permissions for the /opt/ec2-api-tools/bin/ec2-describe-reserved-instances command? If the nagios user cannot execute it, then your "command" will not run, therefor never allowing "reservations" to get data, etc etc.

[steved]

Is it valid to test by:
#su - nagios
Then run the command? If so, the nagios user can run the command without issue, and can run my shell script without issue.

[tmcdonald]

Can you show us the output of you actually su'ing to nagios, then running the script specifying the full path name? While you're at it, wanna actually show the output of "ls -l /opt/ec2-api-tools/bin/ec2-describe-reserved-instances" again as nagios? One last thing, try "touch /opt/ec2-api-tools/bin/test" for good measure.

Be sure to sanitize any potential sensitive information. Shouldn't be any in this case, but always a good thing to keep in mind.

[steved]

For sure:

Code: Select all

[root@linserver2 ~]# su - nagios
-bash-4.1$ /opt/ec2-api-tools/bin/ec2-describe-reserved-instances | head -n 1
RESERVEDINSTANCES	***************************************	us-east-1b	m1.small	Linux/UNIX	1y 	227.5	0.03	1	2011-10-06T12:33:23+0000	retired	USD	default	Medium Utilization

-bash-4.1$ /etc/nagios/scripts/ec2-reserve-warn.sh 
A reserved instance will expire in 33 hours. Please renew via EC2 console.
nowdate is 1381255882
reserveddate is 1349841600
diffdate is 31414282
remaininghours is 33

-bash-4.1$ ls -l /opt/ec2-api-tools/bin/ec2-describe-reserved-instances
-rwxr-xr-x 1 root root 728 Sep 10 14:11 /opt/ec2-api-tools/bin/ec2-describe-reserved-instances

-bash-4.1$ touch /opt/ec2-api-tools/bin/test
touch: cannot touch `/opt/ec2-api-tools/bin/test': Permission denied

Script(s) run without issue, nagios doesn't have write permission to that dir, but the x bit is set for other.

Can you show us the output of you actually su'ing to nagios, then running the script specifying the full path name? While you're at it, wanna actually show the output of "ls -l /opt/ec2-api-tools/bin/ec2-describe-reserved-instances" again as nagios? One last thing, try "touch /opt/ec2-api-tools/bin/test" for good measure.

Be sure to sanitize any potential sensitive information. Shouldn't be any in this case, but always a good thing to keep in mind.

[yancy]

steved,

I can run the script from a shell without issue, but when Nagios runs it as a check some values are not populated:

Can you be more specific about what values are and are not populating?

-Yancy

[steved]

Yep, in the OP there are two code sections showing the output.

The script is echoing the variables for debugging purposes. In the shell version all 4 variables contain values. In the Nagios web UI version there are empty variables such as $reserveddate. Because there is no value, the other calculated variables have the wrong values as well.

steved,

I can run the script from a shell without issue, but when Nagios runs it as a check some values are not populated:

Can you be more specific about what values are and are not populating?

-Yancy

[abrist]

Judging from the output on your first post, it definitely seems like the $command is not running or returning bad data, as the variables are wrong/empty from that point on.

[MichaelMoritz]

We know reserveddate is not getting set, is reservations getting set?

[sreinhardt]

I would venture to say that it is not, however you could always modify and set an echo after each line to print that particular value and be 100% certain!