NRPE Not Returning Value

[danielgaraventa_ats]

Hello,

We have a custom check that we have written for our Oracle database servers to check the number of days left until passwords expire via an SQL command. We are currently testing on moving our databases to RHEL 7 from SUSE 11. Running on SUSE 11 the check works as expected when we run the script through NRPE. On RHEL 7 the output from the SQL command does not seem to be being passed to NRPE It does work fine however when I run the check locally on the RHEL 7 server.

Code: Select all

#!/bin/bash

# Nagios return codes
UNKNOWN=3
CRIT=2
WARN=1
OK=0

OUT=$(/opt/oracle/product/11.2.0.4/bin/sqlplus -S redacted/redacted@svr-rhel7/SID << EOF
set head off;
select nvl( min(trunc(expiry_date) - trunc(sysdate)),90) days_remaining from dba_users u where expiry_date is not null and account_status = 'OPEN';
exit
EOF
)

export OUT

if test "$OUT" -gt 30; then
  echo "OK - Passwords are not set to expire for another "$OUT" days"
  exit $OK
elif test "$OUT" -gt 14; then
  echo "WARN - Passwords are scheduled to expire in "$OUT" days"
  exit $WARN
else
  echo "CRIT - Passwords are scheduled to expire in "$OUT" days"
  exit $CRIT
fi

When I run the script locally on the RHEL 7 box I get the following output.

Code: Select all

[root@svr-rhel7 plugins]# runuser -u nrpe /usr/lib64/nagios/plugins/test2
OK - Passwords are not set to expire for another  90 days

When I run the check_nrpe on our Nagios XI server I see the following output.

Code: Select all

[root@monitor libexec]# ./check_nrpe -H svr-rhel7 -t 60 -p 5666 -4 -c test2
CRIT - Passwords are scheduled to expire in  days

We are currently running Nagios XI 5.4.11 and I have also disabled SELinux on the RHEL 7 server.

Thank you for any advice,
Daniel

[npolovenko]

Hello, @danielgaraventa_ats.
What're the permissions on this folder?

Code: Select all

/opt/oracle/product/11.2.0.4/bin/sqlplus

When you run it locally it runs as a sudo user but if you run it with Nagios it runs as a Nagios user.
I'd probably do chmod +x on that folder to see if that resolves the problem.

[danielgaraventa_ats]

Hello npolovenko,

sqlplus is a file in /opt/oracle/product/11.2.0.4/bin which has the following permissions.

Code: Select all

[root@ats-db-04 ~]# ls -la /opt/oracle/product/11.2.0.4/bin/sqlplus 
-rwxr-x--x. 1 oracle oinstall 13575 Jun 24  2014 /opt/oracle/product/11.2.0.4/bin/sqlplus

/opt/oracle/product/11.2.0.4/bin has the following permissions set.

Code: Select all

[root@ats-db-04 ~]# ls -lad /opt/oracle/product/11.2.0.4/bin/ 
drwxr-xr-x. 2 oracle oinstall 12288 Nov 30 08:56 /opt/oracle/product/11.2.0.4/bin/

I have added the nagios and the nrpe user to the oinstall group.

Code: Select all

[root@ats-db-04 ~]# groups nrpe
nrpe : nrpe oinstall nagios

Code: Select all

[root@ats-db-04 ~]# groups nagios
nagios : nagios oinstall

I still seem to get the same results sadly.

Thanks,
Daniel

[npolovenko]

@danielgaraventa_ats, On the remote server, can you change the username to Nagios:

Code: Select all

su - nagios

And then attempt to run the script locally. Does it work?
Are you running nrpe as a standalone service or under xinetd daemon?

[danielgaraventa_ats]

If I su to the nagios user locally the script runs normally.

Code: Select all

[nagios@ats-db-04 plugins]$ ./test2
OK - Passwords are not set to expire for another  90 days

As for the nrpe version on the rhel box I am using the standalone version I believe. It is the one that you can install from EPEL.

Code: Select all

yum list installed nrpe
nrpe.x86_64     3.2.0-6.el7         @epel

[npolovenko]

@danielgaraventa_ats, Actually, I think your version of nrpe runs checks under nrpe user. You can make sure by opening /usr/local/nagios/nrpe.cfg file, and looking for the lines that say nrpe_user= and nrpe_group=
The first thing you could try is to create a script on the remote server that will call the test2 script and put its output to a local text file ./test2 >1.txt. Then you'd run the parent script with nrpe. That way you will get to know whether results get lost during NRPE transaction or maybe nrpe user still has permission problems. Or I guess you could just save $OUT to a local text file. But I hope you get my point?
Also, you may give this a shot:

Code: Select all

  su - nrpe

Code: Select all

./test2

Keep us updated.

[danielgaraventa_ats]

I stripped down the script to be the following. Taking the SQL command out just for testing purposes.

Code: Select all

#!/bin/bash

# Nagios return codes
UNKNOWN=3
CRIT=2
WARN=1
OK=0


OUTTEST=`cat test.txt`

export OUTTEST

if test "$OUTTEST" -gt 30; then
  echo "OK - Passwords are not set to expire for another "$OUTTEST" days"
  exit $OK
elif test "$OUTTEST" -gt 14; then
  echo "WARN - Passwords are scheduled to expire in "$OUTTEST" days"
  exit $WARN
else
  echo "CRIT - Passwords are scheduled to expire in "$OUTTEST" days"
  exit $CRIT
fi

test.txt contains the number 90. Running the script locally I see this.

Code: Select all

[nrpe@svr-rhel7 plugins]$ ./test2
OK - Passwords are not set to expire for another 90 days

Running the test from our Nagios server via nrpe the 90 is not populated.

Code: Select all

[root@monitor libexec]# ./check_nrpe -H svr-rhel7 -t 60 -p 5666 -4 -c test2
CRIT - Passwords are scheduled to expire in  days

Thanks,
Daniel

[danielgaraventa_ats]

I forgot to add test.txt is owned by nrpe and I change the permissions to 777.

Thanks,
Daniel

[npolovenko]

@danielgaraventa_ats, If you change this line in the script:

Code: Select all

OUTTEST=`cat test.txt`

to

Code: Select all

OUTTEST=`cat /full/path/to/test.txt`

Then it's going to work. I think the main problem with your original script is that sqlplus runs as a sudo, or as a nagios user but it doesn't run as nrpe user. Either because the nrpe user doesn't have a shell, or because of the other permissions problem. Sometimes you also need to include environment variables in order for the script to run.

[danielgaraventa_ats]

I believe I found the issue. I added the following the the script and I am now getting results back.

Code: Select all

ORACLE_BASE=/opt/oracle
ORACLE_HOME=$ORACLE_BASE/product/11.2.0.4
export ORACLE_BASE ORACLE_HOME

Code: Select all

[root@monitor libexec]# ./check_nrpe -H svr-rhel7 -t 60 -p 5666 -4 -c test2
OK - Passwords are not set to expire for another  90 days

Thank you for all your help! I will mark this post as solved.

Daniel