Check_NRPE returns version, but cannot remote execute

[RyanW]

I have seen similar questions asked, but I have not seen one which fits this use case specifically.

Basically executing the base check-nrpe to the remote host will return the remote nrpe version:
[root@nagios agile-u2]# /usr/lib/nagios/plugins/check_nrpe -H agile-db-u2
NRPE v2.15

When you add a command to the end of the check command it returns the following:
[root@nagios agile-u2]# /usr/lib/nagios/plugins/check_nrpe -H agile-db-u2 -c check_root
NRPE: Unable to read output

The remote command defined in /etc/nagios/nrpe.cfg:
command[check_root]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% /dev/mapper/vg_agiledbd1-lv_root

Disk path confirmed:
/dev/mapper/vg_agiledbd1-lv_root

Plugin path confirmed:
[root@agile-db-u2 plugins]# pwd
/usr/lib/nagios/plugins
[root@agile-db-u2 plugins]# ls
check_disk
[root@agile-db-u2 plugins]#

Executing the check_disk command locally works as well:
[root@agile-db-u2 plugins]# /usr/lib/nagios/plugins/check_disk -v -v
DISK OK - free space: / 0 MB (0% inode=97%); /dev/shm 7974 MB (99% inode=99%); /boot 333 MB (73% inode=99%); /u02 23333 MB (24% inode=99%); /agileNAS 138148 MB (38% inode=99%);| /=65256MB;;;0;68165 /dev/shm=0MB;;;0;7975 /boot=117MB;;;0;476 /u02=72208MB;;;0;100661 /agileNAS=220251MB;;;0;358400
[root@agile-db-u2 plugins]#

I have used this same nrpe.cfg file on a couple of other servers and it has worked without issue. This seems to be currently effecting 4 out of 12 in the deployment. All of the systems are running identical version of SSL, NRPE, OS..etc.

Has anyone else experienced this issue?

[hsmith]

How did you install NRPE? Is this running as a standalone daemon, or using xinetd?

[RyanW]

I installed it from the EPEL repos, it is currently running under init.d.
**For startup reasons it's running under init.d, but to actually start the command for testing and such, I'm using "service nrpe restart/start/stop"

I have tried running it standalone: nrpe -n -c /etc/nagios/nrpe.cfg -d and received a socket timeout, which I believe was due to the -n flag.

Actually, I just ran standalone without the -n flag:
[root@agile-db-u2 init.d]# nrpe -c /etc/nagios/nrpe.cfg -d

Tested from the server and received:
[root@nagios agile-u2]# /usr/lib/nagios/plugins/check_nrpe -H agile-db-u2 -c check_root
DISK CRITICAL - free space: / 0 MB (0% inode=97%);| /=65256MB;54532;61348;0;68165

What would be the difference between running under init.d and standalone?

-R

[jdalrymple]

[root@agile-db-u2 init.d]# nrpe -c /etc/nagios/nrpe.cfg -d

There is your difference most likely. Try launching as user nagios and see if you get the same results.

[RyanW]

jdalrymple you are correct:

[root@agile-db-u2 xinetd.d]# ps -ef | grep -i nrpe
nrpe 19770 1 0 13:14 ? 00:00:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

[root@nagios agile-u2]# /usr/lib/nagios/plugins/check_nrpe -H agile-db-u2 -c check_root
NRPE: Unable to read output

When it is run as root I do receive output, as the user specified in the nrpe.cfg (nrpe) I do NOT receive output.

[lmiltchev]

Does it help if you put the following line in sudoers on the remote box?

Code: Select all

nagios   ALL = NOPASSWD:/usr/lib/nagios/plugins/check_disk *

[RyanW]

Does it help if you put the following line in sudoers on the remote box?

Code: Select all

nagios   ALL = NOPASSWD:/usr/lib/nagios/plugins/check_disk *

Just gave it a try and restarted services, but it was unsuccessful.
I'm taking a look at the other systems now to look for discrepancies in users/groups, but so far, the process gets spawned as the nrpe user on other systems and it is working!

Working client:
[root@agile-proxy-u2 ~]# ps -ef | grep -i nrpe
nrpe 14943 1 0 Nov04 ? 00:00:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

Broken client:
[root@agile-db-u2 nagios]# ps -ef | grep -i nrpe
nrpe 20530 1 0 14:27 ? 00:00:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

These are using the same nrpe.cfg file with tweaks to the custom check dev path.

-R

[Box293]

jdalrymple you are correct:

[root@agile-db-u2 xinetd.d]# ps -ef | grep -i nrpe
nrpe 19770 1 0 13:14 ? 00:00:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

[root@nagios agile-u2]# /usr/lib/nagios/plugins/check_nrpe -H agile-db-u2 -c check_root
NRPE: Unable to read output

When it is run as root I do receive output, as the user specified in the nrpe.cfg (nrpe) I do NOT receive output.

IF your user is nrpe then the sudoers line needs to start with nrpe (I'm pretty sure):

Code: Select all

nrpe   ALL = NOPASSWD:/usr/lib/nagios/plugins/check_disk *

[RyanW]

IF your user is nrpe then the sudoers line needs to start with nrpe (I'm pretty sure):

I noticed the user is NRPE and added that line to the sudoers file as well, still returned the same results.

[RyanW]

Thanks for all of the input, from everything I've gathered it's a user/group issue.
As to why it is working on the other machines, I have not been able to figure that one out yet.

If I get a definitive solution I will report back.

Thanks guys!