What am I missing? (NRPE: Unable to read output)

[aneah]

I've done some Googling to try and figure out the problem I'm having, but I can't see to find it what is the problem. Hopefully, you folks can provide some insight.

First, an idea of the environment: 27 RHEL servers, predominantly physical but transitioning over to virtual Soon™. I have two servers running RHEL 6.4, one running RHEL 6.5, one running RHEL 6.6, and the rest are running RHEL 6.7.
The problem server is a physical server running RHEL 6.5.
I am running Nagios Core 4.1.1 and NRPE is v2.15.

Each of the servers runs the following plugins (installed via the EPEL repo): check_users, check_load, check_swap, check_disk, check_procs, check_uptime, and check_mem.

The problem is the check_mem plugin only. I am getting the "NRPE: Unable to read output" error. This is the only error on the server. I've checked permissions, I've checked the commands, and I've gotten to the point where I want to slowly bounce my head off the desk.

Here is the information I think you may need. Please, let me know if there is more that is required:

Output of running the command from the /etc/nagios/nrpe.cfg file

Code: Select all

/usr/lib64/nagios/plugins # /usr/lib64/nagios/plugins/check_mem  -f -w 20 -c 10
CRITICAL - 1.0% (36948 kB) free!|TOTAL=3882020KB;;;; USED=3845072KB;3105616;3493818;; FREE=36948KB;;;; CACHES=3520712KB;;;;

Output of running check_nrpe

Code: Select all

/usr/lib64/nagios/plugins # /usr/lib64/nagios/plugins/check_nrpe -H 127.0.0.1
NRPE v2.15

Permissions for the check_nrpe and check_mem plugins. All plugins in this directory are the same permissions and same owner.

Code: Select all

-rwxr-xr-x. 1 root root    22256 Sep 18  2015 check_nrpe
-rwxr-xr-x. 1 root root    13429 Mar 30 15:57 check_mem

/etc/sysconfig/selinux (NOTE: The Selinux file is the same on all servers.)

Code: Select all

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

/var/log/messages

Code: Select all

Apr  3 03:40:01 <server> rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1945" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Apr  3 03:40:02 <server> rhsmd: In order for Subscription Manager to provide your system with updates, your system must be registered with the Customer Portal. Please enter your Red Hat login to ensure your system is up-to-date.
Apr  3 09:19:01 <server>auditd[1904]: Audit daemon rotating log files
Apr  4 03:49:02 <server>rhsmd: In order for Subscription Manager to provide your system with updates, your system must be registered with the Customer Portal. Please enter your Red Hat login to ensure your system is up-to-date.
Apr  4 08:05:27 <server>nrpe[45446]: Error: Could not complete SSL handshake. 5

[jpeery]

Here's what I'd do,
From the monitoring server, do a nrpe check, but also pass it the command (in this case the memory check). If it works, that tells you the monitoring server can connect to the remote server using nrpe, and it can run the command/check you specify. If that works, I'd go to the remote server, verify the user nrpe is running as, given it's 'nagios' I'd run the check as the nagios user to check perms. For example, su - nagios -c "/usr/local/nagios/libexec/check_mem" My bet is, since those files are owned by root:root, if you're running as user nagios it won't work.
HTH

[lmiltchev]

Can you show us the actual command definition on the cliect (remote box), and the check that you run on the nagios server from the command line, along with the output of it?

Example:

Client:

Code: Select all

command[check_mem]=/usr/lib64/nagios/plugins/check_mem  -f -w 20 -c 10

Nagios:

Code: Select all

./check_nrpe -H <client ip> -c check_mem

[aneah]

Here's what I'd do,
From the monitoring server, do a nrpe check, but also pass it the command (in this case the memory check). If it works, that tells you the monitoring server can connect to the remote server using nrpe, and it can run the command/check you specify. If that works, I'd go to the remote server, verify the user nrpe is running as, given it's 'nagios' I'd run the check as the nagios user to check perms. For example, su - nagios -c "/usr/local/nagios/libexec/check_mem" My bet is, since those files are owned by root:root, if you're running as user nagios it won't work.
HTH

Ok, NRPE output from Nagios application server:

Code: Select all

/usr/lib64/nagios/plugins/check_nrpe -H <servername>
NRPE v2.15

NRPE output with check_mem call

Code: Select all

/usr/lib64/nagios/plugins/check_nrpe -H <servername> -c check_mem
NRPE: Unable to read output

I thought about the running as root issue, but if that were the case, wouldn't all the other servers have the same issue? Ownership/permissions are the same on all of them.

[aneah]

Can you show us the actual command definition on the cliect (remote box), and the check that you run on the nagios server from the command line, along with the output of it?

Example:

Client:

Code: Select all

command[check_mem]=/usr/lib64/nagios/plugins/check_mem  -f -w 20 -c 10

Nagios:

Code: Select all

./check_nrpe -H <client ip> -c check_mem

Client:
From /etc/nagios/nrpe.cfg:

Code: Select all

command[check_users]=/usr/lib64/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib64/nagios/plugins/check_load -w 3.6,2.8,2 -c 4,3.2,2.4
command[check_swap]=/usr/lib64/nagios/plugins/check_swap -w 20% -c 10%
command[check_disk]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p / -u GB
command[check_procs]=/usr/lib64/nagios/plugins/check_procs -w 20% -c 10% --m CPU
command[check_uptime]=/usr/lib64/nagios/plugins/check_uptime
command[check_mem]=/usr/lib64/nagios/plugins/check_mem  -f -w 20 -c 10

Actual output of check_mem command from client side:

Code: Select all

 /usr/lib64/nagios/plugins/check_mem  -f -w 20 -c 10
CRITICAL - 0.8% (31756 kB) free!|TOTAL=3882020KB;;;; USED=3850264KB;3105616;3493818;; FREE=31756KB;;;; CACHES=3529708KB;;;;

[rkennedy]

Could you run these two and post the output? I'd like to rule out DNS being the factor for whatever reason. The -n will have it run in insecure mode (which still shouldn't be the case, but this is definitely weird).

Code: Select all

/usr/lib64/nagios/plugins/check_nrpe -H <ip> -c check_mem
/usr/lib64/nagios/plugins/check_nrpe -H <servername> -c check_mem -n

[tgriep]

Login to the remote server and run the following to see if this resolves the issue. Run the following and post the output.

Code: Select all

chown apache.nagios /usr/lib64/nagios/plugins/check_mem
chmod ugo+rx /usr/lib64/nagios/plugins/check_mem
su nagios
/usr/lib64/nagios/plugins/check_mem  -f -w 20 -c 10
/usr/lib64/nagios/plugins/check_mem  --help

If this passes, try running it from the Nagios server to see if that fixed it.

[aneah]

Could you run these two and post the output? I'd like to rule out DNS being the factor for whatever reason. The -n will have it run in insecure mode (which still shouldn't be the case, but this is definitely weird).

Code: Select all

/usr/lib64/nagios/plugins/check_nrpe -H <ip> -c check_mem
/usr/lib64/nagios/plugins/check_nrpe -H <servername> -c check_mem -n

I ran the first command from the client using 127.0.0.1 as the IP and the result was

Code: Select all

NRPE: Unable to read output

. When I ran it again on the client, using the client's IP address I got

Code: Select all

CHECK_NRPE: Error - Could not complete SSL handshake.

Running the first command on the Nagios server with the client server IP address returned

Code: Select all

NRPE: Unable to read output

Now, running the second command, from the client using the hostname returned

Code: Select all

CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.

Running that same command from the Nagios server returned

Code: Select all

CHECK_NRPE: Error receiving data from daemon.

[aneah]

Login to the remote server and run the following to see if this resolves the issue. Run the following and post the output.

Code: Select all

chown apache.nagios /usr/lib64/nagios/plugins/check_mem
chmod ugo+rx /usr/lib64/nagios/plugins/check_mem
su nagios
/usr/lib64/nagios/plugins/check_mem  -f -w 20 -c 10
/usr/lib64/nagios/plugins/check_mem  --help

If this passes, try running it from the Nagios server to see if that fixed it.

I tried, as you asked. Here is the output of ls -la /usr/lib64/nagios/plugins/check_mem after the changes:

Code: Select all

-rwxr-xr-x. 1 apache nagios 13429 Mar 30 15:57 check_mem

I attempted to su nagios and received

Code: Select all

This account is currently not available.

This is the same response I receive when attempting to su nagios from the other servers, as root.

Output of /usr/lib64/nagios/plugins/check_mem -f -w 20 -c 10 from the client

Code: Select all

CRITICAL - 1.0% (37800 kB) free!|TOTAL=3882020KB;;;; USED=3844220KB;3105616;3493818;; FREE=37800KB;;;; CACHES=3491072KB;;;;

Output of /usr/lib64/nagios/plugins/check_mem --help from the client

Code: Select all

/usr/lib64/nagios/plugins/check_mem  --help
check_mem.pl v1.0 - Nagios Plugin
usage:
 check_mem.pl -<f|u> -w <warnlevel> -c <critlevel>
options:
 -f           Check FREE memory
 -u           Check USED memory
 -C           Count OS caches as FREE memory
 -w PERCENT   Percent free/used when to warn
 -c PERCENT   Percent free/used when critical
Copyright (C) 2000 Dan Larsson <dl@tyfon.net>
check_mem.pl comes with absolutely NO WARRANTY either implied or explicit
This program is licensed under the terms of the
MIT License (check source code for details)

Having made those changes to the client, I tried /usr/lib64/nagios/plugins/check_nrpe -H <client IP> -c check_mem from the Nagios server and it returned

Code: Select all

NRPE: Unable to read output

I also tried the /usr/lib64/nagios/plugins/check_nrpe -H <client hostname> -c check_mem -n from the Nagios server and it returned

Code: Select all

CHECK_NRPE: Error receiving data from daemon.

[lmiltchev]

What commands did you run in order to install Nagios Plugins and NRPE?

It is very strange that the nagios user is not available...

This account is currently not available.

What is the output of the following commands?

Code: Select all

grep nag /etc/passwd
grep nag /etc/group
grep "nrpe_" /path/to/the/nrpe.cfg
chage nagios -l