check_nrpe -H localhost error

[awilson]

Hi. We have nrpe running on a remote server that is giving the following error when we trigger from the Nagios server or locally.

$ ./check_nrpe -nH localhost
CHECK_NRPE: Error receiving data from daemon.

From outside the server, nmap says that 5666 is filtered, but the network team and the server admins have told us that there are no firewalls blocking the port. Is this something that you recognize?

Thanks.

[lmiltchev]

Is NRPE running on the client (remote machine) under xinetd or as a "standalone" daemon? How did you install it - by following the document below:
https://assets.nagios.com/downloads/nag ... _Agent.pdf
or you compiled it from source, as described in the KB article below?
https://support.nagios.com/kb/article/n ... e-515.html

Run the following commands on the remote machine, and show the output:

Code: Select all

cat /etc/*release
uname -a
ps -ef | grep nrpe | grep -v grep
netstat -lpn | grep 5666
grep allowed_hosts /usr/local/nagios/etc/nrpe.cfg
grep only_from /etc/xinetd.d/nrpe
find / -name "*nrpe*"
iptables -L -n
cat /etc/hosts
/usr/local/nagios/libexec/check_nrpe -H localhost
/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1

[awilson]

See below
I don't have root on the target server. I'll get the iptables -L -n output and add it.
~~~~~~~~~~~~~~~~
cat /etc/*release

Code: Select all

NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"
Red Hat Enterprise Linux Server release 7.6 (Maipo)
Red Hat Enterprise Linux Server release 7.6 (Maipo)
[nagios@server ~]
$

uname -a
Linux server.domain.tld 3.10.0-957.12.2.el7.x86_64 #1 SMP Fri Apr 19 21:09:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

ps -ef | grep nrpe | grep -v grep
{not found}

netstat -lpn | grep 5666
(No info could be read for "-p": geteuid()=1002 but you should be root.)
tcp6 0 0 :::5666 :::* LISTEN -

grep allowed_hosts /usr/local/nagios/etc/nrpe.cfg
allowed_hosts=127.0.0.1,10.10.83.90,10.10.83.97

grep only_from /etc/xinetd.d/nrpe
only_from = 127.0.0.1 10.10.83.90 10.10.83.97

$ find / -name "*nrpe*" 2> /dev/null

Code: Select all

/usr/lib/systemd/system/nrpe.service
/usr/lib/firewalld/services/nrpe.xml
/usr/local/nagios_old/libexec/check_nrpe
/usr/local/nagios_old/bin/nrpe
/usr/local/nagios_old/bin/nrpe-uninstall
/usr/local/nagios_old/etc/nrpe.cfg.old
/usr/local/nagios_old/etc/nrpe.cfg
/usr/local/nagios_old/etc/nrpe.cfg.rpmnew
/usr/local/nagios/libexec/check_nrpe
/usr/local/nagios/bin/nrpe
/usr/local/nagios/etc/nrpe.cfg.rpmsave
/usr/local/nagios/etc/nrpe.cfg
/var/lib/yum/yumdb/n/ffe2a515104e8fd481927606cbdeb9f3674893cd-nagiosxi-nrpe-5.6.2-1.el7-x86_64
/etc/xinetd.d/nrpe.rpmsave
/etc/xinetd.d/nrpe
/opt/puppetlabs/puppet/share/augeas/lenses/dist/tests/test_nrpe.aug
/opt/puppetlabs/puppet/share/augeas/lenses/dist/nrpe.aug

cat /etc/hosts

Code: Select all

# Ansible managed: /export/ansible/etc/hosts on uusrcprat00.domain.tld.  Changes to this file WILL be overwritten

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

//snip

#-- Nagios
172.30.100.36   uusrcpernag00
10.80.86.100    lbschpnagiosxi00.domain.tld
10.10.83.49     lusrcdnagiosmon00.domain.tld
10.10.83.90     lussvpnagiosxi00.domain.tld
10.10.83.91     lussvpnagiosfus00.domain.tld
10.10.83.92     lussvpnagiosfus01.domain.tld
10.10.83.93     lussvpnagiosdb00.domain.tld
10.10.83.94     lussvpnagiosdb01.domain.tld
10.10.83.95     lussvpnagiosmid00.domain.tld
10.10.83.96     lussvpnagiosmid01.domain.tld
10.10.83.97     lusrcdnagiosxi00.domain.tld
10.10.83.98     lusrcdnagiosfus00.domain.tld
10.10.83.100    lusrcdnagiosmid00.domain.tld
10.10.83.217    lusrcdnag02.domain.tld
10.10.83.216    lusrcdnnv01.domain.tld
10.130.80.90    lcnhkpnagiosxi00.domain.tld

#------------------------------------------------------------------------------
//snip

#-- Entry for this server
10.10.84.89     server.domain.tld  server

$ /usr/local/nagios/libexec/check_nrpe -H localhost
CHECK_NRPE: Error - Could not complete SSL handshake.

$ /usr/local/nagios/libexec/check_nrpe -nH localhost
CHECK_NRPE: Error receiving data from daemon.

$ /usr/local/nagios/libexec/check_nrpe -H 127.0.0.1
NRPE v2.15

[lmiltchev]

$ /usr/local/nagios/libexec/check_nrpe -H localhost
CHECK_NRPE: Error - Could not complete SSL handshake.

This command doesn't work because (most likely) NRPE is only listening on IPv6. Try adding the following to "only_from" line in /etc/xinetd.d/nrpe:

only_from = ::1 127.0.0.1 10.10.83.90 10.10.83.97

and restart xinet, so that changes can take effect:

Code: Select all

service xinetd restart

Test your check again. Is it working now?

$ /usr/local/nagios/libexec/check_nrpe -nH localhost
CHECK_NRPE: Error receiving data from daemon.

This is not going to work as you are telling NRPE not to use SSL by passing "-n".

$ /usr/local/nagios/libexec/check_nrpe -H 127.0.0.1
NRPE v2.15

This works as expected, however you are using a very old agent. I would recommend that you upgrade to NRPE v3. The way to do this would be to remove NRPE completely out of your system, then use our official Linux agent installer:

https://assets.nagios.com/downloads/nag ... _Agent.pdf

Note: I would like to point out that you don't have a "typical", vanilla setup. I see IP entries in both, the /usr/local/nagios/etc/nrpe.cfg and /etc/xinetd.d/nrpe file... It seems like that NRPE on your system is running under xinetd, just by looking at this:

ps -ef | grep nrpe | grep -v grep
{not found}

Keep in mind that NRPE can run either under xinetd or as a "standalone" daemon, but not as both...

I see many "non-standard" files on this system. Are you using puppet to manage nrpe configs? What is the purpose of this file - /usr/lib/firewalld/services/nrpe.xml? NRPE is running under xinetd, but you also have a service set up:

/usr/lib/systemd/system/nrpe.service

So, if you don't want to install a newer version of NRPE, then we can continue troubleshooting your existing setup.

1. Make sure that your Nagios XI server's IP address is one of the IPs, listed in the /etc/xinetd.d/nrpe file. If you are making any changes to the file, you would need to restart xinetd, so that changes can take effect.

2. Test connectivity from the Nagios XI server by running:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H <client ip>
/usr/local/nagios/libexec/check_nrpe -2 -H <client ip>
nmap <client ip> -p 5666

[awilson]

I'm not able to change the nrpe version now. We are in a year-end freeze. We'll start that after the new year.

I think that that nrpe.service and the firewalld nrpe.xml file was added by RHEL from what was found in /etc/rd.d/init.d. We aren't using puppet. Ansible is used in the environment, but I'm fairly certain that it is not adjusting the nrpe configuration. I'm getting confirmation on these.

Code: Select all

[nagios@nagioshost~]$ /usr/local/nagios/libexec/check_nrpe -H server
NRPE v3.2.1
[nagios@nagioshost ~]$ /usr/local/nagios/libexec/check_nrpe -2 -H server
/usr/local/nagios/libexec/check_nrpe: invalid option -- '2'

NRPE Plugin for Nagios
Copyright (c) 1999-2008 Ethan Galstad ([email protected])
Version: 2.15
Last Modified: 09-06-2013
License: GPL v2 with exemptions (-l for more info)
SSL/TLS Available: Anonymous DH Mode, OpenSSL 0.9.6 or higher required

nmap

Code: Select all

[root@lusrcpnagmn01 ~]# nmap uusrcqpla10

Starting Nmap 6.47 ( http://nmap.org ) at 2019-11-07 14:20 CST
Nmap scan report for server (10.10.x.x)
Host is up (0.00029s latency).
Not shown: 993 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
111/tcp  open  rpcbind
5001/tcp open  commplex-link
5666/tcp open  nrpe
6666/tcp open  irc
8009/tcp open  ajp13
8080/tcp open  http-proxy

Nmap done: 1 IP address (1 host up) scanned in 16.60 seconds

[lmiltchev]

Let's step back for a minute. When I asked you to run check_nrpe against 127.0.0.1 on the remote machine (client), you showed us this:

$ /usr/local/nagios/libexec/check_nrpe -H 127.0.0.1
NRPE v2.15

Then, I asked you to run check_nrpe on the Nagios XI server against the remote machine, and you showed us this:

[nagios@nagioshost~]$ /usr/local/nagios/libexec/check_nrpe -H server
NRPE v3.2.1

How is this possible?

Can you run the following commands on both machines, Nagios XI and remote box, and show the output?

Code: Select all

ip addr
/usr/local/nagios/bin/nrpe -V

In any case, if you can run this command successfully:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H server

then I don't see what the issue is... Am I missing something?

[nagios@nagioshost ~]$ /usr/local/nagios/libexec/check_nrpe -2 -H server
/usr/local/nagios/libexec/check_nrpe: invalid option -- '2'

Passing "-2" to the command won't work with the old versions of NRPE. You have to be using ver. 3.

[awilson]

I made an error with the
[nagios@nagioshost~]$ /usr/local/nagios/libexec/check_nrpe -H server
NRPE v3.2.1
I had too many windows open. //sigh

It looks like the IPv6 fix will take care of it. You can close this.

Thanks!

[lmiltchev]

I am glad I could help!