NRPE fails after Nagios XI upgrade

[bchabotdg]

Running CentOS release 6.9 (Final) and just updated Nagios XI to 5.5.2.

I have a series of checks on the local machine that use nrpe which began failing with "Connection reset by peer" after the update.

Verified this by commandline:

Code: Select all

# /usr/local/nagios/libexec/check_nrpe -H 1.2.3.4
CHECK_NRPE: Error - Could not connect to 1.2.3.4: Connection reset by peer
#

Oddly enough, the check by IP or hostname both fail, but using "localhost" works:

Code: Select all

# /usr/local/nagios/libexec/check_nrpe -H MYHOSTNAME
CHECK_NRPE: Error - Could not connect to 1.2.3.4: Connection reset by peer
# /usr/local/nagios/libexec/check_nrpe -H localhost
NRPE v3.2.1
#

iptables shows the port is allowing traffic:

Code: Select all

# service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
...
9    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:5666
...

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

# 

Stopping iptables doesn't help:

Code: Select all

# service iptables stop
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Unloading modules:                               [  OK  ]
# /usr/local/nagios/libexec/check_nrpe -H 1.2.3.4
CHECK_NRPE: Error - Could not connect to 1.2.3.4: Connection reset by peer
#

In nrpe.cfg, I have the following for allowed hosts:

Code: Select all

allowed_hosts=127.0.0.1,::1,1.2.3.4/22

What am I missing here?

[npolovenko]

Hello, @bchabotdg. Let's rerun the check with forced version 2 packets:

/usr/local/nagios/libexec/check_nrpe -2 -H 1.2.3.4

Let me know if this works for you.

[bchabotdg]

Same. Connection reset by peer.

[npolovenko]

@bchabotdg, If the nrpe agent is running under xinetd you'd need to modify this file:

/etc/xinetd.d/nrpe

only_from = 127.0.0.1 1.2.3.4

IP addresses are separated by spaces with no commas inbetween.

After editing the file run:

service xinetd restart

[bchabotdg]

It was:

Code: Select all

# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
        flags           = REUSE
        socket_type     = stream
        port            = 5666
        wait            = no
        user            = nagios
        group           = nagios
        server          = /usr/local/nagios/bin/nrpe
        server_args     = -c /usr/local/nagios/etc/nrpe.cfg --inetd
        log_on_failure  += USERID
        disable = yes
        only_from       = 1.2.3.4 1.2.3.4
}

It is now:

Code: Select all

# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
        flags           = REUSE
        socket_type     = stream
        port            = 5666
        wait            = no
        user            = nagios
        group           = nagios
        server          = /usr/local/nagios/bin/nrpe
        server_args     = -c /usr/local/nagios/etc/nrpe.cfg --inetd
        log_on_failure  += USERID
        disable = yes
        only_from       = 127.0.0.1 1.2.3.4
}

Restarted xinetd.

No change. IP and hostname fail. "localhost" works.

[npolovenko]

@bchabotdg, Can you run

hostname -I

And make sure that it matches 1.2.3.4?

Please show me the output of these commands:

ps aux | grep nrpe
ps aux | grep xinetd

[bchabotdg]

The IP addresses match.

Code: Select all

# ps aux | grep nrpe
nagios   11347  0.0  0.0  41440  3004 ?        S    15:45   0:00 /usr/local/nagios/libexec/check_nrpe -H lunk-01 -t 45 -c check_cpu_stats -a -w 85 -c 95
root     11723  0.0  0.0 103320   900 pts/0    R+   15:45   0:00 grep nrpe
nagios   23490  0.0  0.0  43588  1500 ?        Ss   15:35   0:00 /usr/local/nagios/bin/nrpe -c /usr/local/nagios/etc/nrpe.cfg -d
# ps aux | grep xinetd
root     11905  0.0  0.0 103320   908 pts/0    S+   15:45   0:00 grep xinetd
root     17884  0.0  0.0  21712  1012 ?        Ss   15:22   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
#

[npolovenko]

@bchabotdg, You can turn on the debugging option in the nrpe.cfg file:

# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the
# syslog facility.
# Values: 0=debugging off, 1=debugging on

debug=0

Then restart the nrpe with:

service nrpe restart

Attempt the command a few times and look for entries in /var/log/messages or /var/log/syslog:

/usr/local/nagios/libexec/check_nrpe -H 1.2.3.4

Also, please upload the:

/var/log/httpd/acess_log

[bchabotdg]

Not sure what the http access log has to do with this.

Changed to debug=1. Restarted nrpe.

# vi /usr/local/nagios/etc/nrpe.cfg
# service nrpe restart
Shutting down nrpe [ OK ]
Starting nrpe [ OK ]
# tail /usr/local/nagios/var/nrpe.log
[1535056484] Added command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
[1535056484] Added command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200
[1535056484] INFO: SSL/TLS initialized. All network traffic will be encrypted.
[1535056484] Starting up daemon
[1535056484] SETUP_WAIT_CONN FOR: IPv4 address: 0.0.0.0 ((null))
[1535056484] Server listening on 0.0.0.0 port 5666.
[1535056484] SETUP_WAIT_CONN FOR: IPv4 address: :: ((null))
[1535056484] Server listening on :: port 5666.
[1535056484] Listening for connections on port 5666
[1535056484] Allowing connections from: 127.0.0.1,::1,1.2.3.4/22
# /usr/local/nagios/libexec/check_nrpe -H 1.2.3.4
CHECK_NRPE: Error - Could not connect to 1.2.3.4: Connection reset by peer
# tail /usr/local/nagios/var/nrpe.log
[1535056497] CONN_CHECK_PEER: checking if host is allowed: 1.2.3.4 port 36545
[1535056497] Connection from 1.2.3.4 port 36545
[1535056497] is_an_allowed_host (AF_INET): is host >1.2.3.4< an allowed host >1.2.3.4<
[1535056497] Host 1.2.3.4 is not allowed to talk to us!
[1535056497] Connection from 1.2.3.4 closed.
[1535056503] CONN_CHECK_PEER: checking if host is allowed: 1.2.3.4 port 39108
[1535056503] Connection from 10.23.12.85 port 39108
[1535056503] is_an_allowed_host (AF_INET): is host >1.2.3.4< an allowed host >1.2.3.4<
[1535056503] Host 1.2.3.4 is not allowed to talk to us!
[1535056503] Connection from 1.2.3.4 closed.
#

[npolovenko]

@bchabotdg, Have you reset the nrpe after making changes in the nrpe.cfg file?

service nrpe restart

Try rebooting the server as well.
Try removing the ipv6 option ::1 from the allowed_hosts as well as /22 for testing purposes, and then reset the nrpe again.