NRPE connection refused

[rferebee]

Hello,

I've tried everything I can think of and I can't figure out what's going on. The NRPE service checks for my Nagios XI test server are reporting "connection refused" on port 5666. I know without a doubt this isn't a network permit issue.

It started after I manually performed a restore of XI from the backup of my failover server. This involved restoring a profile from a CentOS 6 box to a CentOS 7 box. After the restore I ran the repair script and everything came up fine with the exception of the NRPE service check.

Here's the command and the resulting message:

/usr/local/nagios/libexec/check_tcp -H 10.131.86.60 -p 5666
connect to address 10.131.86.60 and port 5666: Connection refused

I think it has to do with a configuration somewhere on the box, but I can't find it.

[mbellerue]

This involved restoring a profile from a CentOS 6 box to a CentOS 7 box.

Just a quick verification, is this new CentOS 7 box's IP address is listed in the NRPE client's allowed hosts?

If so, then we'll want to verify that there's no firewall running on the host you're trying to monitor. Is this a Windows host? I know Microsoft likes to tinker with firewall settings during Windows updates.

Also verify that the NRPE client is actually running.

I know, these are all of the easy steps. But sometimes it helps to retrace them just in case one was missed.

[rferebee]

The box has the same IP address as the box it replaced. We built a new box, installed Nagios XI on it, restored the profile from the old CentOS 6 box (repaired after restore), shutdown the CentOS 6 box and moved it's network config over to this CentOS 7 box.

The firewall is not running on the host. It's a Linux host.

I checked the status of the NRPE service and got the following:

root@nagiostest:/root>systemctl status nrpe
● nrpe.service - Nagios Remote Plugin Executor
Loaded: loaded (/usr/lib/systemd/system/nrpe.service; disabled; vendor preset: disabled)
Active: inactive (dead)
Docs: http://www.nagios.org/documentation
root@nagiostest:/root>systemctl start nrpe
root@nagiostest:/root>systemctl status nrpe
● nrpe.service - Nagios Remote Plugin Executor
Loaded: loaded (/usr/lib/systemd/system/nrpe.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2019-08-27 13:46:58 PDT; 3s ago
Docs: http://www.nagios.org/documentation
Process: 10957 ExecStopPost=/bin/rm -f /usr/local/nagios/var/nrpe.pid (code=exited, status=0/SUCCESS)
Process: 10954 ExecStart=/usr/local/nagios/bin/nrpe -c /usr/local/nagios/etc/nrpe.cfg -f (code=exited, status=2)
Main PID: 10954 (code=exited, status=2)

Aug 27 13:46:58 nagiostest.state.nv.us systemd[1]: Started Nagios Remote Plugin Executor.
Aug 27 13:46:58 nagiostest.state.nv.us nrpe[10954]: Starting up daemon
Aug 27 13:46:58 nagiostest.state.nv.us nrpe[10954]: Cannot write to pidfile '/var/run/nrpe.pid' - check your privileges.
Aug 27 13:46:58 nagiostest.state.nv.us systemd[1]: nrpe.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 27 13:46:58 nagiostest.state.nv.us systemd[1]: Unit nrpe.service entered failed state.
Aug 27 13:46:58 nagiostest.state.nv.us systemd[1]: nrpe.service failed.

Is that how I would start NRPE if I'm not using the binary package?

[mbellerue]

Aug 27 13:46:58 nagiostest.state.nv.us nrpe[10954]: Cannot write to pidfile '/var/run/nrpe.pid' - check your privileges.
Aug 27 13:46:58 nagiostest.state.nv.us systemd[1]: nrpe.service: main process exited, code=exited, status=2/INVALIDARGUMENT

Well that's something. It looks like you're starting the service as root, so there shouldn't be a problem with permissions. Maybe do a ps -aux | grep -i nrpe and see if anything comes back. The only thing that should come back is your grep command. If more than that comes back, I bet there's a ghost process out there. Or maybe if there's already a /var/run/nrpe.pid?

[rferebee]

I ran that grep and got this:

root@nagiostest:/root>ps -aux | grep -i nrpe
nagios 69613 0.0 0.0 40528 2500 ? S 14:30 0:00 /usr/local/nagios/libexec/check_nrpe -2 -H 10.131.143.154 -t 30 -c check_memory -a type=physical warn=free<20% crit=free<10%
root 70073 0.0 0.0 112712 996 pts/0 S+ 14:30 0:00 grep --color=auto -i nrpe

[tgriep]

Try running this command as toot to see if the NRPE agent is getting started by xinetd.

Code: Select all

netstat -anp |grep 5666

If it shows that xinetd starts it, make sure the Nagios XI server's IP address is in the /etc/xinetd.d/nrpe folder.

Also, look at this KB article for some troubleshooting steps.
https://support.nagios.com/kb/article/n ... t-622.html

[rferebee]

It doesn't look like my server is even listening on port 5666.

When I look at a netstat output on the other two servers that are working, I see it listening on port 5666 any/any basically. But, when I do netstat from my test server (the one that stopped working) I don't see the line that says it's listening on port 5666.

I have confirmed that xinetd is running, so I'm not sure why it isn't listening. We're not using the firewall on this box, so there's nothing to configure in that arena.

[lmiltchev]

Can you run the following commands and show the output in code wraps?

Code: Select all

yum list installed | grep -i nrpe
cat /etc/xinetd.d/nrpe
ls -lad /var/run/
ls -la /var/run/ | grep nrpe
/usr/local/nagios/bin/nrpe -V
find / -name "nrpe*"

Also, post the nrpe.cfg file on the forum.

[rferebee]

PM sent. Thank you.

[rferebee]

I think my test server has the binary package installed where as the other servers do not.

The servers that are working don't even have /etc/nagios/nrpe.cfg