Nagios Support Forum

Thanks guys, sorry i've been busy doing some other tasks.

Here is the output


[root@TDMAURCMESERV32 ~]# iptables -L -v -n -x
Chain INPUT (policy ACCEPT 3419050248 packets, 696264461404 bytes)
pkts bytes target prot opt in out source destination
7 404 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5666
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5666

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 3443543859 packets, 695464845902 bytes)
pkts bytes target prot opt in out source destination
[root@TDMAURCMESERV32 ~]#

So iptables isn't the problem. In fact, you could just turn it off to double check (service iptables stop). I'll have to re-read your previous notes to see what pops up in my head.

Did you investigate what @millisa wrote:

The one other bit that jumps out at me is that this is RHEL6.3. I remember somewhere around 6.3 there was an errata update for the selinux-policy packages that corrected one of the issues you'd hit with NRPE. (I believe it was this one. If you have selinux enabled, you may want to consider updating (6.4 came out in February 2013, 6.5 was November of 2013). Unless it's been reconfigured, I believe it logs to /var/log/audit/audit.log, it's probably worth checking when you run the check.

If the machine is listening on port 5666, and your iptables aren't blocking anything (and they're not, in fact, they're wide open), then selinux is the only thing that comes to mind as being able to prevent the response from occurring properly.

To add to eloyd's post: you may want to just temporarily disable selinux to test if that is the issue:

Hi Gents,

It looks like SELINUX is already disabled:

[root@:~] getenforce
Disabled


Any other ideas?

My idea is that something got lost in translation somewhere in this message thread. So let's start over with a specific set of commands. If you could paste these commands into a shell window on the remote machine as root, and then paste the output from each command, that would be awesome. I realize that some of these commands may produce no output, but these are the things I would check if I were in front of your keyboard. PLEASE NOTE: This assumes that eth0 is your primary network interface. If it is not, then adjust appropriately.
Thanks!

Thanks eloyd:

[root@TDMAURCMESERV32 ~]# cat /etc/xinetd.d/nrpe
# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
flags = REUSE
socket_type = stream
port = 5666
wait = no
user = nagios
group = nagios
server = /usr/local/nagios/bin/nrpe
server_args = -c /usr/local/nagios/etc/nrpe.cfg --inetd
log_on_failure += USERID
disable = no
only_from = 127.0.0.1, 192.168.0.29
}

[root@TDMAURCMESERV32 ~]# chkconfig --list | grep 3:on
abrt-ccpp 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrt-oops 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrtd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
acpid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
certmonger 0:off 1:off 2:off 3:on 4:on 5:on 6:off
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
haldaemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off
informix 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ip6tables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
irqbalance 0:off 1:off 2:off 3:on 4:on 5:on 6:off
kdump 0:off 1:off 2:off 3:on 4:on 5:on 6:off
lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off
mcelogd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
mdmonitor 0:off 1:off 2:on 3:on 4:on 5:on 6:off
messagebus 0:off 1:off 2:on 3:on 4:on 5:on 6:off
netfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
nfslock 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ntpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
orc 0:off 1:off 2:on 3:on 4:on 5:on 6:off
orcgate 0:off 1:off 2:on 3:on 4:on 5:on 6:off
portreserve 0:off 1:off 2:on 3:on 4:on 5:on 6:off
postfix 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rhnsd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rhsmcertd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rpcbind 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rpcgssd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rpcidmapd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sysstat 0:off 1:on 2:on 3:on 4:on 5:on 6:off
udev-post 0:off 1:on 2:on 3:on 4:on 5:on 6:off
vsftpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
xinetd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
[root@TDMAURCMESERV32 ~]#

[root@TDMAURCMESERV32 ~]# iptables -L -v -n -x
Chain INPUT (policy ACCEPT 5934002721 packets, 1204838745376 bytes)
pkts bytes target prot opt in out source destination
7 404 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5666
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5666

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 5936415567 packets, 1209499608850 bytes)
pkts bytes target prot opt in out source destination
[root@TDMAURCMESERV32 ~]#

[root@TDMAURCMESERV32 ~]# fail2ban status
-bash: fail2ban: command not found
[root@TDMAURCMESERV32 ~]#

[root@TDMAURCMESERV32 ~]# lsof -i:5666
[root@TDMAURCMESERV32 ~]#

[root@TDMAURCMESERV32 ~]# netstat -na | grep 5666
[root@TDMAURCMESERV32 ~]#

[root@TDMAURCMESERV32 ~]# nmap -P0 -p 5666 localhost

Starting Nmap 5.51 ( http://nmap.org ) at 2014-09-24 10:00 CDT
Failed to find device bond0 which was referenced in /proc/net/route
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000025s latency).
Other addresses for localhost (not scanned): 127.0.0.1
PORT STATE SERVICE
5666/tcp closed nrpe

Nmap done: 1 IP address (1 host up) scanned in 0.07 seconds
[root@TDMAURCMESERV32 ~]#


[root@TDMAURCMESERV32 ~]# nmap -P0 -p 5666 `ifconfig em1 | head -2 | tail -1 | awk '{print $2}' | awk -F: '{print $2}'`

Starting Nmap 5.51 ( http://nmap.org ) at 2014-09-24 10:03 CDT
Failed to find device bond0 which was referenced in /proc/net/route
Nmap scan report for TDMAURCMESERV32 (10.126.160.26)
Host is up (0.000024s latency).
PORT STATE SERVICE
5666/tcp closed nrpe

Nmap done: 1 IP address (1 host up) scanned in 0.06 seconds
[root@TDMAURCMESERV32 ~]#

[root@TDMAURCMESERV32 ~]# ls -l /usr/local/nagios/libexec/check_nrpe
-rwxrwxr-x 1 nagios nagios 76825 Sep 15 10:32 /usr/local/nagios/libexec/check_nrpe
[root@TDMAURCMESERV32 ~]#

[root@TDMAURCMESERV32 ~]# /usr/local/nagios/libexec/check_nrpe -H localhost
connect to address ::1 port 5666: Connection refused
connect to address 127.0.0.1 port 5666: Connection refused
connect to host localhost port 5666: Connection refused[root@TDMAURCMESERV32 ~]#

[root@TDMAURCMESERV32 ~]# /usr/local/nagios/libexec/check_nrpe -H localhost -n
connect to address ::1 port 5666: Connection refused
connect to address 127.0.0.1 port 5666: Connection refused
connect to host localhost port 5666: Connection refused[root@TDMAURCMESERV32 ~]#


Thanks for all your help!

NRPE is, quite simply, not running. Otherwise "lsof -i:5666" would have shown something listening.

Next commands: Just interested in the bottom, where it shows the "xinetd based services" stuff. Can you copy/paste the line with "nrpe" in it, please?

xinetd based services:
chargen-dgram: off
chargen-stream: off
cvs: off
daytime-dgram: off
daytime-stream: off
discard-dgram: off
discard-stream: off
echo-dgram: off
echo-stream: off
nrpe: on
rsync: off
tcpmux-server: off
time-dgram: off
time-stream: off
[root@TDMAURCMESERV32 ~]#

Okay, so I would say at this point that "it's broken."

If xinetd thinks it's supposed to be running NRPE, but it doesn't have port 5666 listed in the output of lsof (see below for what it should look like) then something is wrong with xinetd and its ability to run NRPE. A normal lsof output would look like this for a box that runs NRPE out of xinetd: How about this. Try running the nrpe daemon itself from the command line, as both root and after becoming nagios (on the remote box): I expect the first one to fail since nrpe won't start as root (confirm by looking at last lines of /var/log/messages). I expect the second one to work and just sit there waiting for stuff to happen.