Configuring NRPE client CentOS7

[rferebee]

Hello,

I'm having a heck of a time figuring out this latest problem.

As you already know, I recently introduced an additional node into my Log Server cluster. I am trying to setup Nagios XI service checks for the new host and the NRPE checks aren't working.

Everything appears to be the setup and configured the same way between the server that is working and the server that isn't. The only major difference is that the one that is working is CentOS 6 and the one that isn't is CentOS 7.

It looks like NRPE is running from xinetd.d, but the only status I get from the new server for the Logstash and Elasticsearch service checks is: CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected).

Here's the actual check being ran against the server: /usr/local/nagios/libexec/check_nrpe -2 -H 10.128.207.115 -c check_services -t 30 -a 'elasticsearch'

When I run that directly on the server, I get the following: CHECK_NRPE: Error - Could not connect to 10.128.207.115: Connection reset by peer

I'm not sure what else to look at. I was going through this document: https://support.nagios.com/kb/article/n ... e-615.html

I thought maybe it might be the SSL Handshake issue, but this directory (/tmp/nrpe-2.14) does not exist on the server. So, I can't try to enable SSL per the instructions.

Thank you.

[npolovenko]

@rferebee, Is the nrpe agent actually running on the log server? Please open a terminal console to the log server, run these commands and post the output in the thread:

ps -ef | grep nrpe
ps -ef | grep xinetd
netstat -anp

Also, please upload /usr/local/nagios/etc/nrpe.cfg file from the log server.

[rferebee]

Here's the output you requested:

root@nagioslscc4:/root>ps -ef | grep nrpe
nrpe 18112 1 0 08:23 ? 00:00:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f
root 47860 47815 0 14:54 pts/0 00:00:00 grep --color=auto nrpe
root@nagioslscc4:/root>ps -ef | grep xinetd
root 45468 1 0 14:23 ? 00:00:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid
root 47871 47815 0 14:54 pts/0 00:00:00 grep --color=auto xinetd
root@nagioslscc4:/root>netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:34374 0.0.0.0:* LISTEN 11244/rpc.statd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 11651/dnsmasq
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 11185/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 11176/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 11475/master
tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN 18112/nrpe
tcp 0 0 127.0.0.1:46192 127.0.0.1:9200 TIME_WAIT -
tcp 0 64 10.128.207.115:22 10.131.32.33:45884 ESTABLISHED 47749/sshd: rferebe
tcp 0 0 127.0.0.1:46190 127.0.0.1:9200 TIME_WAIT -
tcp 0 0 10.128.207.115:5666 10.131.86.58:34374 TIME_WAIT -
tcp6 0 0 :::54412 :::* LISTEN 11244/rpc.statd
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 51515/java
tcp6 0 0 :::80 :::* LISTEN 51542/httpd
tcp6 0 0 :::9300 :::* LISTEN 51515/java
tcp6 0 0 :::22 :::* LISTEN 11185/sshd
tcp6 0 0 ::1:631 :::* LISTEN 11176/cupsd
tcp6 0 0 ::1:25 :::* LISTEN 11475/master
tcp6 0 0 :::5666 :::* LISTEN 18112/nrpe
tcp6 0 0 10.128.207.115:37122 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:37134 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37122 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37128 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:37136 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37124 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37134 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37140 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:37124 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37138 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:37144 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:37140 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:37130 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37126 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37116 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:37138 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:37142 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:37132 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37136 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:37128 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37144 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37142 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:37116 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37130 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:37126 10.128.207.115:9300 ESTABLISHED 51515/java
tcp6 0 0 10.128.207.115:9300 10.128.207.115:37132 ESTABLISHED 51515/java
udp 0 0 192.168.122.1:53 0.0.0.0:* 11651/dnsmasq
udp 0 0 0.0.0.0:67 0.0.0.0:* 11651/dnsmasq
udp 0 0 0.0.0.0:111 0.0.0.0:* 1/systemd
udp 0 0 192.168.122.1:123 0.0.0.0:* 51387/ntpd
udp 0 0 10.128.207.115:123 0.0.0.0:* 51387/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 51387/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 51387/ntpd
udp 0 0 0.0.0.0:33156 0.0.0.0:* 11244/rpc.statd
udp 0 0 0.0.0.0:798 0.0.0.0:* 11223/rpcbind
udp 0 0 127.0.0.1:820 0.0.0.0:* 11244/rpc.statd
udp 0 0 0.0.0.0:5353 0.0.0.0:* 10882/avahi-daemon:
udp 0 0 0.0.0.0:39031 0.0.0.0:* 10882/avahi-daemon:
udp6 0 0 :::111 :::* 1/systemd
udp6 0 0 fe80::250:56ff:fea0:123 :::* 51387/ntpd
udp6 0 0 ::1:123 :::* 51387/ntpd
udp6 0 0 :::123 :::* 51387/ntpd
udp6 0 0 :::798 :::* 11223/rpcbind
udp6 0 0 :::34577 :::* 11244/rpc.statd
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 3 [ ] DGRAM 12298 1/systemd /run/systemd/notify
unix 2 [ ] DGRAM 12300 1/systemd /run/systemd/cgroups-agent
unix 2 [ ACC ] STREAM LISTENING 12314 1/systemd /run/systemd/journal/stdout
unix 5 [ ] DGRAM 12317 1/systemd /run/systemd/journal/socket
unix 2 [ ACC ] STREAM LISTENING 52511 10874/VGAuthService /var/run/vmware/guestServicePipe
unix 22 [ ] DGRAM 12319 1/systemd /dev/log
unix 2 [ ACC ] STREAM LISTENING 51358 10863/gssproxy /var/lib/gssproxy/default.sock
unix 2 [ ACC ] STREAM LISTENING 47584 1/systemd @ISCSID_UIP_ABSTRACT_NAMESPACE
unix 2 [ ACC ] STREAM LISTENING 59484 11294/libvirtd /var/run/libvirt/libvirt-sock
unix 2 [ ACC ] STREAM LISTENING 29787 1/systemd /run/lvm/lvmetad.socket
unix 2 [ ACC ] STREAM LISTENING 59486 11294/libvirtd /var/run/libvirt/libvirt-sock-ro
unix 2 [ ACC ] STREAM LISTENING 59488 11294/libvirtd /var/run/libvirt/libvirt-admin-sock
unix 2 [ ACC ] STREAM LISTENING 56423 10875/abrtd /var/run/abrt/abrt.socket
unix 2 [ ACC ] STREAM LISTENING 52209 11475/master public/pickup
unix 2 [ ACC ] STREAM LISTENING 52213 11475/master public/cleanup
unix 2 [ ACC ] STREAM LISTENING 52216 11475/master public/qmgr
unix 2 [ ACC ] STREAM LISTENING 52220 11475/master private/tlsmgr
unix 2 [ ACC ] STREAM LISTENING 52223 11475/master private/rewrite
unix 2 [ ACC ] STREAM LISTENING 59394 11475/master private/bounce
unix 2 [ ACC ] STREAM LISTENING 59397 11475/master private/defer
unix 2 [ ACC ] STREAM LISTENING 59400 11475/master private/trace
unix 2 [ ACC ] STREAM LISTENING 59403 11475/master private/verify
unix 2 [ ACC ] STREAM LISTENING 59406 11475/master public/flush
unix 2 [ ACC ] STREAM LISTENING 59409 11475/master private/proxymap
unix 2 [ ACC ] STREAM LISTENING 59412 11475/master private/proxywrite
unix 2 [ ACC ] STREAM LISTENING 59415 11475/master private/smtp
unix 2 [ ACC ] STREAM LISTENING 59418 11475/master private/relay
unix 2 [ ACC ] STREAM LISTENING 59421 11475/master public/showq
unix 2 [ ACC ] STREAM LISTENING 59424 11475/master private/error
unix 2 [ ACC ] STREAM LISTENING 59427 11475/master private/retry
unix 2 [ ACC ] STREAM LISTENING 59430 11475/master private/discard
unix 2 [ ACC ] STREAM LISTENING 59433 11475/master private/local
unix 2 [ ACC ] STREAM LISTENING 59436 11475/master private/virtual
unix 2 [ ACC ] STREAM LISTENING 59439 11475/master private/lmtp
unix 2 [ ACC ] STREAM LISTENING 59442 11475/master private/anvil
unix 2 [ ACC ] STREAM LISTENING 59445 11475/master private/scache
unix 2 [ ACC ] STREAM LISTENING 29839 1/systemd /run/lvm/lvmpolld.socket
unix 2 [ ACC ] SEQPACKET LISTENING 29841 1/systemd /run/udev/control
unix 2 [ ACC ] STREAM LISTENING 51359 10863/gssproxy /run/gssproxy.sock
unix 2 [ ACC ] STREAM LISTENING 47577 1/systemd @ISCSIADM_ABSTRACT_NAMESPACE
unix 2 [ ACC ] STREAM LISTENING 50890 10852/lsmd /var/run/lsm/ipc/sim
unix 2 [ ACC ] STREAM LISTENING 47566 1/systemd /var/run/cups/cups.sock
unix 2 [ ACC ] STREAM LISTENING 50895 10852/lsmd /var/run/lsm/ipc/simc
unix 2 [ ACC ] STREAM LISTENING 47568 1/systemd /var/run/avahi-daemon/socket
unix 2 [ ACC ] STREAM LISTENING 47571 1/systemd /var/run/libvirt/virtlockd-sock
unix 2 [ ACC ] STREAM LISTENING 47574 1/systemd /run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 47578 1/systemd /var/run/rpcbind.sock
unix 2 [ ACC ] STREAM LISTENING 3353312 1/systemd /run/systemd/private
unix 2 [ ACC ] STREAM LISTENING 47585 1/systemd /var/run/libvirt/virtlogd-sock
unix 2 [ ] DGRAM 27128 1/systemd /run/systemd/shutdownd
unix 3 [ ] STREAM CONNECTED 60440 11294/libvirtd
unix 3 [ ] STREAM CONNECTED 52214 11475/master
unix 3 [ ] STREAM CONNECTED 52293 10872/rngd
unix 3 [ ] STREAM CONNECTED 66205 22373/packagekitd
unix 3 [ ] STREAM CONNECTED 52294 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 4276967 47778/abrt-dbus
unix 2 [ ] DGRAM 24296 5544/systemd-udevd
unix 3 [ ] STREAM CONNECTED 52358 10874/VGAuthService
unix 3 [ ] STREAM CONNECTED 3358761 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 52211 11475/master
unix 3 [ ] STREAM CONNECTED 52359 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 59444 11475/master
unix 3 [ ] STREAM CONNECTED 52227 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 4275725 1/systemd /run/systemd/journal/stdout
unix 2 [ ] STREAM CONNECTED 3356236 51515/java
unix 3 [ ] STREAM CONNECTED 52215 11475/master
unix 3 [ ] STREAM CONNECTED 49590 10849/systemd-login
unix 2 [ ] DGRAM 49139 10863/gssproxy
unix 2 [ ] DGRAM 55285 11481/qmgr
unix 2 [ ] DGRAM 3345401 50438/rsyslogd
unix 3 [ ] STREAM CONNECTED 52207 11475/master
unix 2 [ ] DGRAM 60594 11651/dnsmasq
unix 3 [ ] STREAM CONNECTED 3358041 51542/httpd
unix 3 [ ] STREAM CONNECTED 59447 11475/master
unix 3 [ ] STREAM CONNECTED 59395 11475/master
unix 3 [ ] STREAM CONNECTED 51331 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 4276968 47778/abrt-dbus
unix 2 [ ] DGRAM 3333840 51006/crond
unix 2 [ ] DGRAM 4137660 1/systemd
unix 3 [ ] STREAM CONNECTED 59396 11475/master
unix 3 [ ] STREAM CONNECTED 52218 11475/master
unix 3 [ ] STREAM CONNECTED 59493 10856/dbus-daemon /run/dbus/system_bus_socket
unix 2 [ ] DGRAM 4261709 44587/pickup
unix 3 [ ] STREAM CONNECTED 67065 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 59446 11475/master
unix 3 [ ] STREAM CONNECTED 59393 11475/master
unix 3 [ ] STREAM CONNECTED 52217 11475/master
unix 3 [ ] STREAM CONNECTED 59440 11475/master
unix 3 [ ] STREAM CONNECTED 59398 11475/master
unix 3 [ ] STREAM CONNECTED 52210 11475/master
unix 3 [ ] STREAM CONNECTED 51341 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 59443 11475/master
unix 3 [ ] STREAM CONNECTED 49138 10863/gssproxy
unix 3 [ ] STREAM CONNECTED 56416 10885/polkitd
unix 3 [ ] STREAM CONNECTED 49652 10869/irqbalance
unix 3 [ ] STREAM CONNECTED 4275726 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 52208 11475/master
unix 2 [ ] DGRAM 30071 5507/systemd-journa
unix 3 [ ] STREAM CONNECTED 69441 22373/packagekitd
unix 3 [ ] STREAM CONNECTED 3354813 1/systemd
unix 3 [ ] STREAM CONNECTED 66206 10856/dbus-daemon /run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 52224 11475/master
unix 3 [ ] STREAM CONNECTED 3355705 10856/dbus-daemon /run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 52500 10856/dbus-daemon /run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 59441 11475/master
unix 2 [ ] DGRAM 4155954 18112/nrpe
unix 2 [ ] DGRAM 56425 10875/abrtd
unix 3 [ ] STREAM CONNECTED 48044 10879/abrt-watch-lo
unix 3 [ ] STREAM CONNECTED 47981 10876/abrt-watch-lo
unix 3 [ ] STREAM CONNECTED 59399 11475/master
unix 3 [ ] STREAM CONNECTED 52221 11475/master
unix 3 [ ] STREAM CONNECTED 56047 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 52430 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 47551 10816/auditd
unix 3 [ ] STREAM CONNECTED 59417 11475/master
unix 3 [ ] STREAM CONNECTED 59419 11475/master
unix 3 [ ] STREAM CONNECTED 51299 10818/audispd
unix 3 [ ] STREAM CONNECTED 59407 11475/master
unix 3 [ ] STREAM CONNECTED 24294 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 3353562 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 55666 11185/sshd
unix 2 [ ] DGRAM 48125 10882/avahi-daemon:
unix 3 [ ] STREAM CONNECTED 59414 11475/master
unix 2 [ ] DGRAM 70870 22373/packagekitd
unix 3 [ ] STREAM CONNECTED 59401 11475/master
unix 2 [ ] DGRAM 47553 10816/auditd
unix 3 [ ] STREAM CONNECTED 59411 11475/master
unix 3 [ ] STREAM CONNECTED 3355103 51006/crond
unix 3 [ ] STREAM CONNECTED 51339 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 59410 11475/master
unix 3 [ ] STREAM CONNECTED 59404 11475/master
unix 3 [ ] STREAM CONNECTED 59432 11475/master
unix 3 [ ] STREAM CONNECTED 4277676 47754/sshd: rferebe
unix 3 [ ] STREAM CONNECTED 59435 11475/master
unix 3 [ ] STREAM CONNECTED 52222 11475/master
unix 3 [ ] STREAM CONNECTED 59405 11475/master
unix 3 [ ] STREAM CONNECTED 50906 10856/dbus-daemon
unix 3 [ ] STREAM CONNECTED 4277684 47778/abrt-dbus
unix 3 [ ] STREAM CONNECTED 51337 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 59437 11475/master
unix 2 [ ] DGRAM 51300 10818/audispd
unix 3 [ ] STREAM CONNECTED 55305 10856/dbus-daemon /run/dbus/system_bus_socket
unix 2 [ ] STREAM CONNECTED 3356316 51515/java
unix 2 [ ] DGRAM 50881 10849/systemd-login
unix 3 [ ] STREAM CONNECTED 4276969 10856/dbus-daemon /run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 47554 10816/auditd
unix 3 [ ] STREAM CONNECTED 55298 10923/avahi-daemon:
unix 3 [ ] STREAM CONNECTED 59420 11475/master
unix 2 [ ] DGRAM 57362 11475/master
unix 3 [ ] STREAM CONNECTED 59408 11475/master
unix 3 [ ] DGRAM 29015 5544/systemd-udevd
unix 3 [ ] STREAM CONNECTED 28941 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 50905 10856/dbus-daemon
unix 3 [ ] STREAM CONNECTED 59434 11475/master
unix 3 [ ] STREAM CONNECTED 55429 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 52428 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 31830 5539/lvmetad
unix 3 [ ] STREAM CONNECTED 58394 10856/dbus-daemon /run/dbus/system_bus_socket
unix 2 [ ] DGRAM 4270338 45468/xinetd
unix 3 [ ] STREAM CONNECTED 47552 10816/auditd
unix 3 [ ] STREAM CONNECTED 59428 11475/master
unix 3 [ ] STREAM CONNECTED 47644 10852/lsmd
unix 3 [ ] STREAM CONNECTED 59431 11475/master
unix 2 [ ] DGRAM 4277704 47813/sudo
unix 2 [ ] DGRAM 55028 11244/rpc.statd
unix 3 [ ] STREAM CONNECTED 53454 10925/bash
unix 3 [ ] STREAM CONNECTED 54768 11181/python2
unix 3 [ ] STREAM CONNECTED 52434 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 59416 11475/master
unix 3 [ ] STREAM CONNECTED 55235 11294/libvirtd
unix 3 [ ] STREAM CONNECTED 55302 10882/avahi-daemon:
unix 3 [ ] STREAM CONNECTED 52451 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 59413 11475/master
unix 2 [ ] DGRAM 3353902 51387/ntpd
unix 3 [ ] STREAM CONNECTED 53493 1/systemd /run/systemd/journal/stdout
unix 2 [ ] DGRAM 55354 10885/polkitd
unix 3 [ ] STREAM CONNECTED 50908 10856/dbus-daemon /run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 47710 10849/systemd-login
unix 3 [ ] STREAM CONNECTED 28997 5544/systemd-udevd
unix 3 [ ] STREAM CONNECTED 57000 11181/python2
unix 3 [ ] STREAM CONNECTED 47555 10816/auditd
unix 3 [ ] STREAM CONNECTED 59438 11475/master
unix 2 [ ] DGRAM 51431 10874/VGAuthService
unix 3 [ ] STREAM CONNECTED 59425 11475/master
unix 3 [ ] DGRAM 29016 5544/systemd-udevd
unix 3 [ ] STREAM CONNECTED 59402 11475/master
unix 3 [ ] STREAM CONNECTED 4277677 47749/sshd: rferebe
unix 3 [ ] STREAM CONNECTED 47703 10856/dbus-daemon
unix 3 [ ] STREAM CONNECTED 51298 10820/sedispatch
unix 3 [ ] STREAM CONNECTED 51735 10856/dbus-daemon /run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 46860 10882/avahi-daemon:
unix 3 [ ] STREAM CONNECTED 54543 11176/cupsd
unix 2 [ ] DGRAM 4277720 47814/su
unix 3 [ ] STREAM CONNECTED 55586 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 46983 10886/vmtoolsd
unix 3 [ ] STREAM CONNECTED 47645 10852/lsmd
unix 3 [ ] STREAM CONNECTED 52363 1/systemd /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 59422 11475/master
unix 2 [ ] DGRAM 50910 10856/dbus-daemon
unix 3 [ ] STREAM CONNECTED 59426 11475/master
unix 3 [ ] STREAM CONNECTED 59429 11475/master
unix 3 [ ] STREAM CONNECTED 56647 11176/cupsd
unix 3 [ ] STREAM CONNECTED 59423 11475/master
unix 3 [ ] STREAM CONNECTED 55297 10882/avahi-daemon:
unix 3 [ ] STREAM CONNECTED 51343 1/systemd /run/systemd/journal/stdout
unix 2 [ ] DGRAM 4279620 47749/sshd: rferebe

[npolovenko]

@rferebee, I'm not seeing the "check_services" command you're trying to use defined in the nrpe.cfg file.
It could be in this folder on the Log Server:

/usr/local/nagios/etc/nrpe

But can you double check?

If you do not have this command defined, add the following line at the bottom of the npre.cfg file:

command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$

And restart the nrpe:

service nrpe restart

Also, can you open a terminal session to the log server, run these commands and show me the output?

/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1
/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1 -c check_services -t 30 -a 'elasticsearch'
/usr/local/nagios/libexec/check_nrpe -2 -H 127.0.0.1
/usr/local/nagios/libexec/check_nrpe -2 -H 127.0.0.1 -c check_services -t 30 -a 'elasticsearch'

[rferebee]

It appears that check_services is defined in the common.cfg file:

root@nagioslscc4:/usr/local/nagios/etc/nrpe>cat common.cfg

### GENERIC SERVICES ###
command[check_init_service]=sudo /usr/local/nagios/libexec/check_init_service $ARG1$
command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$

### MISC SYSTEM METRICS ###
#command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_users]=/usr/local/nagios/libexec/check_users $ARG1$
command[check_load]=/usr/local/nagios/libexec/check_load $ARG1$
command[check_swap]=/usr/local/nagios/libexec/check_swap $ARG1$
command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats.sh $ARG1$
command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$

### SYSTEM UPDATES ###
command[check_yum]=/usr/local/nagios/libexec/check_yum
command[check_apt]=/usr/local/nagios/libexec/check_apt

### DISK ###
command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
command[check_ide_smart]=/usr/local/nagios/libexec/check_ide_smart $ARG1$

### PROCESSES ###
command[check_all_procs]=/usr/local/nagios/libexec/custom_check_procs
command[check_procs]=/usr/local/nagios/libexec/check_procs $ARG1$

### OPEN FILES ###
command[check_open_files]=/usr/local/nagios/libexec/check_open_files.pl $ARG1$

### NETWORK CONNECTIONS ###
command[check_netstat]=/usr/local/nagios/libexec/check_netstat.pl -p $ARG1$ $ARG2$

Here are the outputs you requested:

root@nagioslscc4:/usr/local/nagios/etc/nrpe>/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1
NRPE v3.2.1
root@nagioslscc4:/usr/local/nagios/etc/nrpe>/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1 -c check_services -t 30 -a 'elasticsearch'
NRPE: Command 'check_services' not defined
root@nagioslscc4:/usr/local/nagios/etc/nrpe>/usr/local/nagios/libexec/check_nrpe -2 -H 127.0.0.1
NRPE v3.2.1
root@nagioslscc4:/usr/local/nagios/etc/nrpe>/usr/local/nagios/libexec/check_nrpe -2 -H 127.0.0.1 -c check_services -t 30 -a 'elasticsearch'
CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected).
root@nagioslscc4:/usr/local/nagios/etc/nrpe>

[npolovenko]

@rferebee, This is weird since you do have the command defined:

root@nagioslscc4:/usr/local/nagios/etc/nrpe>/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1 -c check_services -t 30 -a 'elasticsearch'
NRPE: Command 'check_services' not defined

But then I looked at the list of processes and it appears that your nrpe is using a different configuration file --> /etc/nagios/nrpe.cfg

nrpe 18112 1 0 08:23 ? 00:00:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f

On the log server open this nrpe config: /etc/nagios/nrpe.cfg and make sure this command is defined:

command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$

If its missing you can add it and then restart the nrpe again:

service nrpe restart

And then try this again:

root@nagioslscc4:/usr/local/nagios/etc/nrpe>/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1 -c check_services -t 30 -a 'elasticsearch'

[rferebee]

Ok, I defined the command in nrpe.cfg. I restarted the NRPE service.

Now when I run the command locally it returns nothing. Just moves to the next line:

root@nagioslscc4:/etc/nagios>/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1 -c check_services -t 30 -a 'elasticsearch'
root@nagioslscc4:/etc/nagios>

When I run the check from within Nagios I get this:

CHECK_NRPE: No output returned from daemon.

[rferebee]

This is how I have the check configured in Nagios, should it be another way?

[npolovenko]

@rferebee, Can you run this command from the log server command line and show me the output:

su - nagios

/usr/local/nagios/libexec/check_services -p elasticsearch

Also, show me the output of:

ll /etc/xinetd.d/

And upload the whole config file:

/etc/nagios/nrpe.cfg

[rferebee]

root@nagioslscc4:/etc/nagios>su - nagios
[nagios@nagioslscc4 ~]$ /usr/local/nagios/libexec/check_services -p elasticsearch
elasticsearch: 1
[nagios@nagioslscc4 ~]$ ll /etc/xinetd.d/
total 48
-rw------- 1 root root 1157 Nov 5 2016 chargen-dgram
-rw------- 1 root root 1159 Nov 5 2016 chargen-stream
-rw------- 1 root root 1157 Nov 5 2016 daytime-dgram
-rw------- 1 root root 1159 Nov 5 2016 daytime-stream
-rw------- 1 root root 1157 Nov 5 2016 discard-dgram
-rw------- 1 root root 1159 Nov 5 2016 discard-stream
-rw------- 1 root root 1148 Nov 5 2016 echo-dgram
-rw------- 1 root root 1150 Nov 5 2016 echo-stream
-rw-r--r-- 1 root root 470 Apr 25 07:52 nrpe
-rw------- 1 root root 1212 Nov 5 2016 tcpmux-server
-rw------- 1 root root 1149 Nov 5 2016 time-dgram
-rw------- 1 root root 1150 Nov 5 2016 time-stream
[nagios@nagioslscc4 ~]$

Attached is the config file.