Child nrpe process will block When via bad VPN in CentOS7.2

[[email protected]]

Hello.

Child nrpe process will block When via bad VPN connection.

I use nrpe-2.15 in CentOS7.2.
nrpe-2.15 was installed in epel.

please help me.
--

Code: Select all

[root@myserver ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@myserver ~]# uname -a
Linux myserver 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@myserver ~]# rpm -q nrpe
nrpe-2.15-7.el7.x86_64
[root@myserver ~]# rpm -q glibc
glibc-2.17-106.el7_2.4.x86_64
[root@myserver ~]# rpm -q openssl
openssl-1.0.1e-51.el7_2.4.x86_64

[root@myserver ~]# ps ax | grep nrpe
 1685 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
 2045 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
 6374 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
 8565 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
10253 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
12036 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
12172 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
13022 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
14443 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
17689 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
18630 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
21593 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
21652 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
21958 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
22271 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
23013 pts/0    S+     0:00 grep --color=auto nrpe
24561 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
26119 ?        Ss     0:01 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
26541 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
26997 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
27837 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
28369 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
31618 ?        S      0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

[root@myserver ~]# journalctl -x -u nrpe

log when normal:

Code: Select all

Apr 13 23:05:27 myserver nrpe[19464]: Connection from 192.168.179.201 port 61879
Apr 13 23:05:27 myserver nrpe[19464]: Host address is in allowed_hosts
Apr 13 23:05:27 myserver nrpe[19464]: Handling the connection...
Apr 13 23:05:27 myserver nrpe[19464]: Host is asking for command 'check_cpu' to be run...
Apr 13 23:05:27 myserver nrpe[19464]: Running command: /usr/lib64/nagios/plugins/check_cpu.pl -w 5 -c 1 30
Apr 13 23:05:28 myserver nrpe[19464]: Command completed with return code 0 and output: CPU usage OK - 0% used 100% idle
Apr 13 23:05:29 myserver nrpe[19464]: Return Code: 0, Output: CPU usage OK - 0% used 100% idle
Apr 13 23:05:29 myserver nrpe[19464]: [30B blob data]

log when a block occurred:

Code: Select all

Apr 13 23:15:28 myserver nrpe[22271]: Connection from 192.168.179.201 port 65209
Apr 13 23:15:28 myserver nrpe[22271]: Host address is in allowed_hosts
Apr 13 23:15:28 myserver nrpe[22271]: Handling the connection...
Apr 13 23:15:28 myserver nrpe[22271]: Host is asking for command 'check_cpu' to be run...
Apr 13 23:15:28 myserver nrpe[22271]: Running command: /usr/lib64/nagios/plugins/check_cpu.pl -w 5 -c 1 30
Apr 13 23:15:29 myserver nrpe[22271]: Command completed with return code 0 and output: CPU usage OK - 1% used 99% idle

gdb backtrace:

Code: Select all

[root@myserver ~]# gdb -p 22271
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
...snip...
0x00007fa37a9aba40 in __read_nocancel () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install nrpe-2.15-7.el7.x86_64
(gdb) bt
#0  0x00007fa37a9aba40 in __read_nocancel () from /lib64/libc.so.6
#1  0x00007fa37b18496b in sock_read () from /lib64/libcrypto.so.10
#2  0x00007fa37b18294b in BIO_read () from /lib64/libcrypto.so.10
#3  0x00007fa37b4bc414 in ssl3_read_n () from /lib64/libssl.so.10
#4  0x00007fa37b4bd5c5 in ssl3_read_bytes () from /lib64/libssl.so.10
#5  0x00007fa37b4ba778 in ssl3_shutdown () from /lib64/libssl.so.10
#6  0x0000000000404de6 in complete_SSL_shutdown ()
#7  0x00000000004054a9 in handle_connection ()
#8  0x0000000000405d75 in wait_for_connections ()
#9  0x0000000000403107 in main ()

[tgriep]

In the /etc/nagios/nrpe.cfg file is the following option you may want to change for when the NRPE agent loses connection to the monitoring server.

Code: Select all

connection_timeout=300

Try adjusting the timeout settings and see if that helps on the issue.

[[email protected]]

Blocking keeps more than 300 seconds.
It continues for more than several days.

/etc/nagios/nrpe.cfg (without comment):

Code: Select all

log_facility=daemon
pid_file=/var/run/nrpe/nrpe.pid
server_port=5666
nrpe_user=nrpe
nrpe_group=nrpe
allowed_hosts=127.0.0.1,[mynetwork addreses]
dont_blame_nrpe=1
allow_bash_command_substitution=0
debug=1
command_timeout=60
connection_timeout=300
command[check_users]=/usr/lib64/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib64/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib64/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib64/nagios/plugins/check_procs -w 150 -c 200
include_dir=/etc/nrpe.d/

[tgriep]

There have been some fixes to the NRPE Agent in the 2.16 RC2 that may fix the issue you are having.
The link below is where you can download and compile it for your system
https://github.com/NagiosEnterprises/nr ... e-2-16-RC2
Other then that, you can run the NRPE agent out of xinetd and have that timeout when the connection is lost.

[[email protected]]

Thank you very much for advice.

This article seems relevant.
https://github.com/NagiosEnterprises/nr ... 8c3d47bfb2

I don't have authority to compile/install to my server(layer8/9 ), so I consider to switch nrpe start from systemd to xinetd.
This may not be a good way.

/etc/xinetd.d/nrpe :

Code: Select all

...snip...
       server          = /usr/bin/timeout
       server_args     = 10s /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -i
...snip...

[tgriep]

That is too bad that you cannot compile the updated NRPE agent on your servers, maybe you can ask someone with the access to do it for you.