Page 1 of 1
CHECK_NRPE: Socket timeout after 10 seconds
Posted: Wed Jul 26, 2017 7:33 am
by bsivavani
Hi,
We have built one custom script for C-Drive cleanup and we are calling the script using Event Handler, it is working on few servers but in few servers we are seeing the communication is getting lost as shown below.
[nagios@s930l3092 ~]$ /usr/local/nagios/libexec/check_nrpe -H s930a3657.group.rwe.com
CHECK_NRPE: Socket timeout after 10 seconds.
[nagios@s930l3092 ~]$
custom script placed under C:\Program Files\NSClient++\scripts, before placing the script it is working fine.
Telnet output
[nagios@s930l3092 ~]$ telnet s930a3657.group.rwe.com 12489
Trying 10.87.217.208...
telnet: connect to address 10.87.217.208: Connection refused
[nagios@s930l3092~]$
Please find attached nsclient.ini file and log file.
Kindly let us know possible ways for the connection lost issue between Nagios and Remote server.
Re: CHECK_NRPE: Socket timeout after 10 seconds
Posted: Wed Jul 26, 2017 10:19 am
by cdienger
The command should look something like:
Code: Select all
/usr/local/nagios/libexec/check_nrpe -H s930a3657.group.rwe.com -c check_cdrivecleanup
You can increase the default timeout of 10 seconds with the "-t" option:
Code: Select all
/usr/local/nagios/libexec/check_nrpe -H s930a3657.group.rwe.com -t 30 -c check_cdrivecleanup
The nsclient.ini shows s930l3092.rwe.com as the only host able to connect:
Try adding the IP address of the XI server to the list and restart the nsclient service:
Code: Select all
allowed hosts = s930l3092.rwe.com,XI_IP_ADDR
The nsclient.ini currently has the cleanup scripts commented out:
Code: Select all
[/settings/external scripts/scripts]
...
; check_cdrivecleanup=cmd /c echo scripts\\Disk_MaintenanceV1.3.ps1 ; exit($lastexitcode) | powershell.exe –ExecutionPolicy Bypass -command -
; check_cdrivecleanup=cmd /c echo scripts\\Disk_MaintenanceV1.3.ps1 ; exit($lastexitcode) | powershell.exe -command - >> DiskMain_Nagios.log
Make sure to remove the semi colon from one of the lines and restarting the nsclient before testing again:
Code: Select all
[/settings/external scripts/scripts]
...
check_cdrivecleanup=cmd /c echo scripts\\Disk_MaintenanceV1.3.ps1 ; exit($lastexitcode) | powershell.exe –ExecutionPolicy Bypass -command -
; check_cdrivecleanup=cmd /c echo scripts\\Disk_MaintenanceV1.3.ps1 ; exit($lastexitcode) | powershell.exe -command - >> DiskMain_Nagios.log
Re: CHECK_NRPE: Socket timeout after 10 seconds
Posted: Wed Jul 26, 2017 11:32 am
by bsivavani
The command should look something like:
Code: Select all/usr/local/nagios/libexec/check_nrpe -H s930a3657.group.rwe.com -c check_cdrivecleanup
Yes. I am aware of this, I am using command /usr/local/nagios/libexec/check_nrpe -H s930a3657.group.rwe.com to check communication.
You can increase the default timeout of 10 seconds with the "-t" option:
Code: Select all/usr/local/nagios/libexec/check_nrpe -H s930a3657.group.rwe.com -t 30 -c check_cdrivecleanup
Yes. I tried by increasing timeout.
The nsclient.ini shows s930l3092.rwe.com as the only host able to connect:
Yes. It is only one host to connect.
The nsclient.ini currently has the cleanup scripts commented out:
Code: Select all[/settings/external scripts/scripts]
...
; check_cdrivecleanup=cmd /c echo scripts\\Disk_MaintenanceV1.3.ps1 ; exit($lastexitcode) | powershell.exe –ExecutionPolicy Bypass -command -
; check_cdrivecleanup=cmd /c echo scripts\\Disk_MaintenanceV1.3.ps1 ; exit($lastexitcode) | powershell.exe -command - >> DiskMain_Nagios.log
Yes. To resolve communication issue we have commented out the line which we added but still no luck and telnet on port 12489 itself not working.
Make sure to remove the semi colon from one of the lines and restarting the nsclient before testing again:
Yes. I tried by removing semi colon, but still not working.
Re: CHECK_NRPE: Socket timeout after 10 seconds
Posted: Wed Jul 26, 2017 12:00 pm
by cdienger
I completely missed the telnet test! Sorry about that!
Make sure that port 12389 is up and listening on the Windows' server with:
If this is up and listening then there's likely a firewall in between the XI server and Windows' machine. If it's not up and listening then there's likely a problem with the config or something on the machine preventing the port from being opened.
Re: CHECK_NRPE: Socket timeout after 10 seconds
Posted: Wed Jul 26, 2017 12:31 pm
by bsivavani
Hi,
Windows firewall is OFF in remote server.
Please find below netstat output.
C:\Users\ui413220>netstat -na | find "12489"
TCP 0.0.0.0:12489 0.0.0.0:0 LISTENING
TCP 0.0.0.0:12489 0.0.0.0:0 LISTENING
TCP 0.0.0.0:12489 0.0.0.0:0 LISTENING
TCP 0.0.0.0:12489 0.0.0.0:0 LISTENING
TCP 10.87.217.208:12489 10.87.212.167:20184 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:20229 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:20254 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:20386 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:20435 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:20463 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:20995 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:21030 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:21138 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:21326 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:21577 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:21752 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:21974 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:21976 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:22091 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:22103 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:22131 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:22367 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:22505 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:22748 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:22924 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:22962 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:23371 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:23664 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:23671 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:23726 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:23925 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:24348 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:24536 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:24789 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:24875 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:25219 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:25407 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:25459 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:25562 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:25658 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:26072 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:26326 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:26477 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:27325 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:27510 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:27956 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:28127 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:28383 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:28810 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:29024 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:29165 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:29196 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:29598 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:29855 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:29892 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:30018 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:30461 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:30833 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:31032 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:31037 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:31451 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:31650 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:31848 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:32492 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:32653 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:32660 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:33067 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:33153 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:33324 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:33481 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:33768 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:34297 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:34364 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:34485 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:34929 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:35152 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:35415 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:35835 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:36084 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:36212 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:36264 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:36665 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:36933 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:37101 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:37901 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:38077 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:38487 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:38688 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:38770 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:38946 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:39358 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:39365 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:39607 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:39756 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:39759 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:39947 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:40146 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:40555 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:41383 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:41561 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:41964 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:42024 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:42192 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:42413 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:42622 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:42837 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:43080 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:43184 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:43207 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:43290 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:43711 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:43984 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:44103 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:44979 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:45122 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:45549 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:45784 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:46024 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:46401 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:46787 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:46836 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:47259 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:47516 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:47629 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:47672 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:48239 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:48489 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:48658 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:48834 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:49087 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:49309 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:49484 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:49862 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:50117 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:50289 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:50311 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:50719 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:50942 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:50993 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:51079 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:51487 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:52012 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:52089 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:52134 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:52578 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:52814 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:53025 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:53427 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:53700 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:53858 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:53917 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:54328 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:54596 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:54692 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:55545 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:55719 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:56153 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:56358 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:56529 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:56558 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:56969 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:57147 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:57235 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:57387 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:57425 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:57672 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:57810 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:58072 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:58182 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:59046 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:59202 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:59609 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:59806 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:59836 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:60016 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:60417 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:60428 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:60693 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:60841 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:60945 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:60976 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:61372 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:61666 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:61735 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:62494 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:62637 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:62778 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:63207 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:63448 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:63460 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:63537 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:63623 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:63629 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:63713 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:63821 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:64037 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:64298 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:64456 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:64474 CLOSE_WAIT
TCP 10.87.217.208:12489 10.87.212.167:64903 CLOSE_WAIT
TCP [::]:12489 [::]:0 LISTENING
TCP [::]:12489 [::]:0 LISTENING
C:\Users\ui413220>
Re: CHECK_NRPE: Socket timeout after 10 seconds
Posted: Wed Jul 26, 2017 3:18 pm
by cdienger
That's a lot of connections the nsclient is keeping in close_wait. Do you get the timeout errors right away after restarting the nsclient service? Do any of the other scripts have the same timeout issue? What version of nsclient is this?
I'd be curious to see a tcpdump while running the command:
Code: Select all
yum -y install tcpdump
tcpdump -s 0 -i any host 10.87.217.208 and port 12489 -w nsclient.pcap
let this run just long enough to reproduce the timeout error and use ctrl+c to stop it. The file can be reviewed with wireshark. Feel free to attach it to the thread or PM it to me.
Re: CHECK_NRPE: Socket timeout after 10 seconds
Posted: Thu Jul 27, 2017 8:57 am
by bsivavani
Hi,
We have used below tcpdump command to check traffic
tcpdump -s 0 -i eth0 src 10.87.217.208 and port 12489 -w nsclient.pcap
I sent output through PM.
And also Kindly let us know if there is any possibility to clear CLOSE_WAIT connections, so that reboot of server is not required.
Re: CHECK_NRPE: Socket timeout after 10 seconds
Posted: Thu Jul 27, 2017 9:20 am
by cdienger
The connection is being held open by nsclient so restarting it should clear them up.
Re: CHECK_NRPE: Socket timeout after 10 seconds
Posted: Thu Jul 27, 2017 11:01 am
by bsivavani
Restarting NSClient service didn't clear CLOSE_WAIT connections.
Re: CHECK_NRPE: Socket timeout after 10 seconds
Posted: Thu Jul 27, 2017 2:48 pm
by tgriep
The check_nrpe plugin uses port 5666 for connecting and not port 12489, make sure the server is listening on that port as well.