I kicked off the script on the client side to each server in turn. Here is the httpd access_log on the server that is not working during that time.
::1 - - [22/Dec/2015:15:33:22 +0000] "POST /nagiosxi/backend/ HTTP/1.1" 200 809 "-" "BinGet/1.00.A (
http://www.bin-co.com/php/scripts/load/)"
173.1.1.123 - - [22/Dec/2015:15:33:34 +0000] "POST /nrdp/ HTTP/1.1" 200 170 "-" "libwww-perl/6.13"
173.1.1.123 - - [22/Dec/2015:15:33:34 +0000] "POST /nrdp/ HTTP/1.1" 200 170 "-" "libwww-perl/6.13"
173.1.1.123 - - [22/Dec/2015:15:33:34 +0000] "POST /nrdp/ HTTP/1.1" 200 170 "-" "libwww-perl/6.13"
173.1.1.123 - - [22/Dec/2015:15:33:34 +0000] "POST /nrdp/ HTTP/1.1" 200 170 "-" "libwww-perl/6.13"
173.1.1.123 - - [22/Dec/2015:15:33:34 +0000] "POST /nrdp/ HTTP/1.1" 200 170 "-" "libwww-perl/6.13"
I also noticed on the non-working server there thousands of files in the /usr/local/nagios/var/spool/checkresults directory. They disappear on the working server.
./nagios/var/spool/checkresults/coKR17N.ok
./nagios/var/spool/checkresults/cjDqNoA.ok
./nagios/var/spool/checkresults/cazFUGj.ok
./nagios/var/spool/checkresults/cTFV01J.ok
./nagios/var/spool/checkresults/cesmxto.ok
./nagios/var/spool/checkresults/cA3pxbi.ok
On the working server I see this:
::1 - - [22/Dec/2015:15:38:22 +0000] "POST /nagiosxi/backend/ HTTP/1.1" 200 810 "-" "BinGet/1.00.A (
http://www.bin-co.com/php/scripts/load/)"
173.1.1.123 - - [22/Dec/2015:15:38:34 +0000] "POST /nrdp/ HTTP/1.1" 200 170 "-" "libwww-perl/6.13"
173.1.1.123 - - [22/Dec/2015:15:38:34 +0000] "POST /nrdp/pmstat_r.php/ HTTP/1.1" 200 156 "-" "curl/7.35.0"
173.1.1.123 - - [22/Dec/2015:15:38:34 +0000] "POST /nrdp/ HTTP/1.1" 200 170 "-" "libwww-perl/6.13"
173.1.1.123 - - [22/Dec/2015:15:38:34 +0000] "POST /nrdp/ HTTP/1.1" 200 170 "-" "libwww-perl/6.13"
173.1.1.123 - - [22/Dec/2015:15:38:34 +0000] "POST /nrdp/ HTTP/1.1" 200 170 "-" "libwww-perl/6.13"
173.1.1.123 - - [22/Dec/2015:15:38:34 +0000] "POST /nrdp/ HTTP/1.1" 200 170 "-" "libwww-perl/6.13"
Are there other logs that might shed more light?