CRITICAL - Socket timeout http URL

[rtsupport]

Hi Team,

we are monitoring multiple http url's but one of them suddenly goes in critical state which was working fine.

- URL is up and running fine (opening in IE and Mozilla only) i am sure that should not be an issue.
- from app team there were no changes done from there end.

referring other nagios post checked below details which seems to be fine, but unable to understand what could be the issue.

Code: Select all

-bash-4.1$ ./check_http -w 5 -c 10 -H 13.129.00.00 -p 9809 -N -u /FileNet/Engine -v
GET /FileNet/Engine HTTP/1.1
User-Agent: check_http/v2.2.1 (nagios-plugins 2.2.1)
Connection: close
Host: 13.129.00.00:9809
Accept: */*


http://13.129.00.00:9809/FileNet/Engine is 12 characters
STATUS: GIOP
CRITICAL - Socket timeout

Code: Select all

./check_http -c 15 -t 15 -f follow -H 13.129.00.00 -p 9809 -s GIOP -u /FileNet/Engine
CRITICAL - Socket timeout

Code: Select all

[-bash-4.1$]# nmap usa03000000.apps.mc.xerox.com

Starting Nmap 5.51 ( http://nmap.org ) at 2019-02-26 07:55 EST
Stats: 0:00:00 elapsed; 0 hosts completed (0 up), 1 undergoing Ping Scan
Ping Scan Timing: About 100.00% done; ETC: 07:55 (0:00:00 remaining)
Nmap scan report for usa03000000.apps.mc.xerox.com (13.129.63.110)
Host is up (0.00083s latency).
Not shown: 988 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
25/tcp    open  smtp
111/tcp   open  rpcbind
587/tcp   open  submission
1556/tcp  open  veritas_pbx
2809/tcp  open  corbaloc
4045/tcp  open  lockd
5666/tcp  open  nrpe
9080/tcp  open  glrpc
9900/tcp  open  iua
13782/tcp open  netbackup
32775/tcp open  sometimes-rpc13

Nmap done: 1 IP address (1 host up) scanned in 27.48 seconds
[root@usa0300lv1561 libexec]# nmap -Pn -p T:9809 usa03000000.apps.mc.xerox.com

Starting Nmap 5.51 ( http://nmap.org ) at 2019-02-26 07:56 EST
Nmap scan report for usa0300000.apps.mc.xerox.com (13.129.63.110)
Host is up (0.00062s latency).
PORT     STATE SERVICE
9809/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 0.24 seconds

Code: Select all

-bash-4.1$ telnet usa03000000.apps.mc.xerox.com 9809
Trying 13.129.63.110...
Connected to usa03000000.apps.mc.xerox.com.

Code: Select all

-bash-4.1$ wget http://13.129.00.00:9809/FileNet/Engine
--2019-02-26 06:16:03--  http://13.129.63.110:9809/FileNet/Engine
Connecting to 13.129.63.110:9809... connected.
HTTP request sent, awaiting response... 200 No headers, assuming HTTP/0.9
Length: unspecified
Saving to: “Engine.1”

    [ <=>                                                                                             ] 12          --.-K/s   in 0s

2019-02-26 06:16:03 (1.13 MB/s) - “Engine.1” saved [12]

[cdienger]

STATUS: GIOP

is interesting. We'd expect to see a HTTP status code here. Run the following to get a tcpdump and PM me the output.pcap it creates:

yum -y install tcpdump
tcpdump -s 0 -i any host 13.129.00.00 -w output.pcap

Run the check_http command again while that is running then use CTRL+C to stop it.

[rtsupport]

please refer attached output on your PM hope this is you are looking for, got the output by running below command..

tcpdump -s 0 -i any host 13.129.00.00 -w output.pcap
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
/usr/local/nagios/libexec/check_http -w 5 -c 10 -H 13.129.00.00 -p 9809 -N -u /FileNet/Engine -v
^C71 packets captured
75 packets received by filter
0 packets dropped by kernel

[cdienger]

The PM didn't come through. Can you send it again?

[rtsupport]

update again, could you please check now..

[cdienger]

It came through that time but didn't contain any of the traffic for the site. Try running it again but capturing based on the port:

tcpdump -s 0 -i any port 9809 -w output.pcap

When you run this you will want to open another terminal to the same machine and then run the check_http from the new terminal.

[rtsupport]

run again as per instruction and have shared new output file to your PM, please check.

hope so this time you get something to tell us.

[root@usa03000000 nagios]# tcpdump -s 0 -i any port 9809 -w output.pcap
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
^C10 packets captured
673 packets received by filter
618 packets dropped by kernel

[cdienger]

The server is replying really oddly with just:

GIOP........

Try this:

./check_http -c 15 -t 15 -f follow -H 13.129.00.00 -p 9809 -e GIOP -u /FileNet/Engine

or:

./check_http -c 15 -t 15 -f follow -H 13.129.00.00 -p 9809 -d GIOP -u /FileNet/Engine

[rtsupport]

same result with both command

-bash-4.1$ ./check_http -c 15 -t 15 -f follow -H 13.129.00.00 -p 9809 -e GIOP -u /FileNet/Engine
CRITICAL - Socket timeout
-bash-4.1$ ./check_http -c 15 -t 15 -f follow -H 13.129.00.00 -p 9809 -d GIOP -u /FileNet/Engine
CRITICAL - Socket timeout

[cdienger]

I don't think check_http will work in this case then since the response isn't a typical http response. You could try using check_tcp instead to verify that the port is up and accepting connections instead.

./check_tcp -H 13.129.00.00 -p 9809