Service check timed out after 60.01 seconds

[Prabha2028]

Hi ,

When i am running the check_jmx from nagios server to other remote servers(windows servers), i am getting the below error.

Can someone help me on this?

[root@brxpdlibmonl01 Prod]# /usr/local/nagios/plugins/check_jmx_wrapper -U service:jmx:rmi:///jndi/rmi://172.18.1.91:19080/jmxrmi -O Catalina:type=ThreadPool,name=http-9080 -A currentThreadCount -vvvv -w 175 -c 250
JMX CRITICAL Connection refused to host: 172.18.1.91; nested exception is:
java.net.ConnectException: Connection timed out connecting to Catalina:type=ThreadPool,name=http-9080 by URL service:jmx:rmi:///jndi/rmi://172.18.1.91:19080/jmxrmijava.rmi.ConnectException: Connection refused to host: 172.18.1.91; nested exception is:
java.net.ConnectException: Connection timed out
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:129)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:227)
at org.nagios.JMXQuery.connect(JMXQuery.java:53)
at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

Ports are connected successfully:
[root@brxpdlibmonl01 Prod]# telnet 172.18.1.91 9080
Trying 172.18.1.91...
Connected to 172.18.1.91.
Escape character is '^]'.
^C^CConnection closed by foreign host.
[root@brxpdlibmonl01 Prod]# telnet 172.18.1.91 19080
Trying 172.18.1.91...
Connected to 172.18.1.91.
Escape character is '^]'.



Thanks,
Prabhas

[pbroste]

Hello Prabhas,

Let's check to see if the ports are listening:

Code: Select all

sudo ss -tulwn | grep '9080|19080'

Also here is a line to test the check_jmx plugin:

Code: Select all

./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192

Regards,
Perry

[Prabha2028]

Hi Perry,

Thanks for your support.

I have excuted given commands and pasted the results below.

[root@brxpdlibmonl01 nagios]# sudo ss -tulwn | grep '9080|19080'
sudo: PAM account management error: Permission denied
[root@brxpdlibmonl01 nagios]# ./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:8080/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192
-bash: ./check_jmx: No such file or directory
[root@brxpdlibmonl01 nagios]# ./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192
-bash: ./check_jmx: No such file or directory
[root@brxpdlibmonl01 nagios]#



Thanks,
Prabhas

[mcapra]

Depending on the firewall in-use on the 172.18.1.91 machine, telnet may not be enough to test that the port is truly open and that the check_jmx traffic is not being filtered out by the firewall. Might try nmap or netcat instead. Both are great tools to have on your Nagios server for troubleshooting.

Code: Select all

yum install nmap nc
netcat -z -n -v 172.18.1.91 19080
nmap -sS -O -p19080 172.18.1.91


It would also be useful to see specifically how you're configuring the JMX endpoint within your Tomcat instance. Specifically, the values of com.sun.management.jmxremote.local.only and java.rmi.server.hostname. But there are other properties that could cause connection failures.