check_jmx with certificate

[rkennedy]

Great to see some verbosity!

Looking around online suggests that JMX uses a DIFFERENT port after the connection, so this could be why. Once it decides to use a different port, I believe it's being blocked at a firewall level now. Can you try appending two more things to your command?

Code: Select all

-Dcom.sun.management.jmxremote.port=8084 
-Dcom.sun.management.jmxremote.rmi.port=8084 

This should restrict it to 8084.

[haile711]

rkennedy

I think i've a mistake on the ip but here is the infos we got after i've added these 2 new options

Code: Select all

java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.48.132:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -Dcom.sun.management.jmxremote.port=8084 -Dcom.sun.management.jmxremote.rmi.port=8084 -vvvv
JMX CRITICAL Connection refused to host: 10.72.48.132; nested exception is:
        java.net.ConnectException: Connection timed out connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://10.72.48.132:8084/jmxrmijava.rmi.ConnectException: Connection refused to host: 10.72.48.132; nested exception is:
        java.net.ConnectException: Connection timed out
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:129)
        at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
        at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:227)
        at org.nagios.JMXQuery.connect(JMXQuery.java:53)
        at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:637)
        at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:425)
        at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
        at javax.rmi.ssl.SslRMIClientSocketFactory.createSocket(SslRMIClientSocketFactory.java:120)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
        ... 10 more

[rkennedy]

Looking back, you've posted multiple different IP's while we try to work through this. This could bring in additional issues. Can you please run it against the one you ran an nmap against? 10.72.48.129

Also, when you run it, try this -

Code: Select all

java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -Dcom.sun.management.jmxremote.port=8084 -Dcom.sun.management.jmxremote.rmi.port=8084 -U service:jmx:rmi:///jndi/rmi://10.72.48.129:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv

[haile711]

all these ip belong to the same cluster, I have done a telnet test to all these server on port 8084, and i got connectivity.

This is the output for 10.72.48.129

Code: Select all

java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -Dcom.sun.management.jmxremote.port=8084 -Dcom.sun.management.jmxremote.rmi.port=8084 -U service:jmx:rmi:///jndi/rmi://10.72.48.129:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv
JMX CRITICAL Connection refused to host: 10.72.48.129; nested exception is:
        java.net.ConnectException: Connection timed out connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://10.72.48.129:8084/jmxrmijava.rmi.ConnectException: Connection refused to host: 10.72.48.129; nested exception is:
        java.net.ConnectException: Connection timed out
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:129)
        at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
        at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:227)
        at org.nagios.JMXQuery.connect(JMXQuery.java:53)
        at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:637)
        at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:425)
        at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
        at javax.rmi.ssl.SslRMIClientSocketFactory.createSocket(SslRMIClientSocketFactory.java:120)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)

[rkennedy]

all these ip belong to the same cluster, I have done a telnet test to all these server on port 8084, and i got connectivity.

This is the output for 10.72.48.129

Code: Select all

java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -Dcom.sun.management.jmxremote.port=8084 -Dcom.sun.management.jmxremote.rmi.port=8084 -U service:jmx:rmi:///jndi/rmi://10.72.48.129:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv
JMX CRITICAL Connection refused to host: 10.72.48.129; nested exception is:
        java.net.ConnectException: Connection timed out connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://10.72.48.129:8084/jmxrmijava.rmi.ConnectException: Connection refused to host: 10.72.48.129; nested exception is:
        java.net.ConnectException: Connection timed out
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:129)
        at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
        at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:227)
        at org.nagios.JMXQuery.connect(JMXQuery.java:53)
        at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:637)
        at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:425)
        at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
        at javax.rmi.ssl.SslRMIClientSocketFactory.createSocket(SslRMIClientSocketFactory.java:120)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)

Got it, just wanted to make sure. I just wanted to make sure since we had only seen one nmap.

Code: Select all

java -cp jmxquery.jar org.nagios.JMXQuery -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 -Dcom.sun.management.jmxremote.port=8084 -Dcom.sun.management.jmxremote.rmi.port=8084 -U service:jmx:rmi:///jndi/rmi://10.72.48.129:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv

Not sure if the positioning as JMXQuery helps out at all, but could you please give it a try? I just went back to the check_jmx script, and that's the only difference I can see since it's starting like this -

Code: Select all

java -cp $RDIR/jmxquery.jar org.nagios.JMXQuery $@

For reference, what is the java application you're checking against? I can try to set something up on my end to try and replicate.

[haile711]

rkennedy,
We trying to monitor our 3rd vendor party application, called "Mashery Local" by tibco. I'm not too familiar with this application myself. We were asked to be part of the monitoring solution for them.

[rkennedy]

I looked around online, and couldn't find much about them. I did read this in one of their 'flyers'. -

Code: Select all

Mashery runs a complex 
collection of watchdogs built from third-party tools (such as 
Webmetrics and Nagios) and our own instrumentation. The 
watchdogs monitor each system level: component, application, 
zone, region, network, Internet, and our customer data centers. 
Failover and restart of our components is automatic and dynamic.   

I assume this is for their 'cloud' solution though. Would you be able to ask them for more information about how you can check the device? It sounds like their internal tools use Nagios, so perhaps they'd be open to help? To be honest, it might not support java.lang:type=Memory -A HeapMemoryUsage -K used, but with the lack of information it's hard to make a clear judgement.

[InscrutableDude]

I have found a reliable workaround for this issue.

What I realized is that the check_jmx script on the NagiosXI machine would run just as well on the remote machine I was trying to monitor. I copied that and jmxquery.jar to the remote machine.

I had to make a couple modifications. Normally, java is installed on the NagiosXI machine so that the check can run. On my remote machine, each instance I'm monitoring has its own version of java locally, so I just pointed at one of those. If you have java globally you won't need to make this edit. You'll also notice though, the one change you HAVE to make: on the next line after the java call, add "echo $?". That echoes the actual error code back to the script on the NagiosXI server.

Code: Select all

#!/bin/sh
#
# Nagios plugin to monitor Java JMX (http://java.sun.com/jmx)attributes.
#
RDIR=`dirname $0`
/path/to/local/java/bin/java -cp $RDIR/jmxquery.jar org.nagios.JMXQuery $@
echo $?

Then, on the NagiosXI host, I created a script called check_jmx_remote
Code follows, but the explanation is simple: I simply establish an ssh connection to the remote host and execute the check_jmx script over there. The result is returned to me, along with the original exit code.

In order to make this work, you'll need to export your NagiosXI key to the remote server. You'll need to have root access, as you'll need to su nagios from root. Then cd ~/.ssh
ssh-copy-id -i [email protected]
It'll ask you for the password for the user on the remote host. This all allows you to make the ssh connection without having to supply the password, thus allowing it in a script.

You'll note in my code that I use -p in my ssh command as we use a nonstandard port, which i've subbed with 55555, but you can delete that if you're on 22.

Code: Select all

#!/bin/bash
#
# Nagios plugin to monitor Java JMX (http://java.sun.com/jmx)attributes.
#

theCommand="$@"

theHost=$(cut -d'/' -f7 <<<"$theCommand")

theHost="$(cut -d':' -f1 <<<$theHost)"

# echo "$theHost"

sendCommand="$(ssh -p 55555 myuser@$theHost /usr/local/nagios/libexec/check_jmx $theCommand)"

theResponse=$(echo "$sendCommand"|head -1)
#echo "$theResponse"
theExitCode=$(echo "$sendCommand"|tail -n1)
#echo "$theExitCode"

if [ "$theExitCode" == "0" ]; then

    echo "OK - Status: $theResponse"
    exit 0;

elif [ "$theExitCode" == "2" ]; then

    echo -e "CRITICAL - Status: $theResponse"
    exit 2;

else

    echo -e "WARNING - Status: $theResponse"
    exit 1;

fi

I've left some lines commented that you can uncomment if you need to run it locally on the cli to check output.

You''ll need to create a new Command for this, mine is called check_jmx_local, just like the file. You can literally duplicate the check_jmx command and change the name, and the file it's pointing to instead of creating one from scratch.

Then you just set up the services. The one thing that's different about the command in $ARG1$ is that you don't put any single quotes. So instead of this:
-a '-U service:jmx:rmi:///jndi/rmi://$HOSTADDRESS$:9999/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 7516188920 -c 8589930194'
You put this:
-a -U service:jmx:rmi:///jndi/rmi://$HOSTADDRESS$:9999/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 7516188920 -c 8589930194

I hope this helps someone. It's taken me about a week to work this all out.

[scottwilkerson]

@InscrutableDude thanks for sharing this!

[InscrutableDude]

@InscrutableDude thanks for sharing this!

You're quite welcome!