JMX SSL monitoring

[nagiosvick]

Hi,

I was able to successfully establish a connection with the JMX port through SSL connection through jconsole. Trying to achieve the same through Nagios. But I am not able to connect. I have attached the screenshot of the parameters I try to pass.

When I tried with standalone Jconsole to connect I had used this command to start jconsole

C:\certs>jconsole -J-Djavax.net.ssl.keyStore=C:\certs\jconsole.keystore -J-Djavax.net.ssl.keyStorePassword=password -J-Djavax.net.ssl.trustStore=C:\certs\jconsole.truststore -J-Djavax.net.ssl.trustStorePassword=trustword 11.168.X.X:9712

where we specify the truststore as well as keystore, but I dont see any mention of keystore in nagios.

The error message I get is this,

JMX CRITICAL Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://11.168.X.X:9712/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]
... 4 more
Caused by: java.rmi.ConnectIOException: non-JRMP server at remote endpoint

Kindly advise.

Thanks

[mcapra]

Can you share the full (sanitized if neccesary) object definition for this service? You can view it by clicking the little floppy disk in the CCM:

2016_12_06_16_10_40_CCM_Nagios_XI.png

Same for the check_jmx command:

2016_12_06_16_12_46_CCM_Nagios_XI.png

[nagiosvick]

Hi,
Find the details. I have attached the check_jmx file.

define host {
host_name ma.federated.fds
use xiwizard_windowsserver_host
address 11.168.x.x
check_command check_jmx!9712!java.lang:type=Memory!NonHeapMemoryUsage!-K used -I HeapMemoryUsage -J used!428302272!5498760192!javax.net.ssl.trustStore=/root/jconsole.truststore!javax.net.ssl.trustStorePassword=trustword
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
contacts nagiosadmin
notification_interval 60
notification_period xi_timeperiod_24x7
icon_image win_server.png
statusmap_image win_server.png
_xiwizard windowsserver
register 1
}

Thanks.

[mcapra]

In your service definition's check_command:

Code: Select all

	check_command			check_jmx!9712!java.lang:type=Memory!NonHeapMemoryUsage!-K used -I HeapMemoryUsage -J used!428302272!5498760192!javax.net.ssl.trustStore=/root/jconsole.truststore!javax.net.ssl.trustStorePassword=trustword

The arguments you are providing do not line up with how the check_jmx command is defined:

Code: Select all

command_line                  		$USER1$/check_jmx -U service:jmx:rmi:///jndi/rmi://$HOSTADDRESS$:$ARG1$/jmxrmi -O $ARG2$ -A $ARG3$ $ARG4$ -vvvv -w $ARG5$ -c $ARG6$

Specifically, your $ARG5$ and $ARG6$ will be javax.net.ssl.trustStore=/root/jconsole.truststore and javax.net.ssl.trustStorePassword=trustword respectively instead of the expected warning and critical thresholds. You will need to alter your service's check_command to conform to the check_jmx command's conventions.

Additionally, it makes more sense to set the Java system properties in the check_jmx script itself like so:

Code: Select all

java -cp $RDIR/jmxquery.jar -Djavax.net.ssl.trustStore=/root/jconsole.truststore -Djavax.net.ssl.trustStorePassword=trustword JMXQuery $@

[nagiosvick]

I think the arg5 and arg6 are meant to be integers which defines the limits.

I get this error if I specify the parameters

COMMAND: /usr/local/nagios/libexec/check_jmx -U service:jmx:rmi:///jndi/rmi://11.168.x.x:9712/jmxrmi -O java.lang:type=Memory -A NonHeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w javax.net.ssl.trustStore=/root/jconsole.truststore -c javax.net.ssl.trustStorePassword=trustword
OUTPUT: JMX UNKNOWN java.lang.NumberFormatException: For input string: "javax.net.ssl.trustStore=/root/jconsole.truststore" connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://11.168.x.x:9712/jmxrmiorg.nagios.ParseError: java.lang.NumberFormatException: For input string: "javax.net.ssl.trustStore=/root/jconsole.truststore"
at org.nagios.JMXQuery.parse(JMXQuery.java:250)
at org.nagios.JMXQuery.main(JMXQuery.java:74)
Caused by: java.lang.NumberFormatException: For input string: "javax.net.ssl.trustStore=/root/jconsole.truststore"

[mcapra]

I messed up the ordering System properties always come before jars!

Code: Select all

java -Djavax.net.ssl.trustStore=/root/jconsole.truststore -Djavax.net.ssl.trustStorePassword=trustword -cp  $RDIR/jmxquery.jar  JMXQuery $@

[nagiosvick]

And when I edited the check_jmx script to this

java -cp $RDIR/jmxquery.jar -Djavax.net.ssl.trustStore=/root/jconsole.truststore -Djavax.net.ssl.trustStorePassword=trustword org.nagios.JMXQuery $@

In nagios console getting htis Error: Could not find or load main class JMXQuery , Error: Could not find or load main class org.nagios.JMXQuery

[mcapra]

Do you encounter any errors using this formatting:

Code: Select all

java -Djavax.net.ssl.trustStore=/root/jconsole.truststore -Djavax.net.ssl.trustStorePassword=trustword -cp  $RDIR/jmxquery.jar  JMXQuery $@

[nagiosvick]

Yes, just checked the earlier message and updated the java command.

I see that when I run the "Test check command", I see that the Output doesn't display anything. And when I rerun it, it says the status is up, but I still see the error as attached. Not sure if it works or not.

Update: I guess the host monitoring works, but the JMX monitoring doesn't seem to work

Please take a look, Thanks,

[mcapra]

Can you show me the Java system properties your application is being run with?

I get a completely different set of exceptions when I switch on the "SSL" switch against my previously working Tomcat machine (192.168.67.96) without changing anything on my Nagios XI machine:

Code: Select all

JMX CRITICAL Connection refused to host: 192.168.67.96: nested exception is:
java.net.ConnectException: Connection refused connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://192.168.67.96:9010/jmxrmijava.rmi.ConnectException: Connection refused to host: 192.168.67.96; nested exception is: 
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179)
at com.sun.proxy.$Proxy0.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
at JMXQuery.connect(JMXQuery.java:51)
at JMXQuery.main(JMXQuery.java:73)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:427)
at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
at javax.rmi.ssl.SslRMIClientSocketFactory.createSocket(SslRMIClientSocketFactory.java:121)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 12 more

I usually just do a ps aux | grep java if the properties are set at run time.