Page 1 of 3
JMX SSL monitoring
Posted: Tue Dec 06, 2016 3:48 pm
by nagiosvick
Hi,
I was able to successfully establish a connection with the JMX port through SSL connection through jconsole. Trying to achieve the same through Nagios. But I am not able to connect. I have attached the screenshot of the parameters I try to pass.
When I tried with standalone Jconsole to connect I had used this command to start jconsole
C:\certs>jconsole -J-Djavax.net.ssl.keyStore=C:\certs\jconsole.keystore -J-Djavax.net.ssl.keyStorePassword=password -J-Djavax.net.ssl.trustStore=C:\certs\jconsole.truststore -J-Djavax.net.ssl.trustStorePassword=trustword 11.168.X.X:9712
where we specify the truststore as well as keystore, but I dont see any mention of keystore in nagios.
The error message I get is this,
JMX CRITICAL Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://11.168.X.X:9712/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]
... 4 more
Caused by: java.rmi.ConnectIOException: non-JRMP server at remote endpoint
Kindly advise.
Thanks
Re: JMX SSL monitoring
Posted: Tue Dec 06, 2016 5:11 pm
by mcapra
Can you share the full (sanitized if neccesary) object definition for this service? You can view it by clicking the little floppy disk in the CCM:
2016_12_06_16_10_40_CCM_Nagios_XI.png
Same for the check_jmx command:
2016_12_06_16_12_46_CCM_Nagios_XI.png
Re: JMX SSL monitoring
Posted: Wed Dec 07, 2016 10:32 am
by nagiosvick
Hi,
Find the details. I have attached the check_jmx file.
define host {
host_name ma.federated.fds
use xiwizard_windowsserver_host
address 11.168.x.x
check_command check_jmx!9712!java.lang:type=Memory!NonHeapMemoryUsage!-K used -I HeapMemoryUsage -J used!428302272!5498760192!javax.net.ssl.trustStore=/root/jconsole.truststore!javax.net.ssl.trustStorePassword=trustword
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
contacts nagiosadmin
notification_interval 60
notification_period xi_timeperiod_24x7
icon_image win_server.png
statusmap_image win_server.png
_xiwizard windowsserver
register 1
}
Thanks.
Re: JMX SSL monitoring
Posted: Wed Dec 07, 2016 4:27 pm
by mcapra
In your service definition's
check_command:
Code: Select all
check_command check_jmx!9712!java.lang:type=Memory!NonHeapMemoryUsage!-K used -I HeapMemoryUsage -J used!428302272!5498760192!javax.net.ssl.trustStore=/root/jconsole.truststore!javax.net.ssl.trustStorePassword=trustword
The arguments you are providing do not line up with how the
check_jmx command is defined:
Code: Select all
command_line $USER1$/check_jmx -U service:jmx:rmi:///jndi/rmi://$HOSTADDRESS$:$ARG1$/jmxrmi -O $ARG2$ -A $ARG3$ $ARG4$ -vvvv -w $ARG5$ -c $ARG6$
Specifically, your $ARG5$ and $ARG6$ will be
javax.net.ssl.trustStore=/root/jconsole.truststore and
javax.net.ssl.trustStorePassword=trustword respectively instead of the expected warning and critical thresholds. You will need to alter your service's
check_command to conform to the
check_jmx command's conventions.
Additionally, it makes more sense to set the Java system properties in the check_jmx script itself like so:
Code: Select all
java -cp $RDIR/jmxquery.jar -Djavax.net.ssl.trustStore=/root/jconsole.truststore -Djavax.net.ssl.trustStorePassword=trustword JMXQuery $@
Re: JMX SSL monitoring
Posted: Wed Dec 07, 2016 10:07 pm
by nagiosvick
I think the arg5 and arg6 are meant to be integers which defines the limits.
I get this error if I specify the parameters
COMMAND: /usr/local/nagios/libexec/check_jmx -U service:jmx:rmi:///jndi/rmi://11.168.x.x:9712/jmxrmi -O java.lang:type=Memory -A NonHeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w javax.net.ssl.trustStore=/root/jconsole.truststore -c javax.net.ssl.trustStorePassword=trustword
OUTPUT: JMX UNKNOWN java.lang.NumberFormatException: For input string: "javax.net.ssl.trustStore=/root/jconsole.truststore" connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://11.168.x.x:9712/jmxrmiorg.nagios.ParseError: java.lang.NumberFormatException: For input string: "javax.net.ssl.trustStore=/root/jconsole.truststore"
at org.nagios.JMXQuery.parse(JMXQuery.java:250)
at org.nagios.JMXQuery.main(JMXQuery.java:74)
Caused by: java.lang.NumberFormatException: For input string: "javax.net.ssl.trustStore=/root/jconsole.truststore"
Re: JMX SSL monitoring
Posted: Thu Dec 08, 2016 11:43 am
by mcapra
I messed up the ordering
System properties always come before jars!
Code: Select all
java -Djavax.net.ssl.trustStore=/root/jconsole.truststore -Djavax.net.ssl.trustStorePassword=trustword -cp $RDIR/jmxquery.jar JMXQuery $@
Re: JMX SSL monitoring
Posted: Thu Dec 08, 2016 12:08 pm
by nagiosvick
And when I edited the check_jmx script to this
java -cp $RDIR/jmxquery.jar -Djavax.net.ssl.trustStore=/root/jconsole.truststore -Djavax.net.ssl.trustStorePassword=trustword org.nagios.JMXQuery $@
In nagios console getting htis Error: Could not find or load main class JMXQuery , Error: Could not find or load main class org.nagios.JMXQuery
Re: JMX SSL monitoring
Posted: Thu Dec 08, 2016 1:50 pm
by mcapra
Do you encounter any errors using this formatting:
Code: Select all
java -Djavax.net.ssl.trustStore=/root/jconsole.truststore -Djavax.net.ssl.trustStorePassword=trustword -cp $RDIR/jmxquery.jar JMXQuery $@
Re: JMX SSL monitoring
Posted: Thu Dec 08, 2016 2:18 pm
by nagiosvick
Yes, just checked the earlier message and updated the java command.
I see that when I run the "Test check command", I see that the Output doesn't display anything. And when I rerun it, it says the status is up, but I still see the error as attached. Not sure if it works or not.
Update: I guess the host monitoring works, but the JMX monitoring doesn't seem to work
Please take a look, Thanks,
Re: JMX SSL monitoring
Posted: Thu Dec 08, 2016 3:39 pm
by mcapra
Can you show me the Java system properties your application is being run with?
I get a completely different set of exceptions when I switch on the "SSL" switch against my previously working Tomcat machine (192.168.67.96) without changing anything on my Nagios XI machine:
Code: Select all
JMX CRITICAL Connection refused to host: 192.168.67.96: nested exception is:
java.net.ConnectException: Connection refused connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://192.168.67.96:9010/jmxrmijava.rmi.ConnectException: Connection refused to host: 192.168.67.96; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179)
at com.sun.proxy.$Proxy0.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
at JMXQuery.connect(JMXQuery.java:51)
at JMXQuery.main(JMXQuery.java:73)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:427)
at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
at javax.rmi.ssl.SslRMIClientSocketFactory.createSocket(SslRMIClientSocketFactory.java:121)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 12 more
I usually just do a
ps aux | grep java if the properties are set at run time.