JMX SSL monitoring

[mcapra]

I was finally able to get this working for my Tomcat 7 machine. I'll attach a copy of the modified jmxquery.jar I was using which contains some extra bits that make SSL work against my test machine. Be aware that the classpath is different on the jmxquery attached to this post (i'm using JMXQuery instead of org.nagios.JMXQuery).

I generated the keystore on the Tomcat machine like so:

Code: Select all

keytool -genkeypair -alias tomcat -keyalg RSA -keystore keystore.jks -dname cn=test,ou=test,dc=example,dc=com

Java system properties for Tomcat:

Code: Select all

[root@localhost tomcat]# ps aux | grep java
tomcat    7977 12.5  4.9 3998064 191748 ?      Ssl  12:51   0:18 /usr/lib/jvm/jre/bin/java -Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=192.168.67.96 -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.registry.ssl=true -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.ssl.need.client.auth=false -Djavax.net.ssl.keyStorePassword=welcome -Djavax.net.ssl.keyStore=/etc/tomcat/keystore.jks -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.security.egd=file:/dev/./urandom -Djava.awt.headless=true -Xmx512m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Properties of note:

Code: Select all

-Dcom.sun.management.jmxremote.registry.ssl=true 
-Dcom.sun.management.jmxremote.ssl=true 
-Dcom.sun.management.jmxremote.ssl.need.client.auth=false 
-Djavax.net.ssl.keyStorePassword=welcome 
-Djavax.net.ssl.keyStore=/etc/tomcat/keystore.jks 
-Dcom.sun.management.jmxremote.local.only=false 
-Dcom.sun.management.jmxremote.authenticate=false

Then I moved the keystore to the Nagios XI machine (placed it in /usr/local/nagios/libexec/keystore.jks) as a lazy way of testing and used it as the truststore in my system properties.

Regular output from the Nagios XI machine using a modified jmxquery:

Code: Select all

[root@localhost libexec]# java -Djavax.net.ssl.trustStore=/usr/local/nagios/libexec/keystore.jks -Djavax.net.ssl.trustStorePassword=welcome -cp jmxquery.jar JMXQuery -U service:jmx:rmi:///jndi/rmi://192.168.67.96:9010/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv
JMX OK HeapMemoryUsage.used=21731024{committed=60817408;init=62914560;max=518979584;used=21731024}|used=21731024;0;0

And here's the output with extra debug info confirming the important SSL bits are functional (it's really long so I put it in a pastebin):
http://pastebin.com/kRrh0Njq

I also made the following changes to the check_jmx wrapper script that Nagios XI leverages:

Code: Select all

#!/bin/sh
#
# Nagios plugin to monitor Java JMX (http://java.sun.com/jmx)attributes.
#
RDIR=`dirname $0`
java -Djavax.net.ssl.trustStore=/usr/local/nagios/libexec/keystore.jks -Djavax.net.ssl.trustStorePassword=welcome -cp $RDIR/jmxquery.jar JMXQuery $@

And was able to use that just as well as the jar:

Code: Select all

[root@localhost libexec]# ./check_jmx -U service:jmx:rmi:///jndi/rmi://192.168.67.96:9010/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv
JMX OK HeapMemoryUsage.used=26155280{committed=60817408;init=62914560;max=518979584;used=26155280}|used=26155280;0;0

[nagiosvick]

jmx4.png

Hi mcapra,

Thanks for the latest jar file. Seems like it is able to get past that error. But is there any change that was done in the encryption protocols in the latest jar file? I'm getting the error error "JRMP connection establishment javax.net.ssl.sslhandshake received fatal alert bad_certificate. It still works from my laptop with jconsole, so getting the doubt according to the link below.

http://stackoverflow.com/questions/3220 ... r-java-6-8

I'm trying to remove openjdk 8 and install jdk 1.8.

Update: Same issue seen with jdk1.8 as well.

[mcapra]

A handshake failure either implies a bad certificate or an incorrect protocol. In this case, it appears as though the certificate provided to the remote host was rejected with bad_certificate.

Have you tried regenerating the keystore using RSA as the algorithm?

You could also add -Djavax.net.debug=all to your system properties to get more verbose output on why the handshake is failing.

[nagiosvick]

I'm already using RSA. Looking into the log file attached and trying out several things. If you know the root cause. pls let me know.

Guess it is because of some unsupported Cipher suite or something. I am using Signature algorithm name: SHA256withRSA Version: 3 in the keystore.

Thanks,

[dwhitfield]

So, just to make sure there isn't something weird going on, can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the Download Profile button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info in the thread (that will at least get us some info).

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

UPDATE: profile received and shared with techs

[nagiosvick]

Done.

[mcapra]

The remote server is definitely responding with a fatal during the TLS handshake:

Code: Select all

%% Invalidated:  [Session-3, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
main, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
main, WRITE: TLSv1.2 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 03 00 02 02 2E                               .......
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I would try generating a dummy keystore as I have and using that for the encryption. If it works, this would at least rule out any further connection related issues and isolate things to the certificate chain.

A co-worker who knows more about SSL than I do seems to think that part of the certificate chain is missing in this setup. Which is re-enforced by this article in which the problem is described for Confluence:
https://confluence.atlassian.com/kb/una ... 55358.html

[nagiosvick]

Yes, you were right. The problem was due to the certificate. I had missed to pass the parameters javax.net.ssl.keyStore=/opt/jconsole.keystore and javax.net.ssl.keyStorePassword in the check_jmx script.

It works fine in console as well. Awesome support folks. Appreciate it and thanks a bunch.

[dwhitfield]

Sounds like we're ready to lock this up, yes?

[nagiosvick]

Sure. You can close it or if anything to be done from my side. Will be glad to.