Page 2 of 4
Re: check_jmx with certificate
Posted: Thu Mar 31, 2016 3:11 pm
by haile711
hey rkennedy,
here is the result of nmap
Code: Select all
nmap 10.72.48.129
Starting Nmap 6.47 ( http://nmap.org ) at 2016-03-31 16:08 EDT
Nmap scan report for 10.72.48.129
Host is up (0.0061s latency).
Not shown: 985 filtered ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
81/tcp closed hosts2-ns
82/tcp closed xfer
83/tcp closed mit-ml-dev
84/tcp closed ctf
444/tcp closed snpp
445/tcp closed microsoft-ds
5666/tcp closed nrpe
8080/tcp closed http-proxy
8084/tcp open unknown
8180/tcp closed unknown
9080/tcp closed glrpc
9081/tcp closed unknown
10000/tcp closed snet-sensor-mgmt
Nmap done: 1 IP address (1 host up) scanned in 4.76 seconds
Here is the input command from nagios side
Code: Select all
./check_jmx1 -Djavax.net.ssl.trustStore="/usr/local/nagios/libexec/mashery-proxy.jks" -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.129:8084/jmxrmi -O "java.lang:type=MemoryPool,name=Perm Gen" -A Usage -K used
Re: check_jmx with certificate
Posted: Thu Mar 31, 2016 5:08 pm
by rkennedy
Can you please try to run these commands (skipping the check_jmx, this assumes your .jks is in the libexec folder) -
Code: Select all
java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.129:8084/jmxrmi -O "java.lang:type=MemoryPool,name=Perm Gen" -A Usage -K used -I Usage -vvvv
and
Code: Select all
java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.129:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv
I'm hoping this will give us a bit more output to work with.
Re: check_jmx with certificate
Posted: Thu Mar 31, 2016 5:09 pm
by jolson
Can you check logs on the remote host? I'm interested in seeing if there are any logs that are more verbose that what we're seeing on the Nagios side of things.
Check for potential logs with:
Are you running your test check commands as the nagios user or as root? According to the permissions of the datastore the 'nagios' user won't be able to read the contents of the keystore, and therefore can't complete the transaction.
Let us know, thanks!
Re: check_jmx with certificate
Posted: Thu Apr 07, 2016 3:36 pm
by haile711
Hey guys,
sorry for the late respond, here are the results from the command. Im still not getting the verbose output.
Code: Select all
[root@MA100DLVMON813 libexec]# java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.129:8084/jmxrmi -O "java.lang:type=MemoryPool,name=Perm Gen" -A Usage -K used -I Usage -vvvv
^C^C[root@MA100DLVMON813 libexec]#
[root@MA100DLVMON813 libexec]#
[root@MA100DLVMON813 libexec]#
[root@MA100DLVMON813 libexec]# java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.129:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv
and
Code: Select all
lsof | grep "\.log"
gzip 645 root 1w REG 0,17 418212 13725 /tmp/hats.log
gzip 645 root 2w REG 0,17 418212 13725 /tmp/hats.log
gzip 645 root 7w REG 0,17 20480 439310681 /tmp/fsfix.log (deleted)
gzip 645 root 8r REG 253,8 2660 7965 /opt/samgr/systemlogs/1456906648.log_messages
sa_comm 2404 root 1w REG 0,17 465395 13713 /tmp/sa_comm.log (deleted)
sa_comm 2404 root 2w REG 0,17 465395 13713 /tmp/sa_comm.log (deleted)
BESClient 2616 root 4u REG 253,6 163248 1843 /var/opt/BESClient/__BESData/__Global/Logs/20160407.log
klzagent 2961 root 1w REG 253,8 817 134916 /opt/IBM/ITM/logs/MA100DLVMON813_lz_1444902595.log
klzagent 2961 root 2w REG 253,8 817 134916 /opt/IBM/ITM/logs/MA100DLVMON813_lz_1444902595.log
klzagent 2961 root 9u REG 253,8 108295 134892 /opt/IBM/ITM/auditlogs/root.ma100dlvmon813_lz_audit.log
klzagent 2961 root 16w REG 253,8 1888298 134944 /opt/IBM/ITM/logs/MA100DLVMON813_lz_klzagent_561f76c3-02.log
gzip 3939 root 1w REG 0,17 418212 13725 /tmp/hats.log
gzip 3939 root 2w REG 0,17 418212 13725 /tmp/hats.log
gzip 3939 root 7w REG 0,17 20480 439310681 /tmp/fsfix.log (deleted)
gzip 3939 root 8r REG 253,8 264 7910 /opt/samgr/systemlogs/1456767012.log_messages
kcawd 5776 root 2w REG 253,8 817 134916 /opt/IBM/ITM/logs/MA100DLVMON813_lz_1444902595.log
kcawd 5776 root 3w REG 253,8 1024000 134917 /opt/IBM/ITM/logs/MA100DLVMON813_lz_klzagent_561f76c3-01.log (deleted)
kcawd 5776 root 8w REG 253,8 38410 134915 /opt/IBM/ITM/logs/MA100DLVMON813_lz_kcawd_561f773d-01.log
kcawd 5776 root 9u REG 253,8 108295 134892 /opt/IBM/ITM/auditlogs/root.ma100dlvmon813_lz_audit.log
php 6236 nagios 1w REG 253,4 255 3544920 /usr/local/nagiosxi/var/perfdataproc.log
php 6236 nagios 2w REG 253,4 255 3544920 /usr/local/nagiosxi/var/perfdataproc.log
php 6240 nagios 1w REG 253,4 721 3544918 /usr/local/nagiosxi/var/feedproc.log
php 6240 nagios 2w REG 253,4 721 3544918 /usr/local/nagiosxi/var/feedproc.log
php 6243 nagios 1w REG 253,4 2277 3544917 /usr/local/nagiosxi/var/eventman.log
php 6243 nagios 2w REG 253,4 2277 3544917 /usr/local/nagiosxi/var/eventman.log
php 6244 nagios 1w REG 253,4 210 3544914 /usr/local/nagiosxi/var/cmdsubsys.log
php 6244 nagios 2w REG 253,4 210 3544914 /usr/local/nagiosxi/var/cmdsubsys.log
php 6245 nagios 1w REG 253,4 8393 3544923 /usr/local/nagiosxi/var/sysstat.log
php 6245 nagios 2w REG 253,4 8393 3544923 /usr/local/nagiosxi/var/sysstat.log
k08agent 6959 root 1w REG 253,8 1121691 134929 /opt/IBM/ITM/logs/MA100DLVMON813_08_1445255762.log
k08agent 6959 root 2w REG 253,8 1121691 134929 /opt/IBM/ITM/logs/MA100DLVMON813_08_1445255762.log
k08agent 6959 root 3w REG 253,8 2386165 134930 /opt/IBM/ITM/logs/MA100DLVMON813_08_k08agent_5624da52-01.log
k08agent 6959 root 16u REG 253,8 3461 134933 /opt/IBM/ITM/auditlogs/root.ma100dlvmon813_08_audit.log
dsmcad 7235 root 3w REG 253,8 375918 7705 /opt/tivoli/tsm/client/ba/bin/dsmerror.log
dsmcad 7235 root 4w REG 253,8 287783 7706 /opt/tivoli/tsm/client/ba/bin/dsmwebcl.log
auditd 7868 root 4w REG 253,6 3455916 1440 /var/log/audit/audit.log
gzip 8475 root 1w REG 0,17 418212 13725 /tmp/hats.log
gzip 8475 root 2w REG 0,17 418212 13725 /tmp/hats.log
gzip 8475 root 7w REG 0,17 20480 439310681 /tmp/fsfix.log (deleted)
gzip 8475 root 8r REG 253,8 661 7944 /opt/samgr/systemlogs/1456882209.log_messages
nagios 9601 nagios 4u REG 253,4 898013 1966809 /usr/local/nagios/var/nagios.log
nagios 9678 nagios 4u REG 253,4 898013 1966809 /usr/local/nagios/var/nagios.log
gzip 15308 root 1w REG 0,17 418212 13725 /tmp/hats.log
gzip 15308 root 2w REG 0,17 418212 13725 /tmp/hats.log
gzip 15308 root 7w REG 0,17 20480 439310681 /tmp/fsfix.log (deleted)
gzip 15308 root 8r REG 253,8 1325 7967 /opt/samgr/systemlogs/1456928970.log_messages
mysqld 24882 mysql 1w REG 253,6 3919 5158 /var/log/mysqld.log
mysqld 24882 mysql 2w REG 253,6 3919 5158 /var/log/mysqld.log
hats.sa 26405 root 1w REG 0,17 418212 13725 /tmp/hats.log
hats.sa 26405 root 2w REG 0,17 418212 13725 /tmp/hats.log
postmaste 27652 postgres 1w REG 253,6 1292 133723 /var/lib/pgsql/pgstartup.log
postmaste 27652 postgres 2w REG 253,6 1292 133723 /var/lib/pgsql/pgstartup.log
postmaste 27652 postgres 4w REG 253,6 1804 136314 /var/lib/pgsql/data/pg_log/postgresql-Thu.log
Re: check_jmx with certificate
Posted: Thu Apr 07, 2016 5:00 pm
by rkennedy
It's getting a bit harder to troubleshoot as there is no verbosity.. Is there anything logging on the JMX server side so we could see where it's failing? At this point any additional information would help a lot.
Re: check_jmx with certificate
Posted: Fri Apr 08, 2016 8:07 am
by haile711
Good morning rkennedy,
We don't owned the JMX server therefore, we couldn't get the logs. Is there a reason why we couldn't get the verbose output for this?
Thanks
Re: check_jmx with certificate
Posted: Fri Apr 08, 2016 1:51 pm
by rkennedy
Can you request logs? It seems to be a local machine. I'm not too sure why it's not giving a verbose output. Do you have anyone with java knowledge on your team?
We're pretty limited to what we can do without seeing some sort of output (on either side), as there isn't anything to follow.
Re: check_jmx with certificate
Posted: Tue Apr 12, 2016 1:55 pm
by haile711
hey rkennedy,
I've got some verbose output, can you check to see if this is helpful information.
Code: Select all
java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.132:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv
JMX CRITICAL Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
java.net.ConnectException: Connection timed out] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://10.72.42.132:8084/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
java.net.ConnectException: Connection timed out]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:227)
at org.nagios.JMXQuery.connect(JMXQuery.java:53)
at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
java.net.ConnectException: Connection timed out]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
at javax.naming.InitialContext.lookup(InitialContext.java:411)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:286)
... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
java.net.ConnectException: Connection timed out
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:341)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
... 9 more
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:147)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
Re: check_jmx with certificate
Posted: Tue Apr 12, 2016 4:13 pm
by ssax
Try changing this:
Code: Select all
-Djavax.net.ssl.trustStore=mashery-proxy.jks
To the full path:
Code: Select all
-Djavax.net.ssl.trustStore=/full/path/to/your/jks/file/mashery-proxy.jks
Re: check_jmx with certificate
Posted: Wed Apr 13, 2016 8:01 am
by haile711
good morning ssax,
Even after changing the path to keystore file I still get the same errors.
Code: Select all
java -cp jmxquery.jar -Djavax.net.ssl.trustStore="/usr/local/nagios/libexec/mashery-proxy.jks" -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.132:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv
JMX CRITICAL Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
java.net.ConnectException: Connection timed out] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://10.72.42.132:8084/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
java.net.ConnectException: Connection timed out]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:227)
at org.nagios.JMXQuery.connect(JMXQuery.java:53)
at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
java.net.ConnectException: Connection timed out]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
at javax.naming.InitialContext.lookup(InitialContext.java:411)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:286)
... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
java.net.ConnectException: Connection timed out
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:341)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
... 9 more
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:147)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 14 more