check_jmx with certificate

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
haile711
Posts: 197
Joined: Thu May 28, 2015 7:36 am

Re: check_jmx with certificate

Post by haile711 »

hey rkennedy,

here is the result of nmap

Code: Select all

nmap 10.72.48.129

Starting Nmap 6.47 ( http://nmap.org ) at 2016-03-31 16:08 EDT
Nmap scan report for 10.72.48.129
Host is up (0.0061s latency).
Not shown: 985 filtered ports
PORT      STATE  SERVICE
22/tcp    open   ssh
80/tcp    open   http
81/tcp    closed hosts2-ns
82/tcp    closed xfer
83/tcp    closed mit-ml-dev
84/tcp    closed ctf
444/tcp   closed snpp
445/tcp   closed microsoft-ds
5666/tcp  closed nrpe
8080/tcp  closed http-proxy
8084/tcp  open   unknown
8180/tcp  closed unknown
9080/tcp  closed glrpc
9081/tcp  closed unknown
10000/tcp closed snet-sensor-mgmt

Nmap done: 1 IP address (1 host up) scanned in 4.76 seconds

Here is the input command from nagios side

Code: Select all

 ./check_jmx1 -Djavax.net.ssl.trustStore="/usr/local/nagios/libexec/mashery-proxy.jks" -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.129:8084/jmxrmi -O "java.lang:type=MemoryPool,name=Perm Gen" -A Usage -K used
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: check_jmx with certificate

Post by rkennedy »

Can you please try to run these commands (skipping the check_jmx, this assumes your .jks is in the libexec folder) -

Code: Select all

java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.129:8084/jmxrmi -O "java.lang:type=MemoryPool,name=Perm Gen" -A Usage -K used -I Usage -vvvv
and

Code: Select all

java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.129:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv
I'm hoping this will give us a bit more output to work with.
Former Nagios Employee
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: check_jmx with certificate

Post by jolson »

Can you check logs on the remote host? I'm interested in seeing if there are any logs that are more verbose that what we're seeing on the Nagios side of things.

Check for potential logs with:

Code: Select all

lsof | grep "\.log"
Are you running your test check commands as the nagios user or as root? According to the permissions of the datastore the 'nagios' user won't be able to read the contents of the keystore, and therefore can't complete the transaction.

Let us know, thanks!
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
haile711
Posts: 197
Joined: Thu May 28, 2015 7:36 am

Re: check_jmx with certificate

Post by haile711 »

Hey guys,

sorry for the late respond, here are the results from the command. Im still not getting the verbose output.

Code: Select all

[root@MA100DLVMON813 libexec]# java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.129:8084/jmxrmi -O "java.lang:type=MemoryPool,name=Perm Gen" -A Usage -K used -I Usage -vvvv
^C^C[root@MA100DLVMON813 libexec]#
[root@MA100DLVMON813 libexec]#
[root@MA100DLVMON813 libexec]#
[root@MA100DLVMON813 libexec]# java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.129:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv
and

Code: Select all

lsof | grep "\.log"
gzip        645      root    1w      REG               0,17    418212      13725 /tmp/hats.log
gzip        645      root    2w      REG               0,17    418212      13725 /tmp/hats.log
gzip        645      root    7w      REG               0,17     20480  439310681 /tmp/fsfix.log (deleted)
gzip        645      root    8r      REG              253,8      2660       7965 /opt/samgr/systemlogs/1456906648.log_messages
sa_comm    2404      root    1w      REG               0,17    465395      13713 /tmp/sa_comm.log (deleted)
sa_comm    2404      root    2w      REG               0,17    465395      13713 /tmp/sa_comm.log (deleted)
BESClient  2616      root    4u      REG              253,6    163248       1843 /var/opt/BESClient/__BESData/__Global/Logs/20160407.log
klzagent   2961      root    1w      REG              253,8       817     134916 /opt/IBM/ITM/logs/MA100DLVMON813_lz_1444902595.log
klzagent   2961      root    2w      REG              253,8       817     134916 /opt/IBM/ITM/logs/MA100DLVMON813_lz_1444902595.log
klzagent   2961      root    9u      REG              253,8    108295     134892 /opt/IBM/ITM/auditlogs/root.ma100dlvmon813_lz_audit.log
klzagent   2961      root   16w      REG              253,8   1888298     134944 /opt/IBM/ITM/logs/MA100DLVMON813_lz_klzagent_561f76c3-02.log
gzip       3939      root    1w      REG               0,17    418212      13725 /tmp/hats.log
gzip       3939      root    2w      REG               0,17    418212      13725 /tmp/hats.log
gzip       3939      root    7w      REG               0,17     20480  439310681 /tmp/fsfix.log (deleted)
gzip       3939      root    8r      REG              253,8       264       7910 /opt/samgr/systemlogs/1456767012.log_messages
kcawd      5776      root    2w      REG              253,8       817     134916 /opt/IBM/ITM/logs/MA100DLVMON813_lz_1444902595.log
kcawd      5776      root    3w      REG              253,8   1024000     134917 /opt/IBM/ITM/logs/MA100DLVMON813_lz_klzagent_561f76c3-01.log (deleted)
kcawd      5776      root    8w      REG              253,8     38410     134915 /opt/IBM/ITM/logs/MA100DLVMON813_lz_kcawd_561f773d-01.log
kcawd      5776      root    9u      REG              253,8    108295     134892 /opt/IBM/ITM/auditlogs/root.ma100dlvmon813_lz_audit.log
php        6236    nagios    1w      REG              253,4       255    3544920 /usr/local/nagiosxi/var/perfdataproc.log
php        6236    nagios    2w      REG              253,4       255    3544920 /usr/local/nagiosxi/var/perfdataproc.log
php        6240    nagios    1w      REG              253,4       721    3544918 /usr/local/nagiosxi/var/feedproc.log
php        6240    nagios    2w      REG              253,4       721    3544918 /usr/local/nagiosxi/var/feedproc.log
php        6243    nagios    1w      REG              253,4      2277    3544917 /usr/local/nagiosxi/var/eventman.log
php        6243    nagios    2w      REG              253,4      2277    3544917 /usr/local/nagiosxi/var/eventman.log
php        6244    nagios    1w      REG              253,4       210    3544914 /usr/local/nagiosxi/var/cmdsubsys.log
php        6244    nagios    2w      REG              253,4       210    3544914 /usr/local/nagiosxi/var/cmdsubsys.log
php        6245    nagios    1w      REG              253,4      8393    3544923 /usr/local/nagiosxi/var/sysstat.log
php        6245    nagios    2w      REG              253,4      8393    3544923 /usr/local/nagiosxi/var/sysstat.log
k08agent   6959      root    1w      REG              253,8   1121691     134929 /opt/IBM/ITM/logs/MA100DLVMON813_08_1445255762.log
k08agent   6959      root    2w      REG              253,8   1121691     134929 /opt/IBM/ITM/logs/MA100DLVMON813_08_1445255762.log
k08agent   6959      root    3w      REG              253,8   2386165     134930 /opt/IBM/ITM/logs/MA100DLVMON813_08_k08agent_5624da52-01.log
k08agent   6959      root   16u      REG              253,8      3461     134933 /opt/IBM/ITM/auditlogs/root.ma100dlvmon813_08_audit.log
dsmcad     7235      root    3w      REG              253,8    375918       7705 /opt/tivoli/tsm/client/ba/bin/dsmerror.log
dsmcad     7235      root    4w      REG              253,8    287783       7706 /opt/tivoli/tsm/client/ba/bin/dsmwebcl.log
auditd     7868      root    4w      REG              253,6   3455916       1440 /var/log/audit/audit.log
gzip       8475      root    1w      REG               0,17    418212      13725 /tmp/hats.log
gzip       8475      root    2w      REG               0,17    418212      13725 /tmp/hats.log
gzip       8475      root    7w      REG               0,17     20480  439310681 /tmp/fsfix.log (deleted)
gzip       8475      root    8r      REG              253,8       661       7944 /opt/samgr/systemlogs/1456882209.log_messages
nagios     9601    nagios    4u      REG              253,4    898013    1966809 /usr/local/nagios/var/nagios.log
nagios     9678    nagios    4u      REG              253,4    898013    1966809 /usr/local/nagios/var/nagios.log
gzip      15308      root    1w      REG               0,17    418212      13725 /tmp/hats.log
gzip      15308      root    2w      REG               0,17    418212      13725 /tmp/hats.log
gzip      15308      root    7w      REG               0,17     20480  439310681 /tmp/fsfix.log (deleted)
gzip      15308      root    8r      REG              253,8      1325       7967 /opt/samgr/systemlogs/1456928970.log_messages
mysqld    24882     mysql    1w      REG              253,6      3919       5158 /var/log/mysqld.log
mysqld    24882     mysql    2w      REG              253,6      3919       5158 /var/log/mysqld.log
hats.sa   26405      root    1w      REG               0,17    418212      13725 /tmp/hats.log
hats.sa   26405      root    2w      REG               0,17    418212      13725 /tmp/hats.log
postmaste 27652  postgres    1w      REG              253,6      1292     133723 /var/lib/pgsql/pgstartup.log
postmaste 27652  postgres    2w      REG              253,6      1292     133723 /var/lib/pgsql/pgstartup.log
postmaste 27652  postgres    4w      REG              253,6      1804     136314 /var/lib/pgsql/data/pg_log/postgresql-Thu.log
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: check_jmx with certificate

Post by rkennedy »

It's getting a bit harder to troubleshoot as there is no verbosity.. Is there anything logging on the JMX server side so we could see where it's failing? At this point any additional information would help a lot.
Former Nagios Employee
Top
haile711
Posts: 197
Joined: Thu May 28, 2015 7:36 am

Re: check_jmx with certificate

Post by haile711 »

Good morning rkennedy,

We don't owned the JMX server therefore, we couldn't get the logs. Is there a reason why we couldn't get the verbose output for this?

Thanks
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: check_jmx with certificate

Post by rkennedy »

Can you request logs? It seems to be a local machine. I'm not too sure why it's not giving a verbose output. Do you have anyone with java knowledge on your team?

We're pretty limited to what we can do without seeing some sort of output (on either side), as there isn't anything to follow.
Former Nagios Employee
Top
haile711
Posts: 197
Joined: Thu May 28, 2015 7:36 am

Re: check_jmx with certificate

Post by haile711 »

hey rkennedy,
I've got some verbose output, can you check to see if this is helpful information.

Code: Select all

java -cp jmxquery.jar -Djavax.net.ssl.trustStore=mashery-proxy.jks -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.132:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv
JMX CRITICAL Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
        java.net.ConnectException: Connection timed out] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://10.72.42.132:8084/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
        java.net.ConnectException: Connection timed out]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:227)
        at org.nagios.JMXQuery.connect(JMXQuery.java:53)
        at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
        java.net.ConnectException: Connection timed out]
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
        at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
        at javax.naming.InitialContext.lookup(InitialContext.java:411)
        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929)
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:286)
        ... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
        java.net.ConnectException: Connection timed out
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:341)
        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
        ... 9 more
Caused by: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:147)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
Top
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: check_jmx with certificate

Post by ssax »

Try changing this:

Code: Select all

-Djavax.net.ssl.trustStore=mashery-proxy.jks
To the full path:

Code: Select all

-Djavax.net.ssl.trustStore=/full/path/to/your/jks/file/mashery-proxy.jks
Top
haile711
Posts: 197
Joined: Thu May 28, 2015 7:36 am

Re: check_jmx with certificate

Post by haile711 »

good morning ssax,

Even after changing the path to keystore file I still get the same errors.

Code: Select all

java -cp jmxquery.jar -Djavax.net.ssl.trustStore="/usr/local/nagios/libexec/mashery-proxy.jks" -Djavax.net.ssl.trustStorePassword=trust1 org.nagios.JMXQuery -U service:jmx:rmi:///jndi/rmi://10.72.42.132:8084/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv
JMX CRITICAL Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
        java.net.ConnectException: Connection timed out] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://10.72.42.132:8084/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
        java.net.ConnectException: Connection timed out]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:227)
        at org.nagios.JMXQuery.connect(JMXQuery.java:53)
        at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
        java.net.ConnectException: Connection timed out]
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
        at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
        at javax.naming.InitialContext.lookup(InitialContext.java:411)
        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929)
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:286)
        ... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host: 10.72.42.132; nested exception is:
        java.net.ConnectException: Connection timed out
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:341)
        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
        ... 9 more
Caused by: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:147)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
        ... 14 more
Top
Locked

Return to “Nagios XI”