Trying to get check_jmx to work

[HIINNS]

All,

I am trying to get the check_jmx plugin to work without success. I have installed the plugin, and the jmxquery.jar, in the /usr/local/nagios/libexec directory on the JMX server, and configured them to be owned by nagios:nagios with rwx privs. I also configured and restarted the nrpe agent on the jmx server. Executing the ./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:8088/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192 command yielded the following error:

JMX CRITICAL - Authentication failed! Credentials required connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://localhost:8088/jmxrmijava.lang.SecurityException: Authentication failed! Credentials required
at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:211)
at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:163)
at sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:227)
at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232)
at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
at javax.management.remote.rmi.RMIJRMPServerImpl$ExportedWrapper.newClient(RMIJRMPServerImpl.java:253)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
at com.sun.proxy.$Proxy0.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at jmxquery.DefaultJMXProvider.getConnector(DefaultJMXProvider.java:17)
at jmxquery.JMXQuery.connect(JMXQuery.java:95)
at jmxquery.JMXQuery.runCommand(JMXQuery.java:70)
at jmxquery.JMXQuery.main(JMXQuery.java:116)

I'have a java instance running configured with JMX on port 8088 but not sure it is correct. Please help.

[HIINNS]

JMX CRITICAL - Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://localhost:8088/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at jmxquery.DefaultJMXProvider.getConnector(DefaultJMXProvider.java:17)
at jmxquery.JMXQuery.connect(JMXQuery.java:95)
at jmxquery.JMXQuery.runCommand(JMXQuery.java:70)
at jmxquery.JMXQuery.main(JMXQuery.java:116)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
... 5 more
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:623)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343)
at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132)
... 10 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at java.net.Socket.connect(Socket.java:556)
at java.net.Socket.<init>(Socket.java:452)
at java.net.Socket.<init>(Socket.java:229)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617)
... 15 more


Firewall is off on machine as well

[kfanselow]

Hi HIINNS,

In the second case it appears that the system is sending a reset packet back which implies the server is not listening on that port/interface. This is what I get back when I run a similar command against port 12345 on one of my lab machines:

Code: Select all

centos-79:/usr/local/nagios/libexec:{32:2} ./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:12345/jmxrmi O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192    JMX CRITICAL - Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
        java.net.ConnectException: Connection refused (Connection refused)] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://localhost:12345/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 

You might want to verify that the server you setup is listening on the loopback interface.

Code: Select all

ss -nl 

or

Code: Select all

netstat -nl 

Thanks and Best Regards,
Keith

[HIINNS]

We have received the same error even with ports that are currently being listened to on that port.

Is there any specific configuration to get jmx setup previously. We have followed the steps according to RedHat and it did not help. I tried everything that I know to setup.

[ssax]

On the system running JMX, what is the output of this command?

Code: Select all

ps aux | grep jmx

[HIINNS]

ps aux | grep jmx on the web server that we are trying to run jmx on
infodba 10859 0.0 0.0 112812 976 pts/0 S+ 06:42 0:00 grep --color=auto jmx

We run JMXProxyServer from the nagios server
nagios 2453 0.1 1.2 7842832 210524 ? Sl Feb22 3:15 /usr/jdk1.8.0_251/bin/java -classpath TcJMXProxyServer.jar:jboss-client.jar com/teamcenter/jmx/proxy/TcJMXProxyServer -pid_file /var/run/TcJMXProxyServer.pid

We enabled JMX through JBOSS on the web server by enabling the .conf files like were described by Redhat but the nagios server cannot connect to the web server

[ssax]

I'm looking into this and will post an update later, I haven't personally used a JMX Proxy but I have seen others use it successfully in the past (not something we technically support).

[ssax]

Can you provide access to that TcJMXProxyServer package? (you can PM it to me)

Your first post said you needed to pass credentials and the 2nd one is having a connection error. What is the difference between them, did you resolve the credentials issue and then after that received the 2nd one?

Can you link me to the guide that you're referring to when you setupTcJMXProxyServer/JMX/etc?

[HIINNS]

Please close out this case. I'm sorry for opening it in the 1st place. Our application administrators who support collecting JMX statistics need to get the application working. I do not have access to the code to make this happen

[kfanselow]

Hi HIINNS,

Not a problem - we'll lock out the forum thread.

Thanks and Best Regards,
Keith