Trying to get check_jmx to work
Trying to get check_jmx to work
All,
I am trying to get the check_jmx plugin to work without success. I have installed the plugin, and the jmxquery.jar, in the /usr/local/nagios/libexec directory on the JMX server, and configured them to be owned by nagios:nagios with rwx privs. I also configured and restarted the nrpe agent on the jmx server. Executing the ./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:8088/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192 command yielded the following error:
JMX CRITICAL - Authentication failed! Credentials required connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://localhost:8088/jmxrmijava.lang.SecurityException: Authentication failed! Credentials required
at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:211)
at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:163)
at sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:227)
at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232)
at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
at javax.management.remote.rmi.RMIJRMPServerImpl$ExportedWrapper.newClient(RMIJRMPServerImpl.java:253)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
at com.sun.proxy.$Proxy0.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at jmxquery.DefaultJMXProvider.getConnector(DefaultJMXProvider.java:17)
at jmxquery.JMXQuery.connect(JMXQuery.java:95)
at jmxquery.JMXQuery.runCommand(JMXQuery.java:70)
at jmxquery.JMXQuery.main(JMXQuery.java:116)
I'have a java instance running configured with JMX on port 8088 but not sure it is correct. Please help.
I am trying to get the check_jmx plugin to work without success. I have installed the plugin, and the jmxquery.jar, in the /usr/local/nagios/libexec directory on the JMX server, and configured them to be owned by nagios:nagios with rwx privs. I also configured and restarted the nrpe agent on the jmx server. Executing the ./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:8088/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192 command yielded the following error:
JMX CRITICAL - Authentication failed! Credentials required connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://localhost:8088/jmxrmijava.lang.SecurityException: Authentication failed! Credentials required
at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:211)
at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:163)
at sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:227)
at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232)
at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
at javax.management.remote.rmi.RMIJRMPServerImpl$ExportedWrapper.newClient(RMIJRMPServerImpl.java:253)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
at com.sun.proxy.$Proxy0.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at jmxquery.DefaultJMXProvider.getConnector(DefaultJMXProvider.java:17)
at jmxquery.JMXQuery.connect(JMXQuery.java:95)
at jmxquery.JMXQuery.runCommand(JMXQuery.java:70)
at jmxquery.JMXQuery.main(JMXQuery.java:116)
I'have a java instance running configured with JMX on port 8088 but not sure it is correct. Please help.
Re: Trying to get check_jmx to work
JMX CRITICAL - Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://localhost:8088/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at jmxquery.DefaultJMXProvider.getConnector(DefaultJMXProvider.java:17)
at jmxquery.JMXQuery.connect(JMXQuery.java:95)
at jmxquery.JMXQuery.runCommand(JMXQuery.java:70)
at jmxquery.JMXQuery.main(JMXQuery.java:116)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
... 5 more
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:623)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343)
at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132)
... 10 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at java.net.Socket.connect(Socket.java:556)
at java.net.Socket.<init>(Socket.java:452)
at java.net.Socket.<init>(Socket.java:229)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617)
... 15 more
Firewall is off on machine as well
java.net.ConnectException: Connection refused (Connection refused)] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://localhost:8088/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at jmxquery.DefaultJMXProvider.getConnector(DefaultJMXProvider.java:17)
at jmxquery.JMXQuery.connect(JMXQuery.java:95)
at jmxquery.JMXQuery.runCommand(JMXQuery.java:70)
at jmxquery.JMXQuery.main(JMXQuery.java:116)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
... 5 more
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:623)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343)
at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132)
... 10 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at java.net.Socket.connect(Socket.java:556)
at java.net.Socket.<init>(Socket.java:452)
at java.net.Socket.<init>(Socket.java:229)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617)
... 15 more
Firewall is off on machine as well
Re: Trying to get check_jmx to work
Hi HIINNS,
In the second case it appears that the system is sending a reset packet back which implies the server is not listening on that port/interface. This is what I get back when I run a similar command against port 12345 on one of my lab machines:
You might want to verify that the server you setup is listening on the loopback interface.
or
Thanks and Best Regards,
Keith
In the second case it appears that the system is sending a reset packet back which implies the server is not listening on that port/interface. This is what I get back when I run a similar command against port 12345 on one of my lab machines:
Code: Select all
centos-79:/usr/local/nagios/libexec:{32:2} ./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:12345/jmxrmi O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192 JMX CRITICAL - Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://localhost:12345/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
Code: Select all
ss -nl
Code: Select all
netstat -nl
Keith
Re: Trying to get check_jmx to work
We have received the same error even with ports that are currently being listened to on that port.
Is there any specific configuration to get jmx setup previously. We have followed the steps according to RedHat and it did not help. I tried everything that I know to setup.
Is there any specific configuration to get jmx setup previously. We have followed the steps according to RedHat and it did not help. I tried everything that I know to setup.
Re: Trying to get check_jmx to work
On the system running JMX, what is the output of this command?
Code: Select all
ps aux | grep jmx
Re: Trying to get check_jmx to work
ps aux | grep jmx on the web server that we are trying to run jmx on
infodba 10859 0.0 0.0 112812 976 pts/0 S+ 06:42 0:00 grep --color=auto jmx
We run JMXProxyServer from the nagios server
nagios 2453 0.1 1.2 7842832 210524 ? Sl Feb22 3:15 /usr/jdk1.8.0_251/bin/java -classpath TcJMXProxyServer.jar:jboss-client.jar com/teamcenter/jmx/proxy/TcJMXProxyServer -pid_file /var/run/TcJMXProxyServer.pid
We enabled JMX through JBOSS on the web server by enabling the .conf files like were described by Redhat but the nagios server cannot connect to the web server
infodba 10859 0.0 0.0 112812 976 pts/0 S+ 06:42 0:00 grep --color=auto jmx
We run JMXProxyServer from the nagios server
nagios 2453 0.1 1.2 7842832 210524 ? Sl Feb22 3:15 /usr/jdk1.8.0_251/bin/java -classpath TcJMXProxyServer.jar:jboss-client.jar com/teamcenter/jmx/proxy/TcJMXProxyServer -pid_file /var/run/TcJMXProxyServer.pid
We enabled JMX through JBOSS on the web server by enabling the .conf files like were described by Redhat but the nagios server cannot connect to the web server
Re: Trying to get check_jmx to work
I'm looking into this and will post an update later, I haven't personally used a JMX Proxy but I have seen others use it successfully in the past (not something we technically support).
Re: Trying to get check_jmx to work
Can you provide access to that TcJMXProxyServer package? (you can PM it to me)
Your first post said you needed to pass credentials and the 2nd one is having a connection error. What is the difference between them, did you resolve the credentials issue and then after that received the 2nd one?
Can you link me to the guide that you're referring to when you setupTcJMXProxyServer/JMX/etc?
Your first post said you needed to pass credentials and the 2nd one is having a connection error. What is the difference between them, did you resolve the credentials issue and then after that received the 2nd one?
Can you link me to the guide that you're referring to when you setupTcJMXProxyServer/JMX/etc?
Re: Trying to get check_jmx to work
Please close out this case. I'm sorry for opening it in the 1st place. Our application administrators who support collecting JMX statistics need to get the application working. I do not have access to the code to make this happen
Re: Trying to get check_jmx to work
Hi HIINNS,
Not a problem - we'll lock out the forum thread.
Thanks and Best Regards,
Keith
Not a problem - we'll lock out the forum thread.
Thanks and Best Regards,
Keith