Page 1 of 2

JMX Connection refused

Posted: Mon Nov 08, 2021 3:42 am
by sacom01
hi team,
i have setup jmx for monitor jvm
- Added parameters to jvm :
-Djavax.management.builder.initial=
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=9000

- port was ready :
[root@dc-nagiosxi-uat01 mSign]# telnet 192.168.104.250 9000
Trying 192.168.104.250...
Connected to 192.168.104.250.
Escape character is '^]'.

- but run check_jvm was getting error :
[root@dc-nagiosxi-uat01 libexec]# /usr/local/nagios/libexec/check_jmx -U service:jmx:rmi:///jndi/rmi://192.168.104.250:9000/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192
JMX CRITICAL - Error opening RMI connection: Connection refused to host: 192.168.104.250; nested exception is:
java.net.ConnectException: Connection timed out (Connection timed out)

Re: JMX Connection refused

Posted: Mon Nov 08, 2021 12:41 pm
by pbroste
Hello @sacom01

Thanks for reaching out, and providing the telnet results on port 9000. Want to verify that there is no security application or anything else blocking since we are getting:
Connection refused to host
Verify selinux is disabled:

Code: Select all

sestatus
Please post the output of this command from your XI server:

Code: Select all

nmap -p 9000 192.168.104.250
and

Code: Select all

sudo ss -tulpn | grep -Ei '9000'
Please let us know the results,
Perry

Re: JMX Connection refused

Posted: Mon Nov 08, 2021 8:28 pm
by sacom01
hi Perry,
Thanks for your reply.

Verify selinux is disabled: --> client server is AIX 7.1, already disable firewalld.
Nagios server is Redhat 7

Re: JMX Connection refused

Posted: Tue Nov 09, 2021 4:03 pm
by pbroste
Hello @sacom01

Thanks for following up with the details and screenshot. What do you have set on the AIX client side?
-Dcom.sun.management.jmxremote.port=9000
-Dcom.sun.management.jmxremote.authenticate=?
-Dcom.sun.management.jmxremote.ssl=?
Do you see anything logged on the AIX client that is failing?

Thanks,
Perry

Re: JMX Connection refused

Posted: Tue Nov 09, 2021 4:21 pm
by pbroste
Hello @sacom01

Please also check to see what 'netstat' says about port 9000 on both.
  • netstat -an | grep -Ei '9000'
Thanks,
Perry

Re: JMX Connection refused

Posted: Tue Nov 09, 2021 10:10 pm
by sacom01
hi Perry,
No log appear in AIX client side.
thanks.

Re: JMX Connection refused

Posted: Wed Nov 10, 2021 12:14 pm
by pbroste
Hello @sacom01

Thanks for following up with the details. We see that JMX monitoring is enabled with "-Dcom.sun.management.jmxremote"

Understand that you want to end up running this check:
/usr/local/nagios/libexec/check_jmx -U service:jmx:rmi:///jndi/rmi://192.168.104.250:9000/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192
For the time being let's verify that we can get a check to circle back with results:
  • Code: Select all

    java -jar check_jvm.jar -s "service:jmx:rmi:///jndi/rmi://yourhostaddresshere:9000/jmxrmi" -u 'username' -p 'password'-t token -C "Uptime" -w 10 -c 20
  • Code: Select all

    java -jar check_jvm.jar -s service:jmx:rmi:///jndi/rmi://yourhostaddresshere:9000/jmxrmi -u 'username' -p 'password' -t token -C Uptime,ThreadCount -w 10 -c 20
Let us know the results,
Perry

Re: JMX Connection refused

Posted: Wed Nov 10, 2021 9:51 pm
by sacom01
hi Perry,
i dont know what is "-u 'username' -p 'password'", i indicated connection protocol is unsecure :
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

If i do not specify username password, the results are :

[root@dc-nagiosxi-uat01 libexec]# java -jar check_jvm.jar -s service:jmx:rmi:///jndi/rmi://192.168.104.250:9000/jmxrmi -t mytoken -C Uptime,ThreadCount -w 10 -c 20
UNKNOWN: Failed to connect: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.104.250; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]


[root@dc-nagiosxi-uat01 libexec]# java -jar check_jvm.jar -s "service:jmx:rmi:///jndi/rmi://192.168.104.250:9000/jmxrmi" -t mytoken -C "Uptime" -w 10 -c 20

UNKNOWN: Failed to connect: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.104.250; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]

Re: JMX Connection refused

Posted: Thu Nov 11, 2021 2:56 pm
by pbroste
Hello @sacom01

Checking on the specifics on the credentials, meantime please check on this, should have both:

Code: Select all

-Dcom.sun.management.jmxremote.port=9000
-Dcom.sun.management.jmxremote.rmi.port=9000
Thanks,
Perry

Re: JMX Connection refused

Posted: Thu Nov 11, 2021 9:41 pm
by sacom01
hi,
i tried but no luck.
-Djavax.management.builder.initial=
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=9000
-Dcom.sun.management.jmxremote.rmi.port=9000

[root@dc-nagiosxi-uat01 libexec]# java -jar check_jvm.jar -s "service:jmx:rmi:///jndi/rmi://192.168.104.250:9000/jmxrmi" -u 'username' -p 'password' -t mytoken -C "Uptime" -w 10 -c 20
UNKNOWN: Failed to connect: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.104.250; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]

Notice: first, i want to use unsecure connection, so we can skip username and pw. And we need monitor JVM, not jettty