Monitor JMX With Nagios XI : JMX CRITICAL Failed to retrieve

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
Nags007
Posts: 58
Joined: Wed Mar 08, 2017 12:32 pm

Monitor JMX With Nagios XI : JMX CRITICAL Failed to retrieve

Post by Nags007 »

Hi Team,
There is a running NagiosXI server,which is working fine. There was a need for monitoring the JMX with NagiosXI.

I am following the instruction of
https://assets.nagios.com/downloads/nag ... 1489001132

In the Pre requisite in the document you have given :
====================================================
Prerequisites
This document assumes you have the following:
• A remote server running JMX
• Nagios XI server with a network route to the JMX server

A. Please let me know, how to setup / configure the - remove server running JMX ? is there any download of JMX ? how to check JMX is running ?
B. What's the meaning of NagiosXI server with network route to JMX server ? ( what is this JMX server )?
C. Where should the check_jmx plugin to be installed ? ( whether in Java running application server ) or NagiosXI server or any other server ?

Now i have two servers :
1. NagiosXI Server and
2. Java Application server ( where NRPE is installed ) and Java applications ports are 8062, 8063, 8064.
In this Java Appliction running server, i have installed the PLUGIN.

Code: Select all

      root@java-apps-server /usr/local/nagios/libexec # ll
        ============================================
  total 6280
-rwxr-xr-x 1 root nagios 179314 Mar 21 14:52 check_apt
-rwxr-xr-x 1 root nagios   6897 Mar 21 14:52 check_asterisk.pl
-rwxr-xr-x 1 root nagios   1978 Mar 21 14:52 check_asterisk_sip_peers.sh
-rwxr-xr-x 1 root nagios   2242 Mar 21 14:52 check_breeze
-rwxr-xr-x 1 root nagios 179635 Mar 21 14:52 check_by_ssh
lrwxrwxrwx 1 root root        9 Mar 21 14:52 check_clamd -> check_tcp
-rwxr-xr-x 1 root nagios 137061 Mar 21 14:52 check_cluster
-rwxr-xr-x 1 root nagios   5582 Mar 21 14:52 check_cpu_stats.sh
-rwsr-xr-x 1 root nagios 175242 Mar 21 14:52 check_dhcp
-rwxr-xr-x 1 root nagios 174357 Mar 21 14:52 check_dig
-rwxr-xr-x 1 root nagios 193444 Mar 21 14:52 check_disk
-rwxr-xr-x 1 root nagios   9289 Mar 21 14:52 check_disk_smb
-rwxr-xr-x 1 root nagios 189011 Mar 21 14:52 check_dns
-rwxr-xr-x 1 root nagios  85099 Mar 21 14:52 check_dummy
-rwxr-xr-x 1 root nagios   3349 Mar 21 14:52 check_file_age
-rwxr-xr-x 1 root nagios   6315 Mar 21 14:52 check_flexlm
lrwxrwxrwx 1 root root        9 Mar 21 14:52 check_ftp -> check_tcp
-rwxr-xr-x 1 root nagios 335788 Mar 21 14:52 check_http
-rwsr-xr-x 1 root nagios 181907 Mar 21 14:52 check_icmp
-rwxr-xr-x 1 root nagios 146187 Mar 21 14:52 check_ide_smart
-rwxr-xr-x 1 root nagios  15123 Mar 21 14:52 check_ifoperstatus
-rwxr-xr-x 1 root nagios  12600 Mar 21 14:52 check_ifstatus
lrwxrwxrwx 1 root root        9 Mar 21 14:52 check_imap -> check_tcp
-rwsr-xr-x 1 root nagios    972 Mar 21 14:52 check_init_service
-rwxr-xr-x 1 root nagios   6887 Mar 21 14:52 check_ircd
lrwxrwxrwx 1 root root        9 Mar 21 14:52 check_jabber -> check_tcp
[b]-rwxr-x--- 1 root root      156 Apr  5 02:27 check_jmx[/b]
-rwxr-xr-x 1 root nagios 158116 Mar 21 14:52 check_load
-rwxr-xr-x 1 root nagios   5989 Mar 21 14:52 check_log
-rwxr-xr-x 1 root nagios  21480 Mar 21 14:52 check_mailq
-rwxr-xr-x 1 root nagios 144293 Mar 21 14:52 check_mrtg
-rwxr-xr-x 1 root nagios 145346 Mar 21 14:52 check_mrtgtraf
-rwxr-xr-x 1 root nagios 157430 Mar 21 14:52 check_nagios
-rwxr-xr-x 1 root nagios  25602 Mar 21 14:52 check_netstat.pl
lrwxrwxrwx 1 root root        9 Mar 21 14:52 check_nntp -> check_tcp
lrwxrwxrwx 1 root root        9 Mar 21 14:52 check_nntps -> check_tcp
-rwxr-xr-x 1 root nagios  76817 Mar 21 14:52 check_nrpe
-rwxr-xr-x 1 root nagios 179698 Mar 21 14:52 check_nt
-rwxr-xr-x 1 root nagios 184033 Mar 21 14:52 check_ntp
-rwxr-xr-x 1 root nagios 173030 Mar 21 14:52 check_ntp_peer
-rwxr-xr-x 1 root nagios 171283 Mar 21 14:52 check_ntp_time
-rwxr-xr-x 1 root nagios 212263 Mar 21 14:52 check_nwstat
-rwxr-xr-x 1 root nagios   3259 Mar 21 14:52 check_open_files.pl
-rwxr-xr-x 1 root nagios   8779 Mar 21 14:52 check_oracle
-rwxr-xr-x 1 root nagios 160061 Mar 21 14:52 check_overcr
-rwxr-xr-x 1 root nagios 183602 Mar 21 14:52 check_ping
lrwxrwxrwx 1 root root        9 Mar 21 14:52 check_pop -> check_tcp
-rwxr-xr-x 1 root nagios 187009 Mar 21 14:52 check_procs
-rwxr-xr-x 1 root nagios 157487 Mar 21 14:52 check_real
-rwxr-xr-x 1 root nagios   9581 Mar 21 14:52 check_rpc
-rwxr-xr-x 1 root nagios   1453 Mar 21 14:52 check_sensors
-rwxr-xr-x 1 root nagios   2174 Mar 21 14:52 check_services
lrwxrwxrwx 1 root root        9 Mar 21 14:52 check_simap -> check_tcp
-rwxr-xr-x 1 root nagios   7599 Mar 21 14:52 check_sip
-rwxr-xr-x 1 root nagios 247746 Mar 21 14:52 check_smtp
lrwxrwxrwx 1 root root        9 Mar 21 14:52 check_spop -> check_tcp
-rwxr-xr-x 1 root nagios 155315 Mar 21 14:52 check_ssh
lrwxrwxrwx 1 root root        9 Mar 21 14:52 check_ssmtp -> check_tcp
-rwxr-xr-x 1 root nagios 141113 Mar 21 14:52 check_swap
-rwxr-xr-x 1 root nagios 230677 Mar 21 14:52 check_tcp
-rwxr-xr-x 1 root nagios 156465 Mar 21 14:52 check_time
lrwxrwxrwx 1 root root        9 Mar 21 14:52 check_udp -> check_tcp
-rwxr-xr-x 1 root nagios 168469 Mar 21 14:52 check_ups
-rwxr-xr-x 1 root nagios 137995 Mar 21 14:52 check_uptime
-rwxr-xr-x 1 root nagios 136145 Mar 21 14:52 check_users
-rwxr-xr-x 1 root nagios   2936 Mar 21 14:52 check_wave
-rwxr-xr-x 1 root nagios    710 Mar 21 14:52 check_yum
-rwxr-xr-x 1 root nagios   3435 Mar 21 14:52 custom_check_mem
-rwxr-xr-x 1 root nagios    915 Mar 21 14:52 custom_check_procs
-rwxr-x--- 1 root root    13225 Apr  5 02:27 jmxquery.jar
-rwxr-xr-x 1 root nagios   4176 Mar 21 14:52 nagisk.pl
-rwxr-xr-x 1 root nagios 131582 Mar 21 14:52 negate
-rwxr-xr-x 1 root nagios  64594 Mar 21 14:52 send_nsca
-rwxr-xr-x 1 root nagios 125828 Mar 21 14:52 urlize
-rwxr-xr-x 1 root nagios   1913 Mar 21 14:52 utils.pm
-rwxr-xr-x 1 root nagios   2791 Mar 21 14:52 utils.sh
Also
in Java running application server, ( where NRPE is running ) , have configured the /usr/local/nagios/etc/nrpe.cfg
added the : below with # ( hash )

Code: Select all

#command[check_jmx]=/usr/local/nagios/libexec/check_jmx $ARG1$
From NagiosXI server
======================
executed the check_nrpe

Code: Select all

[root@NagiosXI server / libexec]# 
==============================
/usr/local/nagios/libexec/check_nrpe -H <Javaapps-running-server-ip>  -c check_jmx -a '-U
service:jmx:rmi:///jndi/rmi://127.0.0.1:7199/jmxrmi -O java.lang:type=Memory -A
HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192'
I got the below message:-

Code: Select all

NRPE: Command 'check_jmx' not defined
but there is already check_jmx file.

Code: Select all

[root@NagiosXI libexec]# ll check_jm*
-rwxr-x--- 1 root nagios 156 Apr  4 23:53 check_jmx
==================================================================

From JAVA running application server:-

Code: Select all

[root@java-app-srvr libexec]# ./check_jmx -U service:jmx:rmi:///jndi/rmi://java-app-serverIP:8062/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage 
-K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192
I am getting the below messages:-

Code: Select all

JMX CRITICAL Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        java.net.SocketTimeoutException: Read timed out] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://localhost:8062/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        java.net.SocketTimeoutException: Read timed out]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
        at org.nagios.JMXQuery.connect(JMXQuery.java:53)
        at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
==============================================================================================
From Java application server:

Code: Select all

[root@JavaAppServer libexec]# cat /etc/sysconfig/iptables

# Generated by iptables-save v1.4.7 on Tue Mar 21 14:51:37 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [10:2367]
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5667 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5666 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8062 -j ACCEPT
COMMIT
# Completed on Tue Mar 21 14:51:37 2017
===========================================================================


From NagiosXI server : IP TABLES:
===============================================

Code: Select all

[root@NagiosXI server]# cat /etc/sysconfig/iptables
# Generated by iptables-save v1.4.7 on Thu Mar 30 08:34:39 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [3876:899605]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 4440 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8062 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Thu Mar 30 08:34:39 2017
============================================================================

Please advice, how to have the setup for the check_jmx plugin with NagiosXI ?

Thank you,
Nagesh.
Last edited by mcapra on Wed Apr 05, 2017 2:07 pm, edited 1 time in total.
Reason: please use [code] tags for technical output
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Monitor JMX With Nagios XI : JMX CRITICAL Failed to retr

Post by mcapra »

Nags007 wrote: A. Please let me know, how to setup / configure the - remove server running JMX ? is there any download of JMX ? how to check JMX is running ?
JMX has to be enabled via the Java properties of the application at runtime.

It is the responsibility of whoever is maintaining the server to ensure JMX is enabled for the application. We just provide documentation/plugins for making JMX connections to remote JMX servers. We don't have documentation for enabling JMX for every conceivable Java application (WebLogic, Tomcat, JBoss, proprietary software, etc); That would be impractical.

Here's Oracle's documentation regarding JMX:
http://docs.oracle.com/javase/7/docs/te ... agent.html
Nags007 wrote:B. What's the meaning of NagiosXI server with network route to JMX server ? ( what is this JMX server )?
I'm not sure what you mean by this. Could you clarify? JMX is a management interface designed by Oracle to help users monitor their Java applications.
Nags007 wrote:C. Where should the check_jmx plugin to be installed ? ( whether in Java running application server ) or NagiosXI server or any other server ?
You could either install check_jmx on the remote machine and leverage an agent like NRPE to execute it, or you could install check_jmx on the Nagios XI machine and let the Nagios XI machine make the JMX connections directly. Most people let NRPE handle it, since that way you can configure the remote system's JMX server to only listen locally (some organizations prefer this).
Nags007 wrote: From NagiosXI server
======================
executed the check_nrpe

Code: Select all

[root@NagiosXI server / libexec]# 
==============================
/usr/local/nagios/libexec/check_nrpe -H <Javaapps-running-server-ip>  -c check_jmx -a '-U
service:jmx:rmi:///jndi/rmi://127.0.0.1:7199/jmxrmi -O java.lang:type=Memory -A
HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192'
I got the below message:-

Code: Select all

NRPE: Command 'check_jmx' not defined
but there is already check_jmx file.

Code: Select all

[root@NagiosXI libexec]# ll check_jm*
-rwxr-x--- 1 root nagios 156 Apr  4 23:53 check_jmx
From the perspective of NRPE, it is not enough to simply include the plugin. There must also be a command definition in the NRPE configuration file that points to that plugin. The command's name can be whatever you like, as long as that's what you're ultimately using in your check_nrpe argument.
Nags007 wrote: From JAVA running application server:-

Code: Select all

[root@java-app-srvr libexec]# ./check_jmx -U service:jmx:rmi:///jndi/rmi://java-app-serverIP:8062/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage 
-K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192
I am getting the below messages:-

Code: Select all

JMX CRITICAL Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        java.net.SocketTimeoutException: Read timed out] connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://localhost:8062/jmxrmijava.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        java.net.SocketTimeoutException: Read timed out]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
        at org.nagios.JMXQuery.connect(JMXQuery.java:53)
        at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
This error implies that there is not a JMX server running at localhost:8062. The JMX server runs completely separate from your actual application. If you have a web application listening on 8062, you should not bind your JMX server to 8062.

So even if you have a web application that you can access just fine on 8062 via a web browser this is not a JMX server.
Former Nagios employee
https://www.mcapra.com/
Top
Locked

Return to “Nagios XI”