Monitoring Jmx remotely without NRPE agent

[tgriep]

I don't know how the authentication is setup for your JMX server but if authentication is required to get any sort of information at all, whether the plugin is run remotely from the Nagios server or locally on the JMX server using NRPE, it would not work unless a plugin is found that supports authentication.

If SSH is available, you could use that instead of NRPE.
https://assets.nagios.com/downloads/nag ... ng_SSH.pdf

[JakeHatMacys]

I don't know how the authentication is setup for your JMX server but if authentication is required to get any sort of information at all, whether the plugin is run remotely from the Nagios server or locally on the JMX server using NRPE, it would not work unless a plugin is found that supports authentication.

If SSH is available, you could use that instead of NRPE.
https://assets.nagios.com/downloads/nag ... ng_SSH.pdf

Yeah authentication is set up, I think there's a few standards when it comes to JVMs but i'm no expert.

I'm connecting currently with Sitescope without issue. The problem is when is I attempted with that plugin after fixing the ^M format issue I think it's expecting a certain format and not what we currently have configured:

service:jmx:remoting-jmx:// is what my jboss team has configured and getting them to change anything is like throwing a request down a bottomless well. I was given a jboss-cli-client.jar file to use with sitescope and uncertain how that would come into play with these Nagios plugins...

Which is why I'd like to use the NRPE plugin if it's officially supported and perhaps get a webex to see if we can get remoting enabled on it. Do you think this is something they'd humor???

[mcapra]

I think there's a few standards when it comes to JVMs but i'm no expert.

The RMI is Java's native "I want to remotely look at this application" connector. That's what most forks of check_jmx are expecting; That you are using Java's native RMI for connections.

JBoss's remoting-jmx functions in a similar-but-different way. WebLogic also has it's own JMX connector that is, again, similar-but-different to Java's native RMI. The nice thing about this is that it makes the JMX requests you send much cleaner for a given application server. The bad thing is these application server specific connectors are relatively new to the scene so documentation/implementation is rather thin at the moment.

I might in the distant future upgrade my AS6 setup to AS7 and do some exploring of how to best leverage remoting-jmx in a Nagios plugin, but to be quite honest that's pretty "on the back burner" for me right now Enabling the native RMI as part of my applications' deployment process is good enough for my purposes.

There's been some limited discussion about an official "check Java" plugin/suite being added to the nagios-plugins package if you'd like to chime in:
https://github.com/nagios-plugins/nagio ... issues/279

[SteveBeauchemin]

This may be harder than the Nagios Wizard way, but it works well for me. There is always more than one way to do things.

I am not sure if this will help, but I have been monitoring Tomcat and other Java based web applications using jmx4perl and using jolokia on an intermediate server. The remote servers under test get no agent, they just need to allow JMX to listen on its own port.

These URL may help get you there.
https://labs.consol.de/jmx4perl/
https://labs.consol.de/jolokia/index.html

Here is some syntax for the tests. This is non trivial stuff. Think of Java as a separate OS and you'll be in the correct mind space.

This is my command definition - I have only one.

Code: Select all

$USER1$/check_jmx4perl --url http://$_HOST_JMX_JOLOKIA$/jolokia --target service:jmx:rmi:///jndi/rmi://$HOSTADDRESS$:$_HOST_JMX_TARGET_PORT$/jmxrmi --target-user $_HOST_JMX_TARGET_USER$ --target-password $_HOST_JMX_TARGET_PASS$ --mbean $ARG1$ --attribute $ARG2$ $ARG3$ $ARG4$ $ARG5$ $ARG6$ $ARG7$ $ARG8$

--url is the intermediate system where I have a tomcat / jolokia setup.
I have jolokia installed and tomcat running on my Nagios XI core host, and on each mod_gearman host
--target is the actual host you want to test
--target-user is the username with permissions to use JMX
--target-password - take a guess...
--mbean is the specific place in the java stack that you want to get data from
--attribute is the exact item that you want a response about

I use 'Free Variables' in the host or host template to provide user, password, port, and whatever else is common in all the tests.

Code: Select all

__JMX_TARGET_USER	jmx_monitor	
__JMX_TARGET_PASS	12345678	
__JMX_JOLOKIA		127.0.0.1:8080	
__JMX_TARGET_PORT	18040

Here are some service test examples.

Code: Select all

JMX_CPU_Load

$ARG1$ java.lang:type=OperatingSystem
$ARG2$ SystemCpuLoad
$ARG3$ --warning 1.25
$ARG4$ --critical 1.50

Code: Select all

JMX_Heap_Memory_Usage

$ARG1$ java.lang:type=Memory
$ARG2$ HeapMemoryUsage 
$ARG3$ --base java.lang:type=Memory/HeapMemoryUsage/max
$ARG4$ --path used
$ARG5$ --warning 90
$ARG6$ --critical 95
$ARG7$ --unit B

Code: Select all

JMX_Hostname
( trust me - add this - if you miss on a name or port you'll be glad this is there )
( because it is possible to have multiple tomcat and multiple jmx on a single host )

$ARG1$ java.lang:type=Runtime
$ARG2$ Name

Code: Select all

JMX_Free_Swap_Space_size

$ARG1$ java.lang:type=OperatingSystem
$ARG2$ FreeSwapSpaceSize
$ARG3$ --warning 10000000000:
$ARG4$ --critical 9000000000:
$ARG5$ --unit B

Code: Select all

JMX_Thread_Count

$ARG1$ java.lang:type=Threading
$ARG2$ ThreadCount
$ARG3$ --warning 900
$ARG4$ --critical 1000

Code: Select all

JMX_Physical_Memory_Free

$ARG1$ java.lang:type=OperatingSystem
$ARG2$ FreePhysicalMemorySize
$ARG3$ --warning 300000000:
$ARG4$ --critical 250000000:
$ARG5$ --unit B

Code: Select all

JMX_Uptime

$ARG1$ java.lang:type=Runtime
$ARG2$ Uptime
$ARG3$ --warning 300:
$ARG4$ --critical 300:

Code: Select all

JMX_Virtual_Memory_Size

$ARG1$ java.lang:type=OperatingSystem
$ARG2$ CommittedVirtualMemorySize
$ARG3$ --warning 107000000000
$ARG4$ --critical 108000000000
$ARG5$ --unit B

Code: Select all

JMX_Garbage_Collection
(beware - can be different system by system)

$ARG1$ "java.lang:name=PS MarkSweep,type=GarbageCollector"
$ARG2$ LastGcInfo 
$ARG3$ --path used
$ARG4$ --warning 1
$ARG5$ --critical 2

It really helps to be able to see all the MBeans you can query. To get a list you can run this from the command line aimed at a host. Each host will be different.

Code: Select all

jmx4perl http://[jolokia proxy host]:[jolokia port]/jolokia \
--target service:jmx:rmi:///jndi/rmi://[remote host IP/name]:[remote JMX port number]/jmxrmi \
--target-user [jmx user] \
--target-password [jmx password] \
list > /tmp/JMX-MBeans-export.txt

The installation of the jmx4perl can be simple, but can be tricky. You have to pay attention that all the perl dependencies are satisfied.
These are the very basic steps I use. but the install takes a very long time and can have many failures that you need to figure out.

Code: Select all

systemctl enable tomcat
systemctl start tomcat

Get JMX for Perl installed

Code: Select all

Perl -MCPAN -e shell
install JMX::Jmx4Perl

Basically answer yes to all questions. This is interactive.
Then, make a link for libexec to find it.
cd /usr/local/nagios/libexec
ln -s /usr/local/bin/check_jmx4perl check_jmx4perl

Ffter the install you should be able to download the jolokia war file by entering the command:

Code: Select all

jolokia

If that fails, use the links below.

The installation of jolokia is easy. Put the war file in the right place and Tomcat extracts it for you.
To acquire jolokia.war look in these places. In a new setup I would get the latest version.
https://jolokia.org/download.html
https://jolokia.org/agent/war.html
https://jolokia.org/client/perl.html
For me, tomcat is here:

Code: Select all

cp jolokia.war /var/lib/tomcat/webapps/

Syntax can be a challenge. Make a MBeans list, read it, play with tests from the command line until you get them working for you.

Good Luck. I hope this gets you jump started.

Steve B

[dwhitfield]

@JakeHatMacys, was Steve's answer useful for you? Do you have additional questions?

Thanks @SteveBeauchemin!

[JakeHatMacys]

Wow yeah I'll take a look @SteveBeauchemin

Thanks for the thorough reply, sorry I've been swamped with other work and haven't been able to check back. In the interim i'm loading our production jvm's on sitescope and hope to get this worked out for everything later as they want Dev metrics for testing and there's no way it'll be economical to do everything in Sitescope...

[dwhitfield]

@JakeHatMacys, please let us know if you have any additional questions.