Hello,
I'm using check_jvm to monitor Apache Tomcat using JMX but the data I get from the GlobalRequestProcessor object are not correctly calculated from the values I can see using JConsole.
RequestsPerMinute check is always returning erroneous and even sometimes negative value and the other checks returns 0 most of the time.
Requests Per Minute
Bytes Per Minute
Bytes Per Request
Errors Per Minute
Processing Time Per Request
I've checked the plugin on different Tomcat installation and I always have the same incorrect behavior.
Here is an example of the command I am using:
check command: check_xi_java_as
$ARG1$: -s service:jmx:rmi:///jndi/rmi://xxxx:xxxx/jmxrmi -u 'xxxx' -p 'xxxx' -t tomcat -C 'RequestsPerMinute:http-apr-8000' -w '10' -c '100'
I have configured the service manually because adding the GlobalRequestProcessor checks does not work using the Wizard.
check_jvm returning incorrect values
Re: check_jvm returning incorrect values
Can you run the following on the Nagios server and post the output so we can view all of the version numbers?
Also, login to the XI GUI and go to the Admin > Manage Config Wizards, check for updates and update the wizard to see if that helps the issue.
Code: Select all
/usr/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar -versionBe sure to check out our Knowledgebase for helpful articles and solutions!
Re: check_jvm returning incorrect values
Here is a list of the version of Java, check_jvm.rar and Nagios XI
/usr/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
/usr/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -version
check_jvm.jar, version 1.0.0, (c) 2017-2018 Nagios Enterprises, LLC
even if check_jvm.jar is really from the 1.0.2 release archive.
Version shown by the GUI:
Apache Tomcat
Monitor Apache Tomcat JVMs through JMX (uses NCPA or requires Java)
1.0.2 2019-07-31 Nagios Enterprises, LLC. Copyright © 2018 Nagios Enterprises, LLC.
Nagios XI version:
Your Nagios XI installation is up to date.
Latest Available Version: 5.6.5
Installed Version: 5.6.5
Here is an example of the data that is returned by the pluggin, I think it returns negative value after a reset of the requestCount. Most of the time it just return 0 even if there are requests and the counter is incremented. sometimes it return high values even if there is not a lot of activity. It just seems completely random.
Another problem is it returns critical state for decimal values like 1.3 when the threshold is a lot higher than that. In the example below, the negative value is obviously lower than my threshold
/usr/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
/usr/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -version
check_jvm.jar, version 1.0.0, (c) 2017-2018 Nagios Enterprises, LLC
even if check_jvm.jar is really from the 1.0.2 release archive.
Version shown by the GUI:
Apache Tomcat
Monitor Apache Tomcat JVMs through JMX (uses NCPA or requires Java)
1.0.2 2019-07-31 Nagios Enterprises, LLC. Copyright © 2018 Nagios Enterprises, LLC.
Nagios XI version:
Your Nagios XI installation is up to date.
Latest Available Version: 5.6.5
Installed Version: 5.6.5
Here is an example of the data that is returned by the pluggin, I think it returns negative value after a reset of the requestCount. Most of the time it just return 0 even if there are requests and the counter is incremented. sometimes it return high values even if there is not a lot of activity. It just seems completely random.
Another problem is it returns critical state for decimal values like 1.3 when the threshold is a lot higher than that. In the example below, the negative value is obviously lower than my threshold
-
swolf
Re: check_jvm returning incorrect values
Hi @unifr,
I can confirm that you're on the latest version. The plugin didn't get any updates in the last two wizard releases, so that's why the versions are mismatched.
You're right about the plugin returning negative values when the counters are reset. It looks like this simply wasn't accounted for when the plugin was first written. I'll look into fixing that, in the meantime you should be able to specify thresholds like
to prevent negative numbers from triggering an alert.
I'm not sure why the numbers are being miscalculated during 'normal' operation, though. How often is the request counter being reset? If it's more than every 5 minutes, that could be one of the causes (assuming the plugin output is too low, rather than too high, you could see the counter reset, but still increase enough for a positive value on the next check).
Separate from either of those, would you be able to share the Object Names you're using to get the correct numbers in JConsole? You should be able to find them by hovering over the entry in the MBeans tab. If the values are sensitive, feel free to PM me the information. If you do that, please also make a post on this thread so that it shows up in our support team's dashboard.
Regards,
Sebastian Wolf
I can confirm that you're on the latest version. The plugin didn't get any updates in the last two wizard releases, so that's why the versions are mismatched.
You're right about the plugin returning negative values when the counters are reset. It looks like this simply wasn't accounted for when the plugin was first written. I'll look into fixing that, in the meantime you should be able to specify thresholds like
Code: Select all
-w '~:10' -c '~:100'I'm not sure why the numbers are being miscalculated during 'normal' operation, though. How often is the request counter being reset? If it's more than every 5 minutes, that could be one of the causes (assuming the plugin output is too low, rather than too high, you could see the counter reset, but still increase enough for a positive value on the next check).
Separate from either of those, would you be able to share the Object Names you're using to get the correct numbers in JConsole? You should be able to find them by hovering over the entry in the MBeans tab. If the values are sensitive, feel free to PM me the information. If you do that, please also make a post on this thread so that it shows up in our support team's dashboard.
Regards,
Sebastian Wolf
Re: check_jvm returning incorrect values
Hello,
Thanks for your reply. during normal operation there is no reset of the counters, I was only manually resetting it during my tests.
I'm looking the requestCount attribute of Catalina:type=GlobalRequestProcessor,name="ajp-apr-8111" object in JConsole, which I believe is used in the calculation of the RequestsPerMinute check.
Here is the command I use for check_jvm:
/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://****:****/jmxrmi -u '****' -p '****' -t tomcat -C 'RequestsPerMinute:ajp-apr-8111' -w '~:10' -c '~:100'
Here is one simple test:
@Time:
jconsole value:
requestCount: 5736356
check_jvm result:
Requests Per Minute returned OK with 0 requests | tomcat_requests_minute=0;~:10;~:100;;
@Time+5min:
jconsole value:
requestCount: 5738446
check_jvm result:
Requests Per Minute returned OK with 0 requests | tomcat_requests_minute=0;~:10;~:100;;
check_jvm returned 0 request per minute where it should have returned 6.96 requests per minutes.
Is my command correct? Where does check_jvm get the time value used in the calculation of the requests per minute?
Thanks for your reply. during normal operation there is no reset of the counters, I was only manually resetting it during my tests.
I'm looking the requestCount attribute of Catalina:type=GlobalRequestProcessor,name="ajp-apr-8111" object in JConsole, which I believe is used in the calculation of the RequestsPerMinute check.
Here is the command I use for check_jvm:
/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://****:****/jmxrmi -u '****' -p '****' -t tomcat -C 'RequestsPerMinute:ajp-apr-8111' -w '~:10' -c '~:100'
Here is one simple test:
@Time:
jconsole value:
requestCount: 5736356
check_jvm result:
Requests Per Minute returned OK with 0 requests | tomcat_requests_minute=0;~:10;~:100;;
@Time+5min:
jconsole value:
requestCount: 5738446
check_jvm result:
Requests Per Minute returned OK with 0 requests | tomcat_requests_minute=0;~:10;~:100;;
check_jvm returned 0 request per minute where it should have returned 6.96 requests per minutes.
Is my command correct? Where does check_jvm get the time value used in the calculation of the requests per minute?
-
swolf
Re: check_jvm returning incorrect values
I don't see anything wrong with your check_command.
In your /usr/local/nagios/libexec folder there should be a file called ".tomcat" which stores counter information from the last plugin run. Can you run and show the output here?
As far as time calculation goes, the plugin is just running
to get the current timestamp in minutes. It records this value alongside the requests counter in the data file.
In your /usr/local/nagios/libexec folder there should be a file called ".tomcat" which stores counter information from the last plugin run. Can you run
Code: Select all
cat /usr/local/nagios/libexec/.tomcatAs far as time calculation goes, the plugin is just running
Code: Select all
new Date().getTime()/60000.0Re: check_jvm returning incorrect values
Hello,
Thank you for your reply. I've noticed my .tomcat file contained the information of a the first server I used to check the command. It seems the file can only contain the value of a single server.
I have delete the .tomcat file and executed the following checks:
1. Check command: t
/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi -u '*' -p '*' -t tomcat -C 'RequestsPerMinute:http-apr-8080' -w '~:10' -c '~:100'
Result
Requests Per Minute returned OK with 0 requests | tomcat_requests_minute=0;~:10;~:100;;
Content of .tomcat
service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi ||| Requests Per Minute ||| 799 ||| 2.6106090015866667E7
2. Check command: t+1min
/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi -u '*' -p '*' -t tomcat -C 'RequestsPerMinute:http-apr-8080' -w '~:10' -c '~:100'
Result
Requests Per Minute returned OK with 7.04 requests | tomcat_requests_minute=7.038005242404108;~:10;~:100;;
Content of .tomcat
service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi ||| Requests Per Minute ||| 806 ||| 2.6106091010466665E7
The results of the check executed on srv1 seems to be match the values i see in jconsole. it works.
Now I execute the check on srv2:
3. Check command: t+2min
/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://srv2:5000/jmxrmi -u '*' -p '*' -t tomcat -C 'RequestsPerMinute:http-apr-8081 -w '~:10' -c '~:100'
Result
Requests Per Minute returned OK with 0 requests | tomcat_requests_minute=0;~:10;~:100;;
Content of .tomcat
service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi ||| Requests Per Minute ||| 806 ||| 2.6106091010466665E7
4. Check command: t+3min
/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://srv2:5000/jmxrmi -u '*' -p '*' -t tomcat -C 'RequestsPerMinute:http-apr-8081' -w '~:10' -c '~:100'
Result
Requests Per Minute returned OK with 0 requests | tomcat_requests_minute=0;~:10;~:100;;
Content of .tomcat
service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi ||| Requests Per Minute ||| 806 ||| 2.6106091010466665E7
On the same server I also have different global request processor objects like http-apr-8080 and http-apr-8081, since .tomcat command does not contain this information. I also get wrong values executing the checks on the same server but using the different objects.
Thank you for your reply. I've noticed my .tomcat file contained the information of a the first server I used to check the command. It seems the file can only contain the value of a single server.
I have delete the .tomcat file and executed the following checks:
1. Check command: t
/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi -u '*' -p '*' -t tomcat -C 'RequestsPerMinute:http-apr-8080' -w '~:10' -c '~:100'
Result
Requests Per Minute returned OK with 0 requests | tomcat_requests_minute=0;~:10;~:100;;
Content of .tomcat
service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi ||| Requests Per Minute ||| 799 ||| 2.6106090015866667E7
2. Check command: t+1min
/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi -u '*' -p '*' -t tomcat -C 'RequestsPerMinute:http-apr-8080' -w '~:10' -c '~:100'
Result
Requests Per Minute returned OK with 7.04 requests | tomcat_requests_minute=7.038005242404108;~:10;~:100;;
Content of .tomcat
service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi ||| Requests Per Minute ||| 806 ||| 2.6106091010466665E7
The results of the check executed on srv1 seems to be match the values i see in jconsole. it works.
Now I execute the check on srv2:
3. Check command: t+2min
/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://srv2:5000/jmxrmi -u '*' -p '*' -t tomcat -C 'RequestsPerMinute:http-apr-8081 -w '~:10' -c '~:100'
Result
Requests Per Minute returned OK with 0 requests | tomcat_requests_minute=0;~:10;~:100;;
Content of .tomcat
service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi ||| Requests Per Minute ||| 806 ||| 2.6106091010466665E7
4. Check command: t+3min
/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://srv2:5000/jmxrmi -u '*' -p '*' -t tomcat -C 'RequestsPerMinute:http-apr-8081' -w '~:10' -c '~:100'
Result
Requests Per Minute returned OK with 0 requests | tomcat_requests_minute=0;~:10;~:100;;
Content of .tomcat
service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi ||| Requests Per Minute ||| 806 ||| 2.6106091010466665E7
On the same server I also have different global request processor objects like http-apr-8080 and http-apr-8081, since .tomcat command does not contain this information. I also get wrong values executing the checks on the same server but using the different objects.
-
swolf
Re: check_jvm returning incorrect values
Hi,
Thanks for providing this information.
If the plugin is wiping out data for other servers, that's not intended behavior. I'll see what I can do about fixing this today.
EDIT: PM'd a possible fix to @unifr
Thanks for providing this information.
If the plugin is wiping out data for other servers, that's not intended behavior. I'll see what I can do about fixing this today.
EDIT: PM'd a possible fix to @unifr
Re: check_jvm returning incorrect values
Hello,
Thank you very much for the updated version.
I've test it a little bit and it seems to work fine when checking different servers.
The only problem I found yet is it returns an error using the ProcessingTimePerRequest check.
For exemple the following command:
/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://xxx:5000/jmxrmi -u '*' -p '*' -t tomcat -C 'ProcessingTimePerRequest:http-apr-8000' -w '~:10' -c '~:100'
Returns this error:
Thank you very much for the updated version.
I've test it a little bit and it seems to work fine when checking different servers.
The only problem I found yet is it returns an error using the ProcessingTimePerRequest check.
For exemple the following command:
/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://xxx:5000/jmxrmi -u '*' -p '*' -t tomcat -C 'ProcessingTimePerRequest:http-apr-8000' -w '~:10' -c '~:100'
Returns this error:
Code: Select all
Exception in thread "main" javax.management.RuntimeOperationsException: Exception occurred trying to invoke the getter on the MBean
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:624)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at com.sun.jmx.remote.security.MBeanServerAccessController.getAttribute(MBeanServerAccessController.java:320)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)
at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:639)
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown Source)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:903)
at JVMCheck.getValues(JVMCheck.java:112)
at GenericASCheck.runChecks(GenericASCheck.java:115)
at GenericASCheck.main(GenericASCheck.java:292)
Caused by: java.lang.IllegalArgumentException: Object name cannot be null
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:624)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at com.sun.jmx.remote.security.MBeanServerAccessController.getAttribute(MBeanServerAccessController.java:320)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)
at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:639)
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
-
swolf
Re: check_jvm returning incorrect values
Thanks for following up! When I made the changes you requested, I also noticed a bug in how the "ProcessingTimePerRequest" worked. It looks like my fix wasn't complete.
I've PM'd you another update to check_jvm.jar. Let me know how it works for you.
I've PM'd you another update to check_jvm.jar. Let me know how it works for you.