check_jvm returning incorrect values

[unifr]

Hello,

Thank you very much for the updated plugin, I have been collecting data for several days on several machines and it is working very well including using the ProcessingTimePerRequest check.

The only issue is it does not create performance graph, is it the correct behavior?

Is is an example of the performance data collected:

tomcat_requests_minute=70.83777608725799;~:1000;~:10000;; tomcat_bytes_minute=6260267.876374784B;~:100000000;~:1000000000;; tomcat_bytes_request=88374.7094017094B;~:100000000;~:1000000000;; tomcat_errors_minute=0.0;~:10:;~:100;; tomcat_proctime_request=84.14529914529915ms;~:10000;~:100000;;

[scottwilkerson]

If this was not working properly before it is very likely that the original RRD for the service was created with the incorrect amount of datasources.

The best bet would be to try deleting the RRD from the following location replacing HOSTNAME/SERVICE_DESC

Code: Select all

/usr/local/nagios/share/perfdata/HOSTNAME/SERVICE_DESC.rrd

This should get the rrd re-generated and performance data should populate within 10-15 minutes

[unifr]

Hello,

Unfortunately I haven't been able to check the performance data issue because the checks suddenly started to return the following error (same errors on all servers):

(No output on stdout) stderr: Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
at CheckParameters.readLastValueWriteNewValue(CheckParameters.java:250)
at CheckParameters.getCheckValue(CheckParameters.java:142)
at JVMCheck.getValues(JVMCheck.java:130)
at GenericASCheck.runChecks(GenericASCheck.java:115)
at GenericASCheck.main(GenericASCheck.java:292)

When I execute the check manually from the linux console (logged as nagios user), it still works fine:

/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi -u '****' -p '****' -t tomcat -C 'RequestsPerMinute:ajp-apr-8000' -w 10 -c 100

If edit the service and perform a "Run Check Command" action form the interface it also still works fine:

/usr/lib/jvm/jre/bin/java -Djava.class.path=:/usr/local/nagios/libexec/check_jvm.jar GenericASCheck -s service:jmx:rmi:///jndi/rmi://srv1:$USER50$/jmxrmi -u '$USER51$' -p '$USER52$' -t tomcat -C 'RequestsPerMinute:ajp-apr-8000' -w 10 -c 100
Requests Per Minute (ajp-apr-8111) returned OK with 0.0 requests | tomcat_requests_minute=0.0;10;100;;

I've delete the .tomcat file and the content of /usr/local/nagios/share/perfdata/srv1 but the error is still here. The check worked perfectly for about one week before the error appeared at the same time on all hosts assigned to this service check.

[swolf]

Hi @unifr,

There are a couple of things we can check here.

First, who owns the .tomcat file?

Code: Select all

ls -la /usr/local/nagios/libexec/.tomcat

If the owner isn't set to nagios:nagios, that might be your issue:

Code: Select all

chown nagios:nagios /usr/local/nagios/libexec/.tomcat

This can happen if you delete the .tomcat file and then run the plugin from the command line before letting nagios run the commands.

Second, if the permissions aren't the cause, please let me know the contents of the .tomcat file:

Code: Select all

cat /usr/local/nagios/libexec/.tomcat

If this information is sensitive, feel free to send it to me or another technician over private message.

[unifr]

Hello,

The owner of the .tomcat file is nagios

The content of the file is:
service:jmx:rmi:///jndi/rmi://srv1:5000/jmxrmi ||| Requests Per Minute (ajp-apr-8000) ||| 0 ||| 2.6128262316866666E7

I have tried to delete the file, the check still get the same error (No output on stdout) stderr: Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1

when I execute the exact same command with the console under the nagios user, the file .tomcat is created and the check return the correct output.

in a test command, I put the same command I can execute from the console (without macro and arguments) and I get the java.lang.ArrayIndexOutOfBoundsException error.

I don't undersand why executing the command from the console using the nagios user and executing the same command as a check can give different results.

[mbellerue]

when I execute the exact same command with the console under the nagios user, the file .tomcat is created and the check return the correct output.

How do you become the nagios user? Do you ssh in directly as the nagios user, or do you su to nagios? If you su to nagios, are you using su - nagios or just su nagios ?

[unifr]

I found the issue with the .tomcat file in /tmp

-rw-r--r-- 1 nagios nagios 9041 Sep 2 21:20 .tomcat

The date of the file is matching the time the checks started to fails. No one was working on the server at this time.

the last line of the .tomcat file was :
7272

the checks are creating the data files in /tmp/ and not in /usr/local/nagios/libexec/ is this the correct behavior?

It would be nice if check_jvm.jar would be able to reset the file if it becomes unreadable.

[unifr]

If this was not working properly before it is very likely that the original RRD for the service was created with the incorrect amount of datasources.

The best bet would be to try deleting the RRD from the following location replacing HOSTNAME/SERVICE_DESC

Code: Select all

/usr/local/nagios/share/perfdata/HOSTNAME/SERVICE_DESC.rrd

This should get the rrd re-generated and performance data should populate within 10-15 minutes

Deleting the rrd solved the graph issue. Thank you for your help.

[mbellerue]

Deleting the rrd solved the graph issue. Thank you for your help.

Just a quick clarification. Did this solve the entire issue, or are there still unresolved problems?

[unifr]

Hello,
All the issue were solved by updated plugin provided by swolf. Thanks for your support.
Best regards