JMX Alerts return (null) values

[madhudeva]

Hi Team ,

We have JMX deamon configured on our recently setup NagiosXi and it would take care of the all "jmx-service" defined checks.
It seems that these alerts tend to return null values at times even while JMX deamon is still runing . And gets remediated only after we restart the JMX deamon on the host.
Could there be any reason why the JMX deamon needs to be restarted so frequently ?
Is there any compatibility factor when it comes to JMX configured on Linux servers doing checks for Solaris remote servers ?
We also notice that the load is always on the higher side whenever this has occured.
Any suggestions are welcome ...

Another thing witnessed on the /var/log/messages is that we needed to repair some of the database recently as it seems to have pointed out the following lately :

Nov 28 07:23:13 dkd1mr7954 ndo2db: mysql_error: 'Table './nagios/nagios_notifications' is marked as crashed and last (automatic?) repair failed'
Nov 25 14:11:15 dkd1mr7954 ndo2db: mysql_error: 'Table './nagios/nagios_externalcommands' is marked as crashed and last (automatic?) repair failed'

Is there some other bigger problem that needs to be dealt with as we have been seeing these mySQL table crashes ?

We would appreciate your help in providing your more clues in fidning a solution to these issues

Thanks
Madhu

[slansing]

Can you show us an example of one of the services that returns null? Navigate to CCM > Services > "jmx service that is returning (null)" > Click the diskette icon and copy + paste the output here in code-wraps. Do the same for it's host, and the command definition that the service is using. To repair your database follow this:

http://assets.nagios.com/downloads/nagi ... tabase.pdf

[clive.dias]

Thanks for the reply. I will provide an example soon ...

Any reason why we need the database to be repaired so frequently ... ?

[abrist]

DB damage is almost always caused by improper shutdowns of mysqld and related services or the server itself. Does your XI box get rebooted frequently? Have you created any automation that restarts mysqld?

[clive.dias]

Hi Team

Assuming I am taking about service "JVM threads" , with the definition in services.cfg as follows

define service {
host_name <HostA>
service_description JVM threads
use generic-service
check_command remote_check_via_<HostB>!check_nrpe_dmz!<Host>!check_jmx!U service:jmx:rmi:///jndi/rmi://localhost:8005/jmxrmi!O java.lang:type=Threading!A ThreadCount!G VALUE!w 100 c 150
check_interval 1
register 1
}

And Host definition as follows as in hosts.cfg

define host {
host_name <HostA>
use defaults
alias <HostA>
address <HostA>
hostgroups unix-servers-dmz
check_command remote_check_via_<HostB>!check_host_alive!<HostA>!!!!!!
register 1
}

and command definition as in commands.cfg is as follows :

define command {
command_name remote_check_via_<HostB>
command_line $USER1$/check_nrpe -n -H <HostB> -t 60 -c $ARG1$ -a $ARG2$ $ARG3$ $ARG4$ $ARG5$ $ARG6$ $ARG7$ $ARG8$ $ARG9$ $ARG10$ $ARG11$ $ARG12$ $ARG13$ $ARG14$ $ARG15$ $ARG16$
}

Please let me know if you need any more details ...

Thanks
Clive

[abrist]

From the DMZ host, can you run the check manually?

Code: Select all

./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:8005/jmxrmi -O java.lang:type=Threading -A ThreadCount -G VALUE -w 100 -c 150

I also noticed your check command is missing the hyphens:

check_command remote_check_via_<HostB>!check_nrpe_dmz!<Host>!check_jmx!U service:jmx:rmi:///jndi/rmi://localhost:8005/jmxrmi!O java.lang:type=Threading!A ThreadCount!G VALUE!w 100 c 150

Should be:

Code: Select all

check_command remote_check_via_<HostB>!check_nrpe_dmz!<Host>!check_jmx!-U service:jmx:rmi:///jndi/rmi://localhost:8005/jmxrmi!-O java.lang:type=Threading!-A ThreadCount!-G VALUE!-w 100- c 150

[clive.dias]

Thanks for the response.

Yes, I can issue the command fine on the DMZ server

(PROD)emarkets@<Host>:/me01/apps/emarkets/nagios/libexec$ ./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:8005/jmxrmi -O java.lang:type=Threading -A ThreadCount -G VALUE -w 100 -c 150
JMX OK ThreadCount=80

Please note that as mentioned JMX is stable most of the time during the day but occasionaly it returns these null values

I dont its has to do with the hyphen.

One thing I would like to mention that the load average of this Nagios Xi server is pretty high whenever we come across case that it returns these kind of nul values. And that the high load could be caused by too many of these JMX checks configured ?

And out of curiousity with our Linux Nagios Xi communicating to Solaris server(s) for these JMX checks ? Would there be any compatibility issues ?

Could it be that we are using an older version of JMX too ?

[abrist]

And out of curiousity with our Linux Nagios Xi communicating to Solaris server(s) for these JMX checks ? Would there be any compatibility issues ?

Could it be that we are using an older version of JMX too ?

I don't believe so. If there were compatibility problems, you would always have issues with the checks.

If the load is high, you may run into timeout issues. Additionally, do you notice the null returns around the same time every day? If so, are your java application servers backing up, in downtime, or doing something load intensive at that time?

[clive.dias]

Thanks for the suggestions. I will check when we get these issues again as for now, it has been quite stable.

[slansing]

Great, just let us know. We're going to close this thread for now, if it recurs start a new one and just link this thread's URL.