Page 1 of 4

NRPE: UNKNOWN Can't connect to the JVM

Posted: Thu Nov 09, 2017 11:26 am
by pmoradiya
Hi,

I have found seemingly inconsistent behavior of NRPE (v.3.2.1). I am running check_jvm plugin on the monitored hosts. All hosts have NRPE installed and configured exactly the same way. However, NRPE command on couple of hosts is not able to connect to JVM. Here is the info on config:

nrpe.cfg

Code: Select all

command[check_tomcat_threads]=/usr/bin/sudo -u tomcat8 /usr/local/nagios/libexec/check_jvm -n org.apache.catalina.startup.Bootstrap -p threads -w 195 -c 225
sudoers file

Code: Select all

nagios ALL=(ALL) NOPASSWD: /usr/local/nagios/libexec/
check_nrpe works as expected for other commands but not for check_jvm plugin

Code: Select all

root@server1:/usr/local/nagios/libexec# ./check_nrpe -H localhost -c check_load
OK - load average: 0.00, 0.00, 0.00|load1=0.000;0.150;0.300;0; load5=0.000;0.100;0.250;0; load15=0.000;0.050;0.200;0;
root@server1:/usr/local/nagios/libexec# ./check_nrpe -H localhost -c check_tomcat8
PROCS OK: 1 process with UID = 112 (tomcat8) | procs=1;;1:5;0;
root@server1:/usr/local/nagios/libexec# ./check_nrpe -H localhost -c check_tomcat_threads
UNKNOWN Can't connect to the JVM:
If the command in nrpe.cfg is run on the shell, it works fine both as root user and nagios user. Hence, no issue with check_jvm plugin too.

Code: Select all

root@server1:/usr/local/nagios/libexec# /usr/bin/sudo -u tomcat8 /usr/local/nagios/libexec/check_jvm -n org.apache.catalina.startup.Bootstrap -p threads -w 195 -c 225
OK 37 |threads=37;;;
I don't know what I am missing. I really appreciate your assistance with this.

Thanks.

Re: NRPE: UNKNOWN Can't connect to the JVM

Posted: Thu Nov 09, 2017 11:33 am
by pmoradiya
Additional information from the debug log at /usr/local/nagios/var/nrpe.log
Showing WARNING as my_system() seteuid(0):Operation not permitted.

Code: Select all

[1510171606] Connection from 127.0.0.1 port 14494
[1510171606] is_an_allowed_host (AF_INET): is host >127.0.0.1< an allowed host >127.0.0.1<
[1510171606] is_an_allowed_host (AF_INET): host is in allowed host list!
[1510171606] Host address is in allowed_hosts
[1510171606] Host 127.0.0.1 is asking for command 'check_tomcat_threads' to be run...
[1510171606] Running command: /usr/bin/sudo -u tomcat8 /usr/local/nagios/libexec/check_jvm -n org.apache.catalina.startup.Bootstrap -p threads -w 195 -c 225
[1510171606] WARNING: my_system() seteuid(0): Operation not permitted
[1510171606] Command completed with return code 3 and output: UNKNOWN Can't connect to the JVM:
[1510171606] Return Code: 3, Output: UNKNOWN Can't connect to the JVM:
[1510171606] Connection from 127.0.0.1 closed.

Re: NRPE: UNKNOWN Can't connect to the JVM

Posted: Thu Nov 09, 2017 12:47 pm
by mcapra
Have you checked the permissions of JvmInspector.jar and validated that the tomcat8 user is able to execute it? That would be my first thought.

Re: NRPE: UNKNOWN Can't connect to the JVM

Posted: Thu Nov 09, 2017 12:54 pm
by pmoradiya
mcapra wrote:Have you checked the permissions of JvmInspector.jar and validated that the tomcat8 user is able to execute it? That would be my first thought.
Yes. It has the same permission as on the other server where it is working.

Code: Select all

root@server2:/usr/local/bin# ls -l
-rw-r--r-- 1 root root 4714063 Nov  3 17:02 JvmInspector.jar

Code: Select all

root@server1:/usr/local/bin# ls -lart
-rw-r--r--  1 root root 4714063 Nov  9 15:32 JvmInspector.jar

Re: NRPE: UNKNOWN Can't connect to the JVM

Posted: Thu Nov 09, 2017 1:49 pm
by npolovenko
Hello, @pmoradiya.
Please make the following change in your sudoers file:
Change:

Code: Select all

nagios ALL=(ALL) NOPASSWD: /usr/local/nagios/libexec/
To:

Code: Select all

nagios ALL=(ALL) NOPASSWD: /usr/local/nagios/libexec/check_jvm
Let us know if that fixes your issue.

Re: NRPE: UNKNOWN Can't connect to the JVM

Posted: Thu Nov 09, 2017 3:04 pm
by pmoradiya
npolovenko wrote:Hello, @pmoradiya.
Please make the following change in your sudoers file:
Change:

Code: Select all

nagios ALL=(ALL) NOPASSWD: /usr/local/nagios/libexec/
To:

Code: Select all

nagios ALL=(ALL) NOPASSWD: /usr/local/nagios/libexec/check_jvm
Let us know if that fixes your issue.

I updated the sudoers file as suggested but the same result.

Code: Select all

nagios ALL=(ALL) NOPASSWD: /usr/local/nagios/libexec/check_jvm

root@server1:/usr/local/nagios/libexec# ./check_nrpe -H localhost -c check_tomcat_threads
UNKNOWN Can't connect to the JVM:

Re: NRPE: UNKNOWN Can't connect to the JVM

Posted: Fri Nov 10, 2017 2:09 pm
by pmoradiya
Appreciate if anyone can point to any small or big part of configuration I might have missed.

Thanks.

Re: NRPE: UNKNOWN Can't connect to the JVM

Posted: Fri Nov 10, 2017 3:11 pm
by npolovenko
@pmoradiya, Can you upload your nrpe.cfg file? Also, since you're running the check from the tomcat8 user, I'd add another line in the sudoers file:

Code: Select all

tomcat8 ALL=(ALL) NOPASSWD: /usr/local/nagios/libexec/check_jvm
Keep us updated on this.

Re: NRPE: UNKNOWN Can't connect to the JVM

Posted: Sat Nov 11, 2017 10:13 am
by pmoradiya
npolovenko wrote:@pmoradiya, Can you upload your nrpe.cfg file? Also, since you're running the check from the tomcat8 user, I'd add another line in the sudoers file:

Code: Select all

tomcat8 ALL=(ALL) NOPASSWD: /usr/local/nagios/libexec/check_jvm
Keep us updated on this.
I have also added this line to sudoers but same result.

I have attached nrpe.cfg

Re: NRPE: UNKNOWN Can't connect to the JVM

Posted: Mon Nov 13, 2017 3:10 pm
by npolovenko
@pmoradiya, Can you manually switch to tomcat8 user and attempt to run the plugin?

Code: Select all

 su - tomcat8
And then:

Code: Select all

/usr/bin/sudo -u tomcat8 /usr/local/nagios/libexec/check_jvm -n org.apache.catalina.startup.Bootstrap -p threads -w 195 -c 225
Does that work?
I'd also modify permissions for JvmInspector.jar to allow for tomcat8 user.

Code: Select all

chown tomcat8 JvmInspector.jar