NRPE sudo check_jvm not working on RHEL 8 or Debian 10

[drakedts]

Ah, OK. Thank you for the explanation about variables. I see what you're getting at then. Before starting this thread i did some exploration into variables, since i thought that could be the problem as well. But i was never able to solve it by setting variables.

Anyway, the strace looks like it gives something useful. I added this line to check_jvm:

Code: Select all

strace -f java -jar "${JVMINSPECTOR}" "${PID}" 2>&1 >> /tmp/NAGIOSTEST.txt

I had to remove the $() wrapper that was on the line above as it suppressed the output.

When i run the check from the Nagios server, no /tmp/NAGIOSTEST.txt gets created on the client. I think that is because Systemd tries to isolate daemon's temporary files from each other with PrivateTmp=yes set by default. But, despite no /tmp file being created, some output makes it back to the server where i ran the command:

Code: Select all

# /usr/local/nagios/libexec/check_nrpe -H lnx-b9ssb-devl -u -t 30 -c tomcat_heap
execve("/usr/bin/java", ["java", "-jar", "/usr/local/libexec/JvmInspector."..., "1322856"], 0x7ffffa9b6600 /* 19 vars */) = 0
brk(NULL)                               = 0x556d925c0000
arch_prctl(0x3001 /* ARCH_??? */, 0x7fffc6e9b220) = -1 EINVAL (Invalid argument)
readlink("/proc/self/exe", "/usr/lib/jvm/java-1.8.0-openjdk-"..., 4096) = 73
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2fb6eda000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/bin/../lib/amd64/jli/tls/haswell/avx512_1/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/bin/../lib/amd64/jli/tls/haswell/avx512_1/x86_64", 0x7fffc6e9a460) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/bin/../lib/amd64/jli/tls/haswell/avx512_1

That is for the RHEL 8 client. It seems to be looking for something in the OpenJDK installation that doesn't exist. Such as looking for this path: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/lib/amd64/jli/tls/haswell/avx512_1/x86_64/libpthread.so.0. The closest to that which exists is this directory: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/lib/amd64/jli/.

I tried adding the same strace command on the Debian 10 client. Similar to RHEL 8, no /tmp/NAGIOSTEST.txt was created on the client system. But the server got back considerably more output than it did from the RHEL client; so much so that i'll attach it as a file rather than pasting it here.

[ssax]

What version of java is installed on that system? 1.8.0?

Which one of these did you install?

* JvmInspector-jdk6.jar - executable jar package for JDK versions < 9
* JvmInspector-jdk11.jar - executable jar package for JDK versions 9+

[drakedts]

RHEL 8 has Java 8, Debian 10 has Java 11. Both have the correct versions of JvmInspector. (The jdk6 version is about 4.5 MB, the jdk11 version is about 7 kB.)

The RHEL 8 machine:

Code: Select all

[root@lnx-b9ssb-devl ~]# java -version
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)
[root@lnx-b9ssb-devl ~]# ls -l /usr/local/libexec/JvmInspector.jar
-r--r--r--. 1 root root 4714063 2020-11-06 14:49:07 /usr/local/libexec/JvmInspector.jar

Debian 10 machine:

Code: Select all

[root@lnx-cms-test plugins]# java -version
openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-post-Debian-1deb10u2)
OpenJDK 64-Bit Server VM (build 11.0.9.1+1-post-Debian-1deb10u2, mixed mode, sharing)
[root@lnx-cms-test plugins]# ls -l /usr/local/libexec/JvmInspector.jar
-r--r--r-- 1 root root 7224 2020-12-15 12:23:18 /usr/local/libexec/JvmInspector.jar

[ssax]

Does NRPE run as the nagios user or the nrpe user on your system?

I labbed it up in EL8.3/java 1.8 and it's working, here's what I have set:

/etc/sudoers entry:

Code: Select all

nagios    ALL=(ALL) NOPASSWD: /usr/lib64/nagios/plugins/check_jvm

nrpe.cfg entry:

Code: Select all

command[tomcat_heap]=/usr/bin/sudo -u tomcat /usr/lib64/nagios/plugins/check_jvm -n org.apache.catalina.startup.Bootstrap -p heap -w 90 -c 101

check_jvm perms:

Code: Select all

-rwxr-xr-x. 1 nagios nagios 6001 Dec 11 08:26 /usr/lib64/nagios/plugins/check_jvm

Test:

Code: Select all

[root@xid ~]# /usr/local/nagios/libexec/check_nrpe -H X.X.X.X -c tomcat_heap
CRITICAL 87M |max=954728448;;; commited=514850816;;; used=90772960;;;

[drakedts]

Hmmm. There is clearly something different between our environments if you were able to get it to work. I wonder what it is?

NRPE runs as the "nrpe" user on RHEL 8. In previous posts in this thread we've already gone over things like users, permissions, and sudo configuration and did not find any problems.

How did you install and start Tomcat? I install from an RPM. I'll attach it to this post. I'll also attach the service file that systemd uses to start it. My best guess as to what is wrong is some systemd security thing, but i don't know what.

[ssax]

Please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:

https://support.nagios.com/tickets/

Thank you!

[ssax]

Locking thread, ticket received, we will continue support through the ticket.

Thank you!

EDIT: The resolution:

The fix is to disable systemd's PrivateTmp feature. To do that, create a directory /etc/systemd/system/nrpe.service.d, and then in that directory a file local.conf (base name doesn't matter, just needs to end with ".conf") with contents:

Code: Select all

[Service]
PrivateTmp=false

And then do:

Code: Select all

systemctl daemon-reload
systemctl restart nrpe.service