NRPE sudo on RHEL 8

[drakedts]

Hello. I'm trying to get NRPE 4.0.3 working correctly on RHEL 8. The part that is not working is NRPE cannot run sudo commands, which i need in order to be able to monitor Tomcat's heap.

From the Nagios XI server i just get back:

Code: Select all

# ./check_nrpe -H lnx-ethosapi2-test -u -t 30 -c tomcat_heap
UNKNOWN Can't connect to the JVM: 

On the client, "tomcat_heap" is defined like so, using sudo:

Code: Select all

command[tomcat_heap]=/usr/bin/sudo -u tomcat8 /usr/lib64/nagios/plugins/check_jvm -n org.apache.catalina.startup.Bootstrap -p heap -w 90 -c 101

NRPE runs as the nrpe user, and i've confirmed that by checking "ps aux". If i log in as that user, i can run the full command manually and it works great:

Code: Select all

# su - nrpe
Last login: Wed Jul  1 16:20:17 CDT 2020 on pts/0
$ /usr/bin/sudo -u tomcat8 /usr/lib64/nagios/plugins/check_jvm -n org.apache.catalina.startup.Bootstrap -p heap -w 90 -c 101
OK 10% | max=6174015488;;; commited=6174015488;;; used=676489776;;;

I've proven that sudo from the nrpe account works, but just for more info this is what is listed in the sudoers configuration:

Code: Select all

Defaults:nrpe !requiretty
nrpe ALL=(ALL) NOPASSWD: ALL

Note that those permissions are pretty broad; i am planning on tightening it once i get things working.

The SELinux boolean "nagios_run_sudo" is on. But just to make sure that SELinux is not causing problems, i have temporararily turned it off:

Code: Select all

# sestatus 
SELinux status:                 disabled

If i check NRPE's output i can see "WARNING: my_system() seteuid(0): Operation not permitted". I have tried Googling for that error but not gotten any hints that fix the problem. Here's the output that shows the seteuid error:

Code: Select all

# systemctl status nrpe -l
Jul 02 10:51:54 lnx-ethosapi2-test.test.drake.edu nrpe[1436]: Host 10.10.3.0 is asking for command 'tomcat_heap' to be run...
Jul 02 10:51:54 lnx-ethosapi2-test.test.drake.edu nrpe[1436]: Running command: /usr/bin/sudo -u tomcat8 /usr/lib64/nagios/plugins/check_jvm -n org.apache.catalina.startup.Bootstrap -p heap -w 90 -c 101
Jul 02 10:51:54 lnx-ethosapi2-test.test.drake.edu nrpe[1437]: WARNING: my_system() seteuid(0): Operation not permitted
Jul 02 10:51:54 lnx-ethosapi2-test.test.drake.edu sudo[1438]:     nrpe : TTY=unknown ; PWD=/ ; USER=tomcat8 ; COMMAND=/usr/lib64/nagios/plugins/check_jvm -n org.apache.catalina.startup.Bootstrap -p heap -w 90 -c 101
Jul 02 10:51:54 lnx-ethosapi2-test.test.drake.edu nrpe[1436]: Command completed with return code 3 and output: UNKNOWN Can't connect to the JVM:
Jul 02 10:51:54 lnx-ethosapi2-test.test.drake.edu nrpe[1436]: Return Code: 3, Output: UNKNOWN Can't connect to the JVM:

Any ideas what else i can do to make this work?

[lmiltchev]

Try adding the following line to the sudoers file:

Code: Select all

nrpe ALL=NOPASSWD: /usr/lib64/nagios/plugins/check_jvm *

to see if this is going to fix your issue. Thanks!

[drakedts]

Thanks for the suggestion! Unfortunately, it has no effect.

[lmiltchev]

Can you show us a successful run of your command as nrpe user on the client (remote machine)?

[drakedts]

Yes! It is in the 3rd code block in my original posting.

[lmiltchev]

I am sorry for my last question! My bad. So, the nrpe user has no issues running the command. What are the permissions of the "/usr/lib64/nagios/plugins/check_jvm" file?

[drakedts]

Permissions look OK I hope:

Code: Select all

# ls -l /usr/lib64/nagios/plugins/check_jvm
-r-xr-xr-x. 1 root root 6130 2020-02-14 12:04:30 /usr/lib64/nagios/plugins/check_jvm*
# getfacl /usr/lib64/nagios/plugins/check_jvm
getfacl: Removing leading '/' from absolute path names
# file: usr/lib64/nagios/plugins/check_jvm
# owner: root
# group: root
user::r-x
group::r-x
other::r-x
# lsattr /usr/lib64/nagios/plugins/check_jvm
-------------------- /usr/lib64/nagios/plugins/check_jvm
# ls -lZ /usr/lib64/nagios/plugins/check_jvm
-r-xr-xr-x. 1 root root system_u:object_r:nagios_unconfined_plugin_exec_t:s0 6130 2020-02-14 12:04:30 /usr/lib64/nagios/plugins/check_jvm*

Note that SELinux is still set to fully disabled mode so that last command doesn't really matter right now, but i included it for completeness.

[swolf]

Hi @drakedts,

Your permissions look right to me. There's only one thing I'd still want to check, which is that your JVM is running under tomcat8. I assume that's why you used sudo in the first place, so I'm not expecting that to be wrong.

I'm still setting up an environment that can reproduce your issue. Let me know if I have any of the details wrong here:

- NRPE installed through EPEL
- Tomcat version 8.5.56 (or something close), extracted from the official Apache site
- check_jvm/jvminspector at the most recent version, acquired from https://fidanov.net/c0d3/nagios-plugins/jvminspector

edit: also, I suppose there are a couple of other things worth checking:
- Are you able to run check_nrpe -H localhost from the Tomcat server and get the correct response?
- Are you able to run check_nrpe from Nagios XI on another command that doesn't require special permissions? (If you don't have another plugin for this server, just make a bash script that prints out a single line)

[drakedts]

Thank you for looking into this, i appreciate it!

I believe this shows that the JVM is running under the tomcat8 user:

Code: Select all

# ps aux | grep java
tomcat8     1080  0.0  3.6 9991792 300832 ?      Ssl  Jul02   0:28 /usr/lib/jvm/jre/bin/java -Xms6144M -Xmx6144M -classpath /usr/share/tomcat8/bin/bootstrap.jar:/usr/share/tomcat8/bin/tomcat-juli.jar: -Dcatalina.base=/usr/share/tomcat8 -Dcatalina.home=/usr/share/tomcat8 -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat8/temp -Djava.util.logging.config.file=/usr/share/tomcat8/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

As for the environment, it is RHEL 8 with NRPE installed via EPEL as you guessed. Tomcat is 8.5.56, installed via an RPM that we build for internal use from the sources from the Apache site. I have confirmed that our copy of JvmInspector.jar is the same as the one on fidanov.net (i downloaded that one and compared the files).

I cannot run check_nrpe from the Tomcat server, as check_nrpe is not installed. Other NRPE checks that originate from the XI server and do not use sudo work just fine though. For example, from the XI server:

Code: Select all

# /usr/local/nagios/libexec/check_nrpe -H lnx-ethosapi2-test -u -t 30 -c mem
OK - Memory usage is 9%
# /usr/local/nagios/libexec/check_nrpe -H lnx-ethosapi2-test -u -t 30 -c disk
DISK OK| /=5555MiB;12129;13556;0;14270 /sys/fs/cgroup=0MiB;3386;3784;0;3984 /dev/shm=0MiB;3386;3784;0;3984 /run=0MiB;3386;3784;0;3984 /tmp=0MiB;1740;1945;0;2048 /boot/efi=6MiB;41;46;0;49 /run/nrpe=0MiB;3386;3784;0;3984 /var/tmp=5555MiB;12129;13556;0;14270 /run/user/1000=0MiB;676;756;0;796

If it would be helpful, i would be happy to provide the Tomcat package that we built for internal use. I believe i could even export the Tomcat server from our VMware ESXi environment and provide that if you need it. Thank you!

[ssax]

Jul 02 10:51:54 lnx-ethosapi2-test.test.drake.edu nrpe[1437]: WARNING: my_system() seteuid(0): Operation not permitted

What is the output of this command?

Code: Select all

mount | grep nosuid

Is this the check_jvm you are using? If not, please send a link or attach the file.

https://exchange.nagios.org/directory/P ... vm/details

The plugin is able to run which is why you are seeing "Can't connect to the JVM" message from the plugin so it's failing inside the plugin.

If you edit the check_jvm plugin and change the first line from this:

Code: Select all

#!/bin/bash

To this:

Code: Select all

#!/bin/bash -x

And then re-run it, what does it output?