check_nrpe problems: Unable to read output and seteuid(0)

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
markgreene
Posts: 11
Joined: Mon Jun 17, 2019 9:44 am

Re: check_nrpe problems: Unable to read output and seteuid(0

Post by markgreene »

Are you seeing any errors in the syslog on the remote machine when trying to execute the above?
Yes, I get this, from my initial post:
Jun 17 10:12:29 cliplsat01 nrpe[20536]: WARNING: my_system() seteuid(0): Operation not permitted

Which is inexplicable, as nothing that the plugin is doing should require root. I did see where there was a setuid fix for nrpe v3.2.0; I am running v 3.2.1, which should have the fix.

Code: Select all

[root@cliplsat01 ~]$ /usr/sbin/nrpe -h

Incorrect command line arguments supplied

NRPE - Nagios Remote Plugin Executor
Version: 3.2.1

Copyright (c) 2009-2017 Nagios Enterprises
              1999-2008 Ethan Galstad (nagios@nagios.org)

Last Modified: 2017-09-01

License: GPL v2 with exemptions (-l for more info)

SSL/TLS Available, OpenSSL 0.9.6 or higher required

TCP Wrappers Available

***************************************************************
** POSSIBLE SECURITY RISK - COMMAND ARGUMENTS ARE SUPPORTED! **
**      Read the NRPE SECURITY file for more information     **
***************************************************************

Usage: nrpe [-V] [-n] -c <config_file> [-4|-6] <mode>

Options:
 -V, --version         Print version info and quit
 -n, --no-ssl          Do not use SSL
 -c, --config=FILE     Name of config file to use
 -4, --ipv4            Use ipv4 only
 -6, --ipv6            Use ipv6 only
 <mode> (One of the following operating modes)
   -i, --inetd         Run as a service under inetd or xinetd
   -d, --daemon        Run as a standalone daemon
   -s, --src           Run as a subsystem under AIX
   -f, --no-forking    Don't fork() (for systemd, launchd, etc.)

Notes:
This program is designed to process requests from the check_nrpe
plugin on the host(s) running Nagios.  It can run as a service
under inetd or xinetd (read the docs for info on this), or as a
standalone daemon. Once a request is received from an authorized
host, NRPE will execute the command/plugin (as defined in the
config file) and return the plugin output and return code to the
check_nrpe plugin.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: check_nrpe problems: Unable to read output and seteuid(0

Post by scottwilkerson »

Is selinux enabled on the remote machine? I found another forum post similar to this and selinus was the culprit

Code: Select all

getenforce
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
markgreene
Posts: 11
Joined: Mon Jun 17, 2019 9:44 am

Re: check_nrpe problems: Unable to read output and seteuid(0

Post by markgreene »

No selinux, but good question, that's all too often in the way:

[root@cliplsat01 ~]$ sestatus
SELinux status: disabled


disabled on the Nagios system as well:

$ sestatus
SELinux status: disabled


firewalld was running on the Nagios system; I stopped it but that didn't make a difference, still getting "NRPE: Unable to read output" when running the check_nrpe command.

No firewalld or iptables on the remote host.

mark
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: check_nrpe problems: Unable to read output and seteuid(0

Post by scottwilkerson »

Can you upload your /etc/nagios/nrpe.cfg for review
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
markgreene
Posts: 11
Joined: Mon Jun 17, 2019 9:44 am

Re: check_nrpe problems: Unable to read output and seteuid(0

Post by markgreene »

Here's the uncommented content:

[root@cliplsat01 ~]$ grep -v "^#" /etc/nagios/nrpe.cfg |grep .

Code: Select all

log_facility=daemon
pid_file=/var/run/nrpe/nrpe.pid
server_port=5666
nrpe_user=nrpe
nrpe_group=nrpe
allowed_hosts=127.0.0.1,xxx.xxx.xxx.xxx  (Nagios host address)

dont_blame_nrpe=1
allow_bash_command_substitution=0
debug=1
command_timeout=60
connection_timeout=300
command[check_nrpe]=/usr/lib64/nagios/plugins/check_nrpe
command[check_disk]=/usr/lib64/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -E -p $ARG3$
command[check_disk_all]=/usr/lib64/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -e
command[check_swap]=/usr/lib64/nagios/plugins/check_swap -w $ARG1$ -c $ARG2$
command[check_apache]=/usr/lib64/nagios/plugins/check_apache.sh
command[check_cron]=/usr/lib64/nagios/plugins/check_cron.sh
command[check_mailq]=/usr/lib64/nagios/plugins/check_mailq -w $ARG1$ -c $ARG2$
command[check_mem]=/usr/lib64/nagios/plugins/check_mem.py -w $ARG1$ -c $ARG2$
command[check_mem.py]=/usr/lib64/nagios/plugins/check_mem.py -c $ARG1$ -w $ARG2$
include_dir=/etc/nrpe.d/

scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: check_nrpe problems: Unable to read output and seteuid(0

Post by scottwilkerson »

Well that looks ok, I'm getting to the point of stumped. What OS/version is this remote host?

Do you get this error in the syslog every time you run the command? I ask because it looks to be a different ID that the commands earlier.

Code: Select all

Jun 17 10:12:29 cliplsat01 nrpe[20536]: WARNING: my_system() seteuid(0): Operation not permitted
Being this works on your other systems and mine, I'm thinking it must have to do with the OS or python version.
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: check_nrpe problems: Unable to read output and seteuid(0

Post by scottwilkerson »

Also, what do you get when you run these commands

Code: Select all

ps -ef|grep nrpe|grep -v grep
cat /var/run/nrpe/nrpe.pid
ls -al /var/run/nrpe/nrpe.pid
ls -dl /var/run/nrpe/
ls -dl /var/run/
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
markgreene
Posts: 11
Joined: Mon Jun 17, 2019 9:44 am

Re: check_nrpe problems: Unable to read output and seteuid(0

Post by markgreene »

ps -ef|grep nrpe|grep -v grep
nrpe 87277 1 0 Jun17 ? 00:00:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f -n

And the pid file has the correct PID in it, both /var/run/nrpe and /usr/lib64/nagios/plugins/ have execute permissions for "other" all the way down the tree so the nrpe user can navigate down the tree and see the files there.

I need to make time today to hack the nrpe start script to wrap it in an strace and log the output so I can catch the exact operation that is throwing the setuid message, because that is the part that makes no sense here. That I'm apparently the first person who has run into this on nrpe v 3.2.1 is what I'm having a hard time believing.

mark
User avatar
lmiltchev
Former Nagios Staff
Posts: 13587
Joined: Mon May 23, 2011 12:15 pm

Re: check_nrpe problems: Unable to read output and seteuid(0

Post by lmiltchev »

Please post the strace output to the forum whenever you get it. Thank you!
Be sure to check out our Knowledgebase for helpful articles and solutions!
markgreene
Posts: 11
Joined: Mon Jun 17, 2019 9:44 am

Re: check_nrpe problems: Unable to read output and seteuid(0

Post by markgreene »

Here's what I ended up running to get the strace:

Code: Select all

strace -o /root/nrpe_strace.log -ff /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f -n
Which produced these files:

-rw-r--r-- 1 root root 588 Jun 18 13:57 nrpe_strace.log
-rw-r--r-- 1 root root 223 Jun 18 15:28 nrpe_strace.log.60978
-rw-r--r-- 1 root root 95112 Jun 18 15:28 nrpe_strace.log.60981
-rw-r--r-- 1 root root 6494 Jun 18 15:28 nrpe_strace.log.60980
-rw-r--r-- 1 root root 3554 Jun 18 15:28 nrpe_strace.log.60979
-rw-r--r-- 1 root root 29021 Jun 18 15:28 nrpe_strace.log.60971

in the .60981 file, there is this:

Code: Select all

write(2, "Usage: check_mem.py -w <warning "..., 74) = 74
write(2, "\n", 1)                       = 1
write(2, "check_mem.py: error: -w option r"..., 52) = 52
In file .60979, there is this:

Code: Select all

recvfrom(6, "check_mem.py!5 10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1017, 0, NULL, NULL) = 1017
...
sendto(4, "<31>Jun 18 15:28:00 nrpe[60979]:"..., 82, MSG_NOSIGNAL, NULL, 0) = 82
sendto(6, "\0\3\0\2\3\232jI\0\3\0\0\0\0\0\33NRPE: Unable to "..., 46, 0, NULL, 0) = 46
So we see the two errors, the one on the remote host and the one on the Nagios system noted previously, being produced. What we don't see is an exclamation point between the "5" and the "10" in the argument list for check_mem.py.


In file .60981, there is this:

Code: Select all

execve("/usr/bin/python", ["python", "/usr/lib64/nagios/plugins/check_"..., "-c", "5", "10", "-w"], [/* 11 vars */]) = 0
...
write(2, "Usage: check_mem.py -w <warning "..., 74) = 74
write(2, "\n", 1)                       = 1
write(2, "check_mem.py: error: -w option r"..., 52) = 52
so we can see here where the incoming arguments are being mangled either by Nagios or by check_nrpe, I cannot tell which. Running that syntax at the command line correctly returns an error code 2, which Nagios should interpret as an UNKNOWN, and yet does not.

Code: Select all

$ python /usr/lib64/nagios/plugins/check_mem.py -c 5 10 -w; echo $?
Usage: check_mem.py -w <warning threshold> -c <critical threshold> [ -h ]

check_mem.py: error: -w option requires an argument
2
Looking in file /usr/local/nagios/etc/services/cliplsat01.pcc.int.cfg, I can see this config for the service check, which looks correct, as this is how it's coded in my Nagios core system:

Code: Select all

define service {
    host_name                cliplsat01.pcc.int
    service_description      RAM
    use                      generic-service
    check_command            check_nrpe!check_mem.py!10!5
    max_check_attempts       3
    check_interval           5
    retry_interval           1
    check_period             xi_timeperiod_24x7
    notification_interval    60
    notification_period      xi_timeperiod_24x7
    notifications_enabled    1
    contacts                 nagiosadmin
    _xiwizard                nrpe
    register                 1
}
So where do I look next?

mark
Locked