check_nrpe problems: Unable to read output and seteuid(0)

This board serves as an open discussion and support collaboration point for Nagios XI. NOTE: Nagios XI customers should use the Customer Support forum to obtain expedited support.

Re: check_nrpe problems: Unable to read output and seteuid(0

Postby markgreene » Mon Jun 17, 2019 12:23 pm

Are you seeing any errors in the syslog on the remote machine when trying to execute the above?


Yes, I get this, from my initial post:
Jun 17 10:12:29 cliplsat01 nrpe[20536]: WARNING: my_system() seteuid(0): Operation not permitted

Which is inexplicable, as nothing that the plugin is doing should require root. I did see where there was a setuid fix for nrpe v3.2.0; I am running v 3.2.1, which should have the fix.

Code: Select all
[root@cliplsat01 ~]$ /usr/sbin/nrpe -h

Incorrect command line arguments supplied

NRPE - Nagios Remote Plugin Executor
Version: 3.2.1

Copyright (c) 2009-2017 Nagios Enterprises
              1999-2008 Ethan Galstad (nagios@nagios.org)

Last Modified: 2017-09-01

License: GPL v2 with exemptions (-l for more info)

SSL/TLS Available, OpenSSL 0.9.6 or higher required

TCP Wrappers Available

***************************************************************
** POSSIBLE SECURITY RISK - COMMAND ARGUMENTS ARE SUPPORTED! **
**      Read the NRPE SECURITY file for more information     **
***************************************************************

Usage: nrpe [-V] [-n] -c <config_file> [-4|-6] <mode>

Options:
-V, --version         Print version info and quit
-n, --no-ssl          Do not use SSL
-c, --config=FILE     Name of config file to use
-4, --ipv4            Use ipv4 only
-6, --ipv6            Use ipv6 only
<mode> (One of the following operating modes)
   -i, --inetd         Run as a service under inetd or xinetd
   -d, --daemon        Run as a standalone daemon
   -s, --src           Run as a subsystem under AIX
   -f, --no-forking    Don't fork() (for systemd, launchd, etc.)

Notes:
This program is designed to process requests from the check_nrpe
plugin on the host(s) running Nagios.  It can run as a service
under inetd or xinetd (read the docs for info on this), or as a
standalone daemon. Once a request is received from an authorized
host, NRPE will execute the command/plugin (as defined in the
config file) and return the plugin output and return code to the
check_nrpe plugin.
markgreene
 
Posts: 11
Joined: Mon Jun 17, 2019 9:44 am

Re: check_nrpe problems: Unable to read output and seteuid(0

Postby scottwilkerson » Mon Jun 17, 2019 12:45 pm

Is selinux enabled on the remote machine? I found another forum post similar to this and selinus was the culprit

Code: Select all
getenforce
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
scottwilkerson
DevOps Engineer
 
Posts: 16184
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises

Re: check_nrpe problems: Unable to read output and seteuid(0

Postby markgreene » Mon Jun 17, 2019 1:09 pm

No selinux, but good question, that's all too often in the way:

[root@cliplsat01 ~]$ sestatus
SELinux status: disabled


disabled on the Nagios system as well:

$ sestatus
SELinux status: disabled


firewalld was running on the Nagios system; I stopped it but that didn't make a difference, still getting "NRPE: Unable to read output" when running the check_nrpe command.

No firewalld or iptables on the remote host.

mark
markgreene
 
Posts: 11
Joined: Mon Jun 17, 2019 9:44 am

Re: check_nrpe problems: Unable to read output and seteuid(0

Postby scottwilkerson » Mon Jun 17, 2019 3:50 pm

Can you upload your /etc/nagios/nrpe.cfg for review
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
scottwilkerson
DevOps Engineer
 
Posts: 16184
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises

Re: check_nrpe problems: Unable to read output and seteuid(0

Postby markgreene » Mon Jun 17, 2019 4:06 pm

Here's the uncommented content:

[root@cliplsat01 ~]$ grep -v "^#" /etc/nagios/nrpe.cfg |grep .
Code: Select all
log_facility=daemon
pid_file=/var/run/nrpe/nrpe.pid
server_port=5666
nrpe_user=nrpe
nrpe_group=nrpe
allowed_hosts=127.0.0.1,xxx.xxx.xxx.xxx  (Nagios host address)

dont_blame_nrpe=1
allow_bash_command_substitution=0
debug=1
command_timeout=60
connection_timeout=300
command[check_nrpe]=/usr/lib64/nagios/plugins/check_nrpe
command[check_disk]=/usr/lib64/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -E -p $ARG3$
command[check_disk_all]=/usr/lib64/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -e
command[check_swap]=/usr/lib64/nagios/plugins/check_swap -w $ARG1$ -c $ARG2$
command[check_apache]=/usr/lib64/nagios/plugins/check_apache.sh
command[check_cron]=/usr/lib64/nagios/plugins/check_cron.sh
command[check_mailq]=/usr/lib64/nagios/plugins/check_mailq -w $ARG1$ -c $ARG2$
command[check_mem]=/usr/lib64/nagios/plugins/check_mem.py -w $ARG1$ -c $ARG2$
command[check_mem.py]=/usr/lib64/nagios/plugins/check_mem.py -c $ARG1$ -w $ARG2$
include_dir=/etc/nrpe.d/

markgreene
 
Posts: 11
Joined: Mon Jun 17, 2019 9:44 am

Re: check_nrpe problems: Unable to read output and seteuid(0

Postby scottwilkerson » Mon Jun 17, 2019 4:15 pm

Well that looks ok, I'm getting to the point of stumped. What OS/version is this remote host?

Do you get this error in the syslog every time you run the command? I ask because it looks to be a different ID that the commands earlier.
Code: Select all
Jun 17 10:12:29 cliplsat01 nrpe[20536]: WARNING: my_system() seteuid(0): Operation not permitted


Being this works on your other systems and mine, I'm thinking it must have to do with the OS or python version.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
scottwilkerson
DevOps Engineer
 
Posts: 16184
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises

Re: check_nrpe problems: Unable to read output and seteuid(0

Postby scottwilkerson » Mon Jun 17, 2019 4:18 pm

Also, what do you get when you run these commands
Code: Select all
ps -ef|grep nrpe|grep -v grep
cat /var/run/nrpe/nrpe.pid
ls -al /var/run/nrpe/nrpe.pid
ls -dl /var/run/nrpe/
ls -dl /var/run/
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
scottwilkerson
DevOps Engineer
 
Posts: 16184
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises

Re: check_nrpe problems: Unable to read output and seteuid(0

Postby markgreene » Tue Jun 18, 2019 8:43 am

ps -ef|grep nrpe|grep -v grep
nrpe 87277 1 0 Jun17 ? 00:00:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f -n

And the pid file has the correct PID in it, both /var/run/nrpe and /usr/lib64/nagios/plugins/ have execute permissions for "other" all the way down the tree so the nrpe user can navigate down the tree and see the files there.

I need to make time today to hack the nrpe start script to wrap it in an strace and log the output so I can catch the exact operation that is throwing the setuid message, because that is the part that makes no sense here. That I'm apparently the first person who has run into this on nrpe v 3.2.1 is what I'm having a hard time believing.

mark
markgreene
 
Posts: 11
Joined: Mon Jun 17, 2019 9:44 am

Re: check_nrpe problems: Unable to read output and seteuid(0

Postby lmiltchev » Tue Jun 18, 2019 2:53 pm

Please post the strace output to the forum whenever you get it. Thank you!
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.

Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
lmiltchev
QA Manager
 
Posts: 12485
Joined: Mon May 23, 2011 12:15 pm

Re: check_nrpe problems: Unable to read output and seteuid(0

Postby markgreene » Tue Jun 18, 2019 3:04 pm

Here's what I ended up running to get the strace:

Code: Select all
strace -o /root/nrpe_strace.log -ff /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f -n


Which produced these files:

-rw-r--r-- 1 root root 588 Jun 18 13:57 nrpe_strace.log
-rw-r--r-- 1 root root 223 Jun 18 15:28 nrpe_strace.log.60978
-rw-r--r-- 1 root root 95112 Jun 18 15:28 nrpe_strace.log.60981
-rw-r--r-- 1 root root 6494 Jun 18 15:28 nrpe_strace.log.60980
-rw-r--r-- 1 root root 3554 Jun 18 15:28 nrpe_strace.log.60979
-rw-r--r-- 1 root root 29021 Jun 18 15:28 nrpe_strace.log.60971

in the .60981 file, there is this:

Code: Select all
write(2, "Usage: check_mem.py -w <warning "..., 74) = 74
write(2, "\n", 1)                       = 1
write(2, "check_mem.py: error: -w option r"..., 52) = 52


In file .60979, there is this:

Code: Select all
recvfrom(6, "check_mem.py!5 10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1017, 0, NULL, NULL) = 1017
...
sendto(4, "<31>Jun 18 15:28:00 nrpe[60979]:"..., 82, MSG_NOSIGNAL, NULL, 0) = 82
sendto(6, "\0\3\0\2\3\232jI\0\3\0\0\0\0\0\33NRPE: Unable to "..., 46, 0, NULL, 0) = 46


So we see the two errors, the one on the remote host and the one on the Nagios system noted previously, being produced. What we don't see is an exclamation point between the "5" and the "10" in the argument list for check_mem.py.


In file .60981, there is this:

Code: Select all
execve("/usr/bin/python", ["python", "/usr/lib64/nagios/plugins/check_"..., "-c", "5", "10", "-w"], [/* 11 vars */]) = 0
...
write(2, "Usage: check_mem.py -w <warning "..., 74) = 74
write(2, "\n", 1)                       = 1
write(2, "check_mem.py: error: -w option r"..., 52) = 52


so we can see here where the incoming arguments are being mangled either by Nagios or by check_nrpe, I cannot tell which. Running that syntax at the command line correctly returns an error code 2, which Nagios should interpret as an UNKNOWN, and yet does not.

Code: Select all
$ python /usr/lib64/nagios/plugins/check_mem.py -c 5 10 -w; echo $?
Usage: check_mem.py -w <warning threshold> -c <critical threshold> [ -h ]

check_mem.py: error: -w option requires an argument
2


Looking in file /usr/local/nagios/etc/services/cliplsat01.pcc.int.cfg, I can see this config for the service check, which looks correct, as this is how it's coded in my Nagios core system:

Code: Select all
define service {
    host_name                cliplsat01.pcc.int
    service_description      RAM
    use                      generic-service
    check_command            check_nrpe!check_mem.py!10!5
    max_check_attempts       3
    check_interval           5
    retry_interval           1
    check_period             xi_timeperiod_24x7
    notification_interval    60
    notification_period      xi_timeperiod_24x7
    notifications_enabled    1
    contacts                 nagiosadmin
    _xiwizard                nrpe
    register                 1
}


So where do I look next?

mark
markgreene
 
Posts: 11
Joined: Mon Jun 17, 2019 9:44 am

PreviousNext

Return to Nagios XI

Who is online

Users browsing this forum: Majestic-12 [Bot] and 15 guests