NRPE Unable to read output - ON CENTOS7 Server

[mccrakem]

Hi

I have setup 2 new Postfix Servers on CentOS7
I have configured nagios the same way on these systems as our original RHEL6 Servers
but when I try and monitor any of the custom monitors all I get back is UNKOWN NRPE: Unable to read output

On our Postfix Server we have the following configured
/etc/nrpe.d/postfix.cfg

# We cannot have dont_blame_nrpe=1
dont_blame_nrpe=1
allowed_hosts=127.0.0.1,<IPADDRESS of Nagios Server>
command[check_cup]=/usr/lib64/nagios/plugins/check_cup
command[check_postfix_mailqueue]=sudo /usr/lib64/nagios/plugins/check_postfix_mailqueue
command[check_postfix]=sudo /usr/lib64/nagios/plugins/check_procs -c 1: -a '/usr/libexec/postfix/master' -u root
command[check_clamsmtp]=sudo /usr/lib64/nagios/plugins/check_procs -c 1: -a 'clamd.clamsmtp -c /etc/clamd.d/clamsmtp.conf' -u clamsmtp
command[check_clamsmtpd]=sudo /usr/lib64/nagios/plugins/check_procs -c 1: -a 'clamsmtpd -f /etc/clamsmtpd.conf' -u clamsmtp
command[check_splunk]=sudo /usr/lib64/nagios/plugins/check_procs -c 1:3 -C splunkd
command[check_mail_ratio]=sudo /usr/lib64/nagios/plugins/check_mail_ratio

The following files do exist in the /usr/lib64/nagios/plugins/ Directory
check_mailq, check_mail_ratio, check_postfix_mailqueue, check_procs
owner is root:root
permissions 755

The same files and permissions are on both of the postfix server types

From the Nagios Server itself I can run the check_nrpe command

[root@dwylbopngios01 libexec]# ./check_nrpe -H <IPADDRESS of RHEL6 Server> -c check_mail_ratio
OK - Successful mail delivery ratio 21%

[root@dwylbopngios01 libexec]# ./check_nrpe -H <IPADDRESS of the CentOS7 Server> -c check_mail_ratio
NRPE: Unable to read output

On the CentOS7 Server the default monitors work fine

check_disk
check_mem
check_cpu

On the CentOS7 Postfix Server

[root@CENTOS7POSTFIX ~]# su - nagios -s /bin/bash
Last login: Wed May 27 08:30:43 EDT 2020 on pts/0
-bash-4.2$ whoami
nagios
-bash-4.2$ sudo /usr/lib64/nagios/plugins/check_mail_ratio
OK - Successful mail delivery ratio 100%-bash-4.2$
-bash-4.2$
-bash-4.2$ exit
logout


[root@CentOS7POSTFIX ~]# su - nrpe -s /bin/bash
Last login: Wed May 27 08:47:15 EDT 2020 on pts/0
-bash-4.2$ whoami
nrpe
-bash-4.2$ sudo /usr/lib64/nagios/plugins/check_mail_ratio
OK - Successful mail delivery ratio 100%-bash-4.2$
-bash-4.2$
-bash-4.2$

So the nagios and the nrpe users can run the command but it seems there is a issue just accessing the system from the Nagios Server

besides 5666 is there another port that is required to be open on the CentOS7 Servers?
or is there an extra plugin required ?

[ssax]

Does your /etc/passwdfor your nagios on the EL7 system have a shell set?

nagios1000:/home/nagios:/bin/bash

What does /var/log/messages show?

Try changing your command to this:

Code: Select all

command[check_mail_ratio]=sudo /usr/lib64/nagios/plugins/check_mail_ratio 2>&1

Restart xinetd/nrpe and then test again. Does it output anything different?

No other ports needed.

[mccrakem]

Hi

Does your /etc/passwdfor your nagios on the EL7 system have a shell set?
No there is no shell set, this is the same on both RHEL6 and CentOS7

Here is a copy of the output

[root@dwylbopemail03 mccrakem]# cat /etc/passwd |grep -i nagios
nagios191:189::/var/spool/nagios:/sbin/nologin
[root@dwylbopemail03 mccrakem]# cat /etc/passwd |grep -i nrpe
nrpe482:482:NRPE user for the NRPE service:/var/run/nrpe:/sbin/nologin
[root@dwylbopemail03 mccrakem]#

-----------------------------------------------------------------------------------------------------------------------------------------------------------------

What does /var/log/messages show ?

I did not see anything in the logs so I stopped and restarted the nrpe service and I then waited 5 minutes and this is all that was in the messages file

May 29 05:01:02 dwylbopemail03 systemd[1]: Stopping User Slice of root.
May 29 05:02:47 dwylbopemail03 nrpe[1095]: Caught SIGTERM - shutting down...
May 29 05:02:47 dwylbopemail03 nrpe[1095]: Daemon shutdown
May 29 05:02:47 dwylbopemail03 systemd[1]: Stopping Nagios Remote Program Executor...
May 29 05:02:47 dwylbopemail03 systemd[1]: Started Nagios Remote Program Executor.
May 29 05:02:47 dwylbopemail03 systemd[1]: Starting Nagios Remote Program Executor...
May 29 05:02:47 dwylbopemail03 nrpe[24551]: Starting up daemon
May 29 05:02:47 dwylbopemail03 nrpe[24551]: Server listening on 0.0.0.0 port 5666.
May 29 05:02:47 dwylbopemail03 nrpe[24551]: Server listening on :: port 5666.
May 29 05:02:47 dwylbopemail03 nrpe[24551]: Warning: Daemon is configured to accept command arguments from clients!
May 29 05:02:47 dwylbopemail03 nrpe[24551]: Listening for connections on port 5666
May 29 05:02:47 dwylbopemail03 nrpe[24551]: Allowing connections from: 127.0.0.1,9.98.11.243

IP Address 9.98.11.243 is our Nagios Server

-----------------------------------------------------------------------------------------------------------------------------------------------------------------

Try changing your command to this:
command[check_mail_ratio]=sudo /usr/lib64/nagios/plugins/check_mail_ratio 2>&1

I restarted the nrpe service and now in the nagios XI GUI is reports

sh: sudo: command not found

thanks

[ssax]

Change your command to the proper sudo path and try again:

Code: Select all

which sudo

Code: Select all

command[check_mail_ratio]=/path/to/your/sudo /usr/lib64/nagios/plugins/check_mail_ratio

I'm wondering if it's just a path issue.

[mccrakem]

Hi

on the postfix serevr I ran teh command which sudo output was
/bin/sudo

So I updated the CFG file
to the following command

command[check_mail_ratio]=/bin/sudo /usr/lib64/nagios/plugins/check_mail_ratio

but I am still seeing the same NRPE Unable to read output

On the Nagios server when I run the command which sudo it reports back
/usr/bin/sudo

So tested this on the postfix server as well but still got the same output

Thanks

[ssax]

Change it to this and do it again:

Code: Select all

command[check_mail_ratio]=/bin/sudo /usr/lib64/nagios/plugins/check_mail_ratio 2>&1

What does it show after doing that?

[mccrakem]

Hi

I changed the command to
command[check_mail_ratio]=/bin/sudo /usr/lib64/nagios/plugins/check_mail_ratio 2>&1

and now I get the message

sh: /bin/sudo: Permission denied


Thanks

[ssax]

What is the output of these commands on the postfix server?

Code: Select all

su - nagios
sudo -l
mount | grep nosuid

[mccrakem]

Hi

If I run the command su - nagios, I get the following

[root@dwylbopemail03 ~]# su - nagios
Last login: Wed Jun 3 06:08:31 EDT 2020 on pts/0
This account is currently not available.
[root@dwylbopemail03 ~]#





But if I run the following command su - nagios -s /bin/bash I get this

[root@dwylbopemail03 ~]# su - nagios -s /bin/bash
Last login: Fri Jun 5 03:34:12 EDT 2020 on pts/0
-bash-4.2$ sudo -l
Matching Defaults entries for nagios on dwylbopemail03:
requiretty, !visiblepw, always_set_home, env_reset, env_keep="COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS",
env_keep+="MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE", env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT
LC_MESSAGES", env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", env_keep+="LC_TIME LC_ALL LANGUAGE LINGUAS
_XKB_CHARSET XAUTHORITY", secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin, !requiretty

User nagios may run the following commands on dwylbopemail03:
(root) NOPASSWD: /opt/puppetlabs/bin/puppet agent -t, /opt/puppetlabs/bin/puppet agent -t --noop, /opt/puppetlabs/bin/puppet
agent -t -d, /opt/puppetlabs/bin/puppet agent -t -d --noop
(ALL) NOPASSWD: /usr/lib64/nagios/plugins/*
-bash-4.2$ mount | grep nosuid
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=3994236k,nr_inodes=998559,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-ag ent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
tmpfs on /run/user/483 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801060k,mode=700,uid=483,gid=483)
tmpfs on /run/user/1718205584 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801060k,mode=700,uid=1718205584,gid=1718200513)
-bash-4.2$


Thanks

[ssax]

You get that because nagios likely has /sbin/nologin as the shell.

Code: Select all

grep nagios /etc/passwd

I don't think that should impact it but let's see what the plugin does:

Please attach this plugin:

Code: Select all

/usr/lib64/nagios/plugins/check_mail_ratio