check_disk error "filesystem does not exist" from NagiosXI

[monit_burb]

Hi,

I just installed from source the nrpe on a RHEL7 with SElinux in permissive mode. The check_disk command works OK for some of the partition like / or /boot but I have another two were I get "does not exist on filesystem" error.

If I run the command locally using the nagios users it does work OK, it only fails when executed from NagiosXI.

Here is an extract from mount -v were the first one works ok with check_disk while the other two don't.

/dev/mapper/rhel-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/rhel-tmp_install on /tmp/install type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/rhel-tmp_download on /tmp/download type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

These are the permission of these two folders
drwxr-xr-x+ 2 root root 6 Apr 9 15:04 download
drwxr-xr-x. 3 root root 16 Nov 17 03:39 install

I enabled debug mode on the nrpe but I cannot see any special error related with this issue, the only message I can see every now and then is

WARNING: my_system() seteuid(0): Operation not permitted

I have a custom scrip for checking mount points that have the same problem, works locally using the nagios user but says those two filesystem does not exist when I execute the command from nagiosxi

This is the script for the mountpoints.
https://github.com/echocat/nagios-plugi ... tpoints.sh

I tried to add to the ACLS the nagios users with the following command but it made no change.
sudo setfacl -m u:nagios:r /tmp/download

Other checks like cpu, ram, uptime, etc works OK on this server, only the two mentioned checks fail.

Any idea? I do not know what else to check.

[ssax]

Generally that indicates that nosuid mount option on the mount is preventing the plugin from working, you can validate with these commands (please attach the full output):

Code: Select all

mount | grep nosuid
grep nosuid fstab

The only solution would be to disable nosuid in the /etc/fstab and then remount them without it:
- Note: Talk to your security team before doing this and understand what it allows
- Note: I'm not sure if it's where the plugin resides that needs it disabled or if it's also needed on the mounted partition you're checking as well so you'll need to test that out.

You can remount it without rebooting with this command but make sure to update your /etc/fstab for a permanent change:

Code: Select all

mount -o nosuid,remount /your/mnt

If you need help, please attach your /etc/fstab and the full output of the mount command.

[monit_burb]

Hi ssax,

I'm afraid this is not the issue as any of the filesystem I'm trying to monitor have the nosuid option because I do run the check disk with the -p option to check specific file systems, I do not run it without it to gather all.

[root@demunhbap01q ~]$ mount | grep nosuid
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=12249584k,nr_inodes=3062396,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuacct,cpu)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_prio,net_cls)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=2452352k,mode=700)
tmpfs on /run/user/1114 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=2452352k,mode=700,uid=1114,gid=1114)
[root@demunhbap01q ~]$ grep nosuid /etc/fstab
[root@demunhbap01q ~]$

As I said, If I switch to the nagios user on the server and run the check_disk command it does work OK, only fails when running it from NagiosXI either the GUI or from a bash shell

From the server itself
[root@demunhbap01q ~]$ sudo su nagios -c "/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /tmp/install"
DISK OK - free space: /tmp/install 8871 MB (57% inode=99%);| /tmp/install=6478MB;12280;13815;0;15350

from nagiosxi server
[root@demunnag01 libexec]$ /usr/local/nagios/libexec/check_nrpe -H 10.100.35.92 -t 30 -c check_disk -a '-w 20% -c 10% -p /tmp/download'
DISK CRITICAL - /tmp/download is not accessible: No such file or directory

[lmiltchev]

drwxr-xr-x+ 2 root root 6 Apr 9 15:04 download

I noticed that you have a plus (+) added to your permissions. Did you set any additional ACLs? What is the output of the following command?

Code: Select all

getfacl /tmp/download

[monit_burb]

drwxr-xr-x+ 2 root root 6 Apr 9 15:04 download

I noticed that you have a plus (+) added to your permissions. Did you set any additional ACLs? What is the output of the following command?

Code: Select all

getfacl /tmp/download

That was me testing if the permissions were the cause so I added extra ACL permissions to the nagios users on that folder but that did not made any change.

getfacl: Removing leading '/' from absolute path names
# file: tmp/download
# owner: root
# group: root
user::rwx
user:nagios:r--
group::r-x
mask::r-x
other::r-x

We have seen the same problem on another server that was installed at the same time and have exactly the same filesystem layout.

Might be the cause some sort of issues with the check_disk itself? We downloaded the nagios-pluing tar file and compiled it on each server.

[lmiltchev]

I doubt it's a problem with the plugin itself, but just to be on the same page, let's make sure you are using the same version of the plugin as we are (in-house). Run the following command and show the output:

Code: Select all

/usr/local/nagios/libexec/check_disk -V

Did you mount the /dev/mapper/rhel-tmp_download partition with the mount command when logged in as root or it is defined in the /etc/fstab? Can you show us the fstab entry if you have one?

Is NRPE running as nagios user? What is the output of the commands below?

Code: Select all

grep 'nrpe_user=\|nrpe_group=' /usr/local/nagios/etc/nrpe.cfg
grep nagios /etc/sudoers

[monit_burb]

I doubt it's a problem with the plugin itself, but just to be on the same page, let's make sure you are using the same version of the plugin as we are (in-house). Run the following command and show the output:

Code: Select all

/usr/local/nagios/libexec/check_disk -V

Did you mount the /dev/mapper/rhel-tmp_download partition with the mount command when logged in as root or it is defined in the /etc/fstab? Can you show us the fstab entry if you have one?

Is NRPE running as nagios user? What is the output of the commands below?

Code: Select all

grep 'nrpe_user=\|nrpe_group=' /usr/local/nagios/etc/nrpe.cfg
grep nagios /etc/sudoers

Hello,

We are using the following version of the plugin

[root@demunhbap01q libexec]$ ./check_disk -V
check_disk v2.0.3 (nagios-plugins 2.0.3)

The server was deployed by a colleague who now is on holidays so I cannot ask him if he mounted manually but I can see the entries on fstab so I guess it was done this way.

/dev/mapper/rhel-tmp_download /tmp/download xfs defaults 0 0
/dev/mapper/rhel-tmp_install /tmp/install xfs defaults 0 0

nrpe is run with the nagios users
[root@demunhbap01q libexec]$ grep 'nrpe_user=\|nrpe_group=' /usr/local/nagios/etc/nrpe.cfg
nrpe_user=nagios
nrpe_group=nagios

I tried to add the following two entries to the sudoers file but originally it had no entry for the Nagios user. Even with those two lines, is still not working properly.
[root@demunhbap01q libexec]$ sudo grep nagios /etc/sudoers
Defaults:nagios !requiretty
nagios ALL=NOPASSWD: /usr/local/nagios/libexec/check_disk *

[lmiltchev]

If you are running NRPE older than 3.2.1, you may be facing some permissions issues, especially if you see this in the log (as you mentioned):

WARNING: my_system() seteuid(0): Operation not permitted

The issue was resolved in NRPE 3.2.1.

Change seteuid error messages to warning/debug (Bryan Heden)

https://github.com/NagiosEnterprises/nr ... ANGELOG.md
I don't believe the plugin's version plays a role here, but just to be on the safe side, you can upgrade your plugins too. For our tests we used check_disk v2.2.1, and you are running v2.0.3.

Here's our KB article on upgrading to NRPE v3 from source:
https://support.nagios.com/kb/article/n ... e-520.html

[monit_burb]

Hi,

I'm running nrpe v 3.2.1 so this is not the cause.
[root@demunhbap01q ~]$ /usr/local/nagios/bin/nrpe -V
NRPE - Nagios Remote Plugin Executor
Version: 3.2.1

About the plugin, I tried to install the latest version but is still not working.

[root@demunhbap01q nagios-plugins-2.2.1]$ /usr/local/nagios/libexec/check_disk -V
check_disk v2.2.1 (nagios-plugins 2.2.1)

[lmiltchev]

At this point I believe we need to move this to a support ticket. We may need to schedule a remote session to further troubleshoot the issue. Please open a ticket via our support center here:

https://support.nagios.com/tickets/