NCPA Disk Used check against nested LVM mounts fails

[asbmosaic]

I have a CentOS system with LVM logical volumes within one volume group that are mount-nested like this:

Code: Select all

NAME                              KNAME TYPE  SIZE MOUNTPOINT
sdf                               sdf   disk    5G 
└─centos_thissrv_data-data        dm-5  lvm     5G /data
sdg                               sdg   disk    2T 
└─centos_thissrv_data-units       dm-6  lvm     2T /data/units
sdh                               sdh   disk  1.8T 
└─centos_thissrv_data-Unit1       dm-7  lvm   1.8T /data/units/Unit1
sdi                               sdi   disk  768G 
└─centos_thissrv_data-Unit2       dm-8  lvm   768G /data/units/Unit2

I'm attempting to use NCPA to check each logical volume's used space. The check command looks like this:
check_xi_ncpa!-t '<REDACTED>' -P 5693 -M 'disk/logical/|data|units' -w 81 -c 90!!!!!!!

The check on the /data logical volume returns properly, but for the nested mounts, the performance data returned says the node can't be found:
The node (data|units) requested does not exist. You may be trying to access the '|' node.

The other two nested mounts return similarly. Am I formatting something incorrectly? Is there another check that would return properly? Is this a bug in the check?

[lmiltchev]

Can you show us the actual command run from the command line along with the output of it (for Unit1 and/or Unit2)? Also, show us the permissions of these mounts?

[asbmosaic]

Code: Select all

[nagios@srvname ~]$ /usr/local/nagios/libexec/check_ncpa.py -H 111.222.333.444 -t '<REDACTED>' -P 5693 -M 'disk/logical/|data|units|Unit1' -w 81 -c 90
/usr/local/nagios/libexec/check_ncpa.py -H 111.222.333.444 -t '<REDACTED>' -P 5693 -M 'disk/logical/|data|units|Unit1' -w 81 -c 90
The node (|data|units|Unit1) requested does not exist. You may be trying to access the '|' node.

Forgive my ignorance, but why would filesystem permissions be an issue here? I'd expect that the check uses a mechanism similar to "df", not something that would require traversing the filesystem as the nagios user. As things stand, if I su -l nagios and try to traverse to the /data folder:-bash: cd: /data/: Permission denied. But the disk check for that volume succeeds:

Code: Select all

[nagios@srvname ~]$ /usr/local/nagios/libexec/check_ncpa.py -H 111.222.333.444 -t '<REDACTED>' -P 5693 -M 'disk/logical/|data' -w 81 -c 90
OK: Used disk space was 63.20 % (Used: 3.15 GiB, Free: 1.83 GiB, Total: 4.99 GiB) | 'used'=3.15GiB;4;4; 'free'=1.83GiB;4;4; 'total'=4.99GiB;4;4;

[lmiltchev]

Forgive my ignorance, but why would filesystem permissions be an issue here?

We've seen cases, where changing the uid and gid from "nagios" to "root" in the /usr/local/ncpa/etc/ncpa.cfg fixes similar issues. In fact, can you run the following command:

Code: Select all

/usr/local/nagios/libexec/check_ncpa.py -H <ip address> -t '<token>' -M 'disk/logical/' -l

before and after changing the uid and gid in ncpa.cfg, and show the output?

In the /usr/local/ncpa/etc/ncpa.cfg, change this:

Code: Select all

uid = nagios
gid = nagios

to this:

Code: Select all

uid = root
gid = root

save, exit and restart ncpa_listener service

Code: Select all

systemctl restart ncpa_listener.service

Note: After running the check to list the logical volumes (above), you can change the uid and gid back to "nagios". This is just for testing purposes.

BTW, what is the filesystem type that you are using for these mounts?

[asbmosaic]

We've seen cases, where changing the uid and gid from "nagios" to "root" in the /usr/local/ncpa/etc/ncpa.cfg fixes similar issues. In fact, can you run the following command:

Code: Select all

/usr/local/nagios/libexec/check_ncpa.py -H <ip address> -t '<token>' -M 'disk/logical/' -l

before and after changing the uid and gid in ncpa.cfg, and show the output?

Before (nagios/nagios):

Code: Select all

[nagios@msdnagios ~]$ /usr/local/nagios/libexec/check_ncpa.py -H 111.222.333.444 -t '<REDACTED>' -P 5693 -M 'disk/logical/' -l
{
    "logical": {
        "|var": {
            "used_percent": [
                6.1, 
                "%"
            ], 
            "used": [
                0.61, 
                "GiB"
            ], 
            "inodes_used": [
                2994, 
                "inodes"
            ], 
            "free": [
                9.37, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv_var-var"
            ], 
            "inodes_free": [
                5237838, 
                "inodes"
            ], 
            "inodes": [
                5240832, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                9.98, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|var|log": {
            "used_percent": [
                0.6, 
                "%"
            ], 
            "used": [
                0.07, 
                "GiB"
            ], 
            "inodes_used": [
                89, 
                "inodes"
            ], 
            "free": [
                10.92, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv_var_log-var_log"
            ], 
            "inodes_free": [
                5765031, 
                "inodes"
            ], 
            "inodes": [
                5765120, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                10.98, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|boot": {
            "used_percent": [
                17.7, 
                "%"
            ], 
            "used": [
                0.18, 
                "GiB"
            ], 
            "inodes_used": [
                335, 
                "inodes"
            ], 
            "free": [
                0.81, 
                "GiB"
            ], 
            "device_name": [
                "/dev/sda1"
            ], 
            "inodes_free": [
                523953, 
                "inodes"
            ], 
            "inodes": [
                524288, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                0.99, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|data": {
            "used_percent": [
                63.2, 
                "%"
            ], 
            "used": [
                3.15, 
                "GiB"
            ], 
            "inodes_used": [
                42770, 
                "inodes"
            ], 
            "free": [
                1.83, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv_data-data"
            ], 
            "inodes_free": [
                2576622, 
                "inodes"
            ], 
            "inodes": [
                2619392, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                4.99, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|home": {
            "used_percent": [
                59.7, 
                "%"
            ], 
            "used": [
                335.85, 
                "GiB"
            ], 
            "inodes_used": [
                832859, 
                "inodes"
            ], 
            "free": [
                226.73, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv_home-home"
            ], 
            "inodes_free": [
                294339237, 
                "inodes"
            ], 
            "inodes": [
                295172096, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                562.59, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|": {
            "used_percent": [
                6.9, 
                "%"
            ], 
            "used": [
                2.13, 
                "GiB"
            ], 
            "inodes_used": [
                76980, 
                "inodes"
            ], 
            "free": [
                28.85, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv-root"
            ], 
            "inodes_free": [
                16173900, 
                "inodes"
            ], 
            "inodes": [
                16250880, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                30.97, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }
    }
}

The additional volumes aren't even listed, it appears.

After (root/root):

Code: Select all

[nagios@srvname ~]$ /usr/local/nagios/libexec/check_ncpa.py -H 111.222.333.444 -t '<REDACTED>' -P 5693 -M 'disk/logical/' -l
{
    "logical": {
        "|data|units": {
            "used_percent": [
                72.4, 
                "%"
            ], 
            "used": [
                1481.68, 
                "GiB"
            ], 
            "inodes_used": [
                4919620, 
                "inodes"
            ], 
            "free": [
                564.82, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv_data-units"
            ], 
            "inodes_free": [
                209828284, 
                "inodes"
            ], 
            "inodes": [
                214747904, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                2046.5, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|var": {
            "used_percent": [
                6.1, 
                "%"
            ], 
            "used": [
                0.61, 
                "GiB"
            ], 
            "inodes_used": [
                2994, 
                "inodes"
            ], 
            "free": [
                9.37, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv_var-var"
            ], 
            "inodes_free": [
                5237838, 
                "inodes"
            ], 
            "inodes": [
                5240832, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                9.98, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|var|log": {
            "used_percent": [
                0.6, 
                "%"
            ], 
            "used": [
                0.07, 
                "GiB"
            ], 
            "inodes_used": [
                89, 
                "inodes"
            ], 
            "free": [
                10.92, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv_var_log-var_log"
            ], 
            "inodes_free": [
                5765031, 
                "inodes"
            ], 
            "inodes": [
                5765120, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                10.98, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|boot": {
            "used_percent": [
                17.7, 
                "%"
            ], 
            "used": [
                0.18, 
                "GiB"
            ], 
            "inodes_used": [
                335, 
                "inodes"
            ], 
            "free": [
                0.81, 
                "GiB"
            ], 
            "device_name": [
                "/dev/sda1"
            ], 
            "inodes_free": [
                523953, 
                "inodes"
            ], 
            "inodes": [
                524288, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                0.99, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|data|units|Unit1": {
            "used_percent": [
                54.7, 
                "%"
            ], 
            "used": [
                420.06, 
                "GiB"
            ], 
            "inodes_used": [
                598780, 
                "inodes"
            ], 
            "free": [
                347.38, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv_data-Unit1"
            ], 
            "inodes_free": [
                402052356, 
                "inodes"
            ], 
            "inodes": [
                402651136, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                767.44, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|data": {
            "used_percent": [
                63.2, 
                "%"
            ], 
            "used": [
                3.15, 
                "GiB"
            ], 
            "inodes_used": [
                42770, 
                "inodes"
            ], 
            "free": [
                1.83, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv_data-data"
            ], 
            "inodes_free": [
                2576622, 
                "inodes"
            ], 
            "inodes": [
                2619392, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                4.99, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|home": {
            "used_percent": [
                59.7, 
                "%"
            ], 
            "used": [
                335.85, 
                "GiB"
            ], 
            "inodes_used": [
                832859, 
                "inodes"
            ], 
            "free": [
                226.73, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv_home-home"
            ], 
            "inodes_free": [
                294339237, 
                "inodes"
            ], 
            "inodes": [
                295172096, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                562.59, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|data|units|Unit2": {
            "used_percent": [
                64.7, 
                "%"
            ], 
            "used": [
                1157.87, 
                "GiB"
            ], 
            "inodes_used": [
                2540344, 
                "inodes"
            ], 
            "free": [
                632.82, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv_data-Unit2"
            ], 
            "inodes_free": [
                185364040, 
                "inodes"
            ], 
            "inodes": [
                187904384, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                1790.69, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }, 
        "|": {
            "used_percent": [
                6.9, 
                "%"
            ], 
            "used": [
                2.12, 
                "GiB"
            ], 
            "inodes_used": [
                76980, 
                "inodes"
            ], 
            "free": [
                28.85, 
                "GiB"
            ], 
            "device_name": [
                "/dev/mapper/centos_thissrv-root"
            ], 
            "inodes_free": [
                16173900, 
                "inodes"
            ], 
            "inodes": [
                16250880, 
                "inodes"
            ], 
            "fstype": "xfs", 
            "total": [
                30.97, 
                "GiB"
            ], 
            "opts": "rw,relatime,attr2,inode64,noquota"
        }
    }
}

...and the missing volumes have appeared.

BTW, what is the filesystem type that you are using for these mounts?

xfs

[lmiltchev]

I believe we are moving to the right direction. This is a permissions issue as you can see the mounts as "root". It is not clear yet what is causing it, but we will try to recreate it in-house, and have our developers look into it. Can you run the following commands, and show the output?

Code: Select all

ls -lad /data /data/units /data/units/Unit1
ls -lad /data /data/units /data/units/Unit2

Also, show us how these volumes are mounted (the /etc/fstab entries).

Is there anything else that could affect this check as far as custom mods that we are not aware of (hardening the server, ACLs, SELinux, etc.)?

[asbmosaic]

Can you run the following commands, and show the output?

Code: Select all

ls -lad /data /data/units /data/units/Unit1
ls -lad /data /data/units /data/units/Unit2

Code: Select all

[root@thissrv data] # ls -ladn /data /data/units /data/units/Unit1
drwxr-x---+  11 0 1000 4096 Sep 10 22:02 /data
drwxr-xr-x+  43 0 1001 4096 Sep 11 04:04 /data/units
drwxrwxr-x+ 185 0    0 8192 Sep 11 00:22 /data/units/Unit1

[root@thissrv data] # ls -ladn /data /data/units /data/units/Unit2
drwxr-x---+  11 0 1000 4096 Sep 10 22:02 /data
drwxr-xr-x+  43 0 1001 4096 Sep 11 04:04 /data/units
drwxrwx---+   8 0 2000 4096 Sep 11 00:00 /data/units/Unit2

Also, show us how these volumes are mounted (the /etc/fstab entries).

Code: Select all

/dev/centos_thissrv_data/data	/data	xfs	defaults	0	0
/dev/centos_thissrv_data/units	/data/units	xfs	defaults	0	0
/dev/centos_thissrv_data/Unit1	/data/units/Unit1	xfs	defaults	0
/dev/centos_thissrv_data/Unit2	/data/units/Unit2	xfs	defaults	0

Is there anything else that could affect this check as far as custom mods that we are not aware of (hardening the server, ACLs, SELinux, etc.)?

• No hardening
• SELinux: disabled
• Heavy use of extended ACLs. This is a Samba fileserver with Windows and OSX clients. The /data filesystem tree is where all that shared data lives.

Nothing else I can think of, it's otherwise a "vanilla" install.

[asbmosaic]

I thought I'd try an experiment, since you think it's permissions.

As the nagios user, I'm unable to traverse the /data directory. So, I'll add nagios:rx perms to that folder:

Code: Select all

[root@thissrv /] # setfacl -m u:nagios:rx /data

Now I can get output from df on the /data/units volume, where I couldn't previously:

Code: Select all

[root@thissrv /] # su -c 'df -h /data/units' nagios
Filesystem                             Size  Used Avail Use% Mounted on
/dev/mapper/centos_thissrv_data-units  2.0T  1.5T  564G  73% /data/units

So now, I'll add nagios:rx perms to the /data/units directory and see if the Unit1 and Unit2 volumes show:

Code: Select all

[root@thissrv /] # setfacl -m u:nagios:rx /data/units
[root@thissrv /] # su -c 'df -h /data/units/Unit1 /data/units/Unit2' nagios
Filesystem                            Size  Used Avail Use% Mounted on
/dev/mapper/centos_thissrv_data-Unit1  768G  421G  348G  55% /data/units/Unit1
/dev/mapper/centos_thissrv_data-Unit2  1.8T  1.2T  636G  65% /data/units/Unit2

Now the NCPA disk checks for the /data/units, /data/units/Unit1 and /data/units/Unit2 volumes complete successfully! I wouldn't have expected this to be a filesystem permissions issue.

EDIT: I backed off the permissions for the nagios user to execute-only for those two folders and it still works.

[lmiltchev]

Yeah, I was going to tell you that "others" won't be able to cd into the /Data directory without the executable bit set.

drwxr-x---+ 11 0 1000 4096 Sep 10 22:02 /data

Most probably, psutil (what we use in the backend) needs to have these permissions in order to access the data. BTW, on one of my test machines, I set the permissions of a directory exactly as yours (750), and my NCPA check failed.

Anyway, I am glad we got to the bottom of this! Is it all right if we lock this topic now or you have more questions? Thanks!

[asbmosaic]

Nothing else. Lock away!