DISK CRITICAL with docker on RHEL7

[michaelptun]

Hi,

[nagios@xxxxxx1 plugins]$ /home/nagios/plugins/check_disk -l -w 5% -c 3%
DISK CRITICAL - net:[4026532527] is not accessible: No such file or directory

I'm having this issue since I have installed docker on rhel 7 and Nagios disk check is showing as critical for some mounts that are not accessible. I downloaded https://nagios-plugins.org/download/nag ... 1492439288, compiled and built it and tested it out with that version and it also showed the same result.

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)

docker --version
Docker version 1.13.0, build 49bf474

I currently have 2 plugins versions in 2 different folders and whichever user (root or nagios) I use, I get the same result with disk being critical for some mount (not sure if it's a docker mount or not). Please see below for the output details.

[nagios@xxxxxx1 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 14G 7.8G 5.2G 61% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 377M 3.5G 10% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sdb1 50G 13G 37G 25% /opt
/dev/sda2 4.0G 781M 3.0G 21% /var
tmpfs 782M 0 782M 0% /run/user/1000
/dev/dm-1 99G 766M 93G 1% /opt/docker/devicemapper/mnt/5971f011ed94af7b39b6875624b642ede1915d8c3e88d0e2bd5cfceece0c7f
shm 64M 0 64M 0% /opt/docker/containers/97df6beb3ff84fd7ef3d3c7881197c29aef6dac748e67ccc6a2d28c916950c/shm
/dev/dm-2 99G 451M 93G 1% /opt/docker/devicemapper/mnt/1e81c9d83dd8b0b77ceada51ed92c677eba2429301be2f568fc61903696c8b
shm 64M 0 64M 0% /opt/docker/containers/ea157866c0413a1270c06f87af32480266268298948e2da8e99ade88f4022b/shm

[nagios@xxxxxx1 ~]$ /home/nagios/plugins/check_disk --version
check_disk v2.0.3 (nagios-plugins 2.0.3)
[nagios@xxxxxxx1 ~]$ /home/nagios/plugins/check_disk -l -w 5% -c 3%
DISK CRITICAL - net:[4026532527] is not accessible: No such file or directory
[nagios@xxxxxx1 ~]$ /home/nagios/plugins/check_disk -l -w 5% -c 3% -X devtmpfs -X tmpfs -X /dev/dm-1 -X /dev/dm-2 -X shm
DISK CRITICAL - net:[4026532527] is not accessible: No such file or directory

[nagios@xxxxxx1 ~]$ /usr/local/src/nagios-plugins-2.1.3/plugins/check_disk --version
check_disk v2.1.3 (nagios-plugins 2.1.3)
[nagios@xxxxxx1 ~]$ /usr/local/src/nagios-plugins-2.1.3/plugins/check_disk -l
DISK CRITICAL - net:[4026532527] is not accessible: No such file or directory
[nagios@xxxxxx1 ~]$ /usr/local/src/nagios-plugins-2.1.3/plugins/check_disk -l -w 5% -c 3% -X devtmpfs -X tmpfs -X /dev/dm-1 -X /dev/dm-2 -X shm
DISK CRITICAL - net:[4026532527] is not accessible: No such file or directory
[nagios@xxxxxx1 ~]$ logout

[root@xxxxxx1 plugins]# /home/nagios/plugins/check_disk -l -w 5% -c 3% -X devtmpfs -X tmpfs -X /dev/dm-1 -X /dev/dm-2 -X shm
DISK CRITICAL - net:[4026532527] is not accessible: No such file or directory
[root@xxxxxx1 plugins]# /usr/local/src/nagios-plugins-2.1.3/plugins/check_disk -l -w 5% -c 3% -X devtmpfs -X tmpfs -X /dev/dm-1 -X /dev/dm-2 -X shm
DISK CRITICAL - net:[4026532527] is not accessible: No such file or directory

Is there any remedy for this issue? Any help is greatly appreciated.

Thank You,
Mike

[jfrickson]

I made some fixes for that type of problem. I don't know if yours is covered or not. Check out version in https://github.com/nagios-plugins/nagio ... rc1.tar.gz

[michaelptun]

Hi jfrickson,

Thank you for your build. I tried it and got the same result.

[root@xxxxxx1 plugins]# /usr/local/src/nagios-plugins-release-2.2.1-rc1/plugins/check_disk -l -w 5% -c 3% -X devtmpfs -X tmpfs -X /dev/dm-1 -X /dev/dm-2 -X shm -X cgroup -X /dev/mapper
DISK CRITICAL - net:[4026532527] is not accessible: No such file or directory

Thanks,
Mike

[jfrickson]

Thank you for your build. I tried it and got the same result.

On or near line 213 of check_disk.c is:

Code: Select all

    for (me = mount_list; me; me = me->me_next) {

right after that line, add the following line:

Code: Select all

      printf("Type:%s\n", me->me_type);

Then compile and run it and post the results.

[michaelptun]

Here's the printf output after recompiling.

[root@dlsedsbx1 plugins]# ./check_disk -l -w 5% -c 3% -X devtmpfs -X tmpfs -X /dev/dm-1 -X /dev/dm-2 -X shm -X cgroup -X /dev/mapper
Type:rootfs
Type:sysfs
Type:proc
Type:devtmpfs
Type:securityfs
Type:tmpfs
Type:devpts
Type:tmpfs
Type:tmpfs
Type:cgroup
Type:pstore
Type:cgroup
Type:cgroup
Type:cgroup
Type:cgroup
Type:cgroup
Type:cgroup
Type:cgroup
Type:cgroup
Type:cgroup
Type:cgroup
Type:configfs
Type:ext4
Type:autofs
Type:debugfs
Type:mqueue
Type:hugetlbfs
Type:ext4
Type:ext4
Type:binfmt_misc
Type:tmpfs
Type:ext4
Type:ext4
Type:tmpfs
Type:proc
Type:ext4
Type:tmpfs
Type:proc
DISK CRITICAL - net:[4026532527] is not accessible: No such file or directory

Thanks,
Mike

[jfrickson]

Ok, take that line out and rerun it again adding a -vvv parameter. ./check_disk -vvv -l -w 5% -c 3%

[michaelptun]

Ok, here's the verbose output.

./check_disk -vvv -l -w 5% -c 3% -X devtmpfs -X tmpfs -X /dev/dm-1 -X /dev/dm-2 -X shm -X cgroup -X /dev/mapper
Thresholds(pct) for / warn: 5.000000 crit 3.000000
calling stat on /
For /, used_pct=60.25 free_pct=39.75 used_units=7964 free_units=5256 total_units=13952 used_inodes_pct=19.9 free_inodes_pct=80.1 fsp.fsu_blocksize=4096 mult=1048576
Freespace_units result=0
Freespace% result=0
Usedspace_units result=0
Usedspace_percent result=0
Usedinodes_percent result=0
Freeinodes_percent result=0
Thresholds(pct) for /sys warn: 5.000000 crit 3.000000
calling stat on /sys
Thresholds(pct) for /proc warn: 5.000000 crit 3.000000
calling stat on /proc
Thresholds(pct) for /dev warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/kernel/security warn: 5.000000 crit 3.000000
calling stat on /sys/kernel/security
Thresholds(pct) for /dev/shm warn: 5.000000 crit 3.000000
Thresholds(pct) for /dev/pts warn: 5.000000 crit 3.000000
calling stat on /dev/pts
Thresholds(pct) for /run warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/systemd warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/pstore warn: 5.000000 crit 3.000000
calling stat on /sys/fs/pstore
Thresholds(pct) for /sys/fs/cgroup/cpuset warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/hugetlb warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/devices warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/perf_event warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/net_cls,net_prio warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/pids warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/cpu,cpuacct warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/blkio warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/memory warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/freezer warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/kernel/config warn: 5.000000 crit 3.000000
calling stat on /sys/kernel/config
Thresholds(pct) for /proc/sys/fs/binfmt_misc warn: 5.000000 crit 3.000000
calling stat on /proc/sys/fs/binfmt_misc
Thresholds(pct) for /sys/kernel/debug warn: 5.000000 crit 3.000000
calling stat on /sys/kernel/debug
Thresholds(pct) for /dev/mqueue warn: 5.000000 crit 3.000000
calling stat on /dev/mqueue
Thresholds(pct) for /dev/hugepages warn: 5.000000 crit 3.000000
calling stat on /dev/hugepages
Thresholds(pct) for /opt warn: 5.000000 crit 3.000000
calling stat on /opt
For /opt, used_pct=24.88 free_pct=75.12 used_units=12370 free_units=37368 total_units=50267 used_inodes_pct=22.1 free_inodes_pct=77.9 fsp.fsu_blocksize=4096 mult=1048576
Freespace_units result=0
Freespace% result=0
Usedspace_units result=0
Usedspace_percent result=0
Usedinodes_percent result=0
Freeinodes_percent result=0
Thresholds(pct) for /var warn: 5.000000 crit 3.000000
calling stat on /var
For /var, used_pct=20.49 free_pct=79.51 used_units=784 free_units=3046 total_units=4060 used_inodes_pct=3.64 free_inodes_pct=96.36 fsp.fsu_blocksize=4096 mult=1048576
Freespace_units result=0
Freespace% result=0
Usedspace_units result=0
Usedspace_percent result=0
Usedinodes_percent result=0
Freeinodes_percent result=0
Thresholds(pct) for /run/user/1000 warn: 5.000000 crit 3.000000
Thresholds(pct) for /opt/docker/devicemapper warn: 5.000000 crit 3.000000
calling stat on /opt/docker/devicemapper
For /opt/docker/devicemapper, used_pct=24.88 free_pct=75.12 used_units=12370 free_units=37368 total_units=50267 used_inodes_pct=22.1 free_inodes_pct=77.9 fsp.fsu_blocksize=4096 mult=1048576
Freespace_units result=0
Freespace% result=0
Usedspace_units result=0
Usedspace_percent result=0
Usedinodes_percent result=0
Freeinodes_percent result=0
Thresholds(pct) for /opt/docker/devicemapper/mnt/695971f011ed94af7b39b6875624b642ede1915d8c3e88d0e2bd5cfceece0c7f warn: 5.000000 crit 3.000000
Thresholds(pct) for /opt/docker/containers/3397df6beb3ff84fd7ef3d3c7881197c29aef6dac748e67ccc6a2d28c916950c/shm warn: 5.000000 crit 3.000000
Thresholds(pct) for net:[4026532527] warn: 5.000000 crit 3.000000
calling stat on net:[4026532527]
stat failed on net:[4026532527]
DISK CRITICAL - net:[4026532527] is not accessible: No such file or directory

[jfrickson]

Apparently I have a change in my version that you don't have in yours.

On or around line 1013 is this line:

Code: Select all

    printf("calling stat on %s\n", p->name);

Change it to look like this:

Code: Select all

    printf("calling stat on %s (%s %s)\n", p->name, p->best_match->me_devname, p->best_match->me_type);

Then compile and run it again. Sorry!

[michaelptun]

I found it in line 993 and commented out the line and added your new printf statement.

Code: Select all

stat_path (struct parameter_list *p)
{
  /* Stat entry to check that dir exists and is accessible */
  if (verbose >= 3)
//    printf("calling stat on %s\n", p->name);
    printf("calling stat on %s (%s %s)\n", p->name, p->best_match->me_devname, p->best_match->me_type);
  if (stat (p->name, &stat_buf[0])) {
    if (verbose >= 3)
      printf("stat failed on %s\n", p->name);
    printf("DISK %s - ", _("CRITICAL"));
    die (STATE_CRITICAL, _("%s %s: %s\n"), p->name, _("is not accessible"), strerror(errno));
  }
}

Here's the output after recompiling.

./check_disk -vvv -l -w 5% -c 3% -X devtmpfs -X tmpfs -X /dev/dm-1 -X /dev/dm-2 -X shm -X cgroup -X /dev/mapper
Thresholds(pct) for / warn: 5.000000 crit 3.000000
calling stat on / (/dev/sda1 ext4)
For /, used_pct=60.25 free_pct=39.75 used_units=7964 free_units=5256 total_units=13952 used_inodes_pct=19.9 free_inodes_pct=80.1 fsp.fsu_blocksize=4096 mult=1048576
Freespace_units result=0
Freespace% result=0
Usedspace_units result=0
Usedspace_percent result=0
Usedinodes_percent result=0
Freeinodes_percent result=0
Thresholds(pct) for /sys warn: 5.000000 crit 3.000000
calling stat on /sys (sysfs sysfs)
Thresholds(pct) for /proc warn: 5.000000 crit 3.000000
calling stat on /proc (proc proc)
Thresholds(pct) for /dev warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/kernel/security warn: 5.000000 crit 3.000000
calling stat on /sys/kernel/security (securityfs securityfs)
Thresholds(pct) for /dev/shm warn: 5.000000 crit 3.000000
Thresholds(pct) for /dev/pts warn: 5.000000 crit 3.000000
calling stat on /dev/pts (devpts devpts)
Thresholds(pct) for /run warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/systemd warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/pstore warn: 5.000000 crit 3.000000
calling stat on /sys/fs/pstore (pstore pstore)
Thresholds(pct) for /sys/fs/cgroup/cpuset warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/hugetlb warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/devices warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/perf_event warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/net_cls,net_prio warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/pids warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/cpu,cpuacct warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/blkio warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/memory warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/fs/cgroup/freezer warn: 5.000000 crit 3.000000
Thresholds(pct) for /sys/kernel/config warn: 5.000000 crit 3.000000
calling stat on /sys/kernel/config (configfs configfs)
Thresholds(pct) for /proc/sys/fs/binfmt_misc warn: 5.000000 crit 3.000000
calling stat on /proc/sys/fs/binfmt_misc (binfmt_misc binfmt_misc)
Thresholds(pct) for /sys/kernel/debug warn: 5.000000 crit 3.000000
calling stat on /sys/kernel/debug (debugfs debugfs)
Thresholds(pct) for /dev/mqueue warn: 5.000000 crit 3.000000
calling stat on /dev/mqueue (mqueue mqueue)
Thresholds(pct) for /dev/hugepages warn: 5.000000 crit 3.000000
calling stat on /dev/hugepages (hugetlbfs hugetlbfs)
Thresholds(pct) for /opt warn: 5.000000 crit 3.000000
calling stat on /opt (/dev/sdb1 ext4)
For /opt, used_pct=24.88 free_pct=75.12 used_units=12370 free_units=37368 total_units=50267 used_inodes_pct=22.1 free_inodes_pct=77.9 fsp.fsu_blocksize=4096 mult=1048576
Freespace_units result=0
Freespace% result=0
Usedspace_units result=0
Usedspace_percent result=0
Usedinodes_percent result=0
Freeinodes_percent result=0
Thresholds(pct) for /var warn: 5.000000 crit 3.000000
calling stat on /var (/dev/sda2 ext4)
For /var, used_pct=20.32 free_pct=79.68 used_units=778 free_units=3053 total_units=4060 used_inodes_pct=3.64 free_inodes_pct=96.36 fsp.fsu_blocksize=4096 mult=1048576
Freespace_units result=0
Freespace% result=0
Usedspace_units result=0
Usedspace_percent result=0
Usedinodes_percent result=0
Freeinodes_percent result=0
Thresholds(pct) for /run/user/1000 warn: 5.000000 crit 3.000000
Thresholds(pct) for /opt/docker/devicemapper warn: 5.000000 crit 3.000000
calling stat on /opt/docker/devicemapper (/dev/sdb1 ext4)
For /opt/docker/devicemapper, used_pct=24.88 free_pct=75.12 used_units=12370 free_units=37368 total_units=50267 used_inodes_pct=22.1 free_inodes_pct=77.9 fsp.fsu_blocksize=4096 mult=1048576
Freespace_units result=0
Freespace% result=0
Usedspace_units result=0
Usedspace_percent result=0
Usedinodes_percent result=0
Freeinodes_percent result=0
Thresholds(pct) for /opt/docker/devicemapper/mnt/695971f011ed94af7b39b6875624b642ede1915d8c3e88d0e2bd5cfceece0c7f warn: 5.000000 crit 3.000000
Thresholds(pct) for /opt/docker/containers/3397df6beb3ff84fd7ef3d3c7881197c29aef6dac748e67ccc6a2d28c916950c/shm warn: 5.000000 crit 3.000000
Thresholds(pct) for net:[4026532527] warn: 5.000000 crit 3.000000
calling stat on net:[4026532527] (proc proc)
stat failed on net:[4026532527]
DISK CRITICAL - net:[4026532527] is not accessible: No such file or directory

[jfrickson]

Thresholds(pct) for net:[4026532527] warn: 5.000000 crit 3.000000
calling stat on net:[4026532527] (proc proc)
stat failed on net:[4026532527]
DISK CRITICAL - net:[4026532527] is not accessible: No such file or directory

Try this version - it should fix your problem : https://github.com/nagios-plugins/nagio ... int.tar.gz
You can probably skip adding all the -X options with this, also.