check_disk v2.0.3 -x failed on centos 7 with docker mounts

[nagmoto]

OS and check_disk info

[root@ops01 ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@ops01 ~]# /usr/lib64/nagios/plugins/check_disk --version
check_disk v2.0.3 (nagios-plugins 2.0.3)
[root@ops01 ~]#

[nagmoto]

I installed the latest version from github, -x still not working.

Code: Select all

[root@ops01 ~]# /tmp/check_disk --version
check_disk v2.1.2.15.g75f5 (nagios-plugins 2.1.3)
[root@ops01 ~]# /tmp/check_disk  -l -w 5% -c 3% -X tmpfs -X devtmpfs -x / -x /boot
DISK CRITICAL - net:[4026532622] is not accessible: No such file or directory
[root@ops01 ~]#
[root@ops01 ~]# /tmp/check_disk  -l  -w 5% -c 3% /
DISK OK - free space: / 24555 MB (67% inode=99%);| /=11870MB;34604;35333;0;36426
[root@ops01 ~]#/tmp/check_disk  -l  -w 5% -c 3% /boot
DISK OK - free space: /boot 194 MB (39% inode=99%);| /boot=302MB;471;481;0;496
 [root@ops01 ~]#/tmp/check_disk    -w 5% -c 3%  /var/lib/docker/devicemapper/mnt/abd59bc3dbcd725cfc2881aa4ddeb7db3f2601a552c8a65f30037323920aa653
DISK OK - free space: /var/lib/docker/devicemapper/mnt/abd59bc3dbcd725cfc2881aa4ddeb7db3f2601a552c8a65f30037323920aa653 9683 MB (94% inode=99%);| /var/lib/docker/devicemapper/mnt/abd59bc3dbcd725cfc2881aa4ddeb7db3f2601a552c8a65f30037323920aa653=545MB;9717;9922;0;10229
[root@ops01 ~]#

[avandemore]

It would help very much to find a way to reproduce this internally. Can you tell us any more details on your setup?

Can you start bisecting this command?

Code: Select all

check_disk  -l -w 5% -c 3% -X tmpfs -X devtmpfs -x / -x /boot

eg

Code: Select all

check_disk  -l -w 5% -c 3% -X tmpfs -X devtmpfs 

or

Code: Select all

check_disk  -l -w 5% -c 3% -x / -x /boot

Please continue with the process of elimination until we have identified the problematic issue. This may also require bisecting of mounts as well. Please use the version of check_disk included with Nagios.

[nagmoto]

Both old and latest version of check_disk output included as following

Code: Select all

[root@ops01 ~]# /usr/lib64/nagios/plugins/check_disk --version
check_disk v2.0.3 (nagios-plugins 2.0.3)
[root@ops01 ~]#  /usr/lib64/nagios/plugins/check_disk -X tmpfs -X devtmpfs -x / -x /boot
DISK CRITICAL - net:[4026532622] is not accessible: No such file or directory
[root@ops01 ~]#  /usr/lib64/nagios/plugins/check_disk -l -w 5% -c 3% -X tmpfs -X devtmpfs
DISK CRITICAL - net:[4026532622] is not accessible: No such file or directory
[root@ops01 ~]#  /usr/lib64/nagios/plugins/check_disk  -l -w 5% -c 3% -x / -x /boot
DISK CRITICAL - net:[4026532622] is not accessible: No such file or directory
[root@ops01 ~]#


[root@ops01 ~]# /tmp/check_disk  --version
check_disk v2.1.2.15.g75f5 (nagios-plugins 2.1.3)
[root@ops01 ~]# /tmp/check_disk  -l -w 5% -c 3% -X tmpfs -X devtmpfs  -x / -x /boot
DISK CRITICAL - net:[4026532622] is not accessible: No such file or directory
[root@ops01 ~]# /tmp/check_disk  -l -w 5% -c 3% -X tmpfs -X devtmpfs
DISK CRITICAL - net:[4026532622] is not accessible: No such file or directory
[root@ops01 ~]# /tmp/check_disk  -l -w 5% -c 3% -x / -x /boot
DISK CRITICAL - net:[4026532622] is not accessible: No such file or directory

[avandemore]

This is quite puzzling. What does just a plain /usr/lib64/nagios/plugins/check_disk -l -w 5% -c 3% show? Has the system been rebooted recently? If not can you try that?

Also please umount the nfs things, or show us the details. You can PM me or sanitize the output without clipping useful informaion if you need it to be confidential.

[nagmoto]

sorry for the slow reply.

1. command output

Code: Select all

[root@ops01 ~]# /usr/lib64/nagios/plugins/check_disk -l -w 5% -c 3%
DISK CRITICAL - net:[4026532622] is not accessible: No such file or directory
[root@ops01 ~]#

2. I will ask for reboot

[avandemore]

Also please note the nfs request, let us know when you are ready to proceed.

[nagmoto]

ops01 got rebooted and nfs mount point un-mounted to run following two test commands.
Let me know if these info is not enough for you to debug.

Code: Select all

[root@ops01 ~]# grep share /etc/fstab
opsnas01.test.com:/share /share              nfs    defaults,vers=3        0 0
[root@ops01 ~]# umount /share
[root@ops01 ~]# /root/check_disk --version
check_disk v2.1.2.15.g75f5 (nagios-plugins 2.1.3)
[root@ops01 ~]# /root/check_disk -l -w 5% -c 3%
DISK CRITICAL - net:[4026532540] is not accessible: No such file or directory
[root@ops01 ~]# /usr/lib64/nagios/plugins/check_disk -l -w 5% -c 3%
DISK CRITICAL - net:[4026532540] is not accessible: No such file or directory
[root@ops01 ~]# uptime
 11:19:43 up 4 min,  1 user,  load average: 0.06, 0.34, 0.19
[root@ops01 ~]#

[avandemore]

Ok, please post the output of cat /proc/mounts:

Please un-mount every non-essential filesystem.

Now try /usr/local/nagios/libexec/check_disk -l -w 5% -c 3%. If it succeeds, remount one of the filesystems and repeat. Continue until you reach a failure.

BTW, Docker should not be running on the Nagios host. Nagios is meant as a stand-alone application on a system with as little modification to the surrounding ecosphere as possible.

[nagmoto]

"proc net:[4026532540]" mount points does show up in /proc/mounts out.
Not sure how to trace back which process/software is using this (/proc/net ?)mount point.

Code: Select all

[root@ops01 ~]# cat /proc/mounts |grep -v nfs
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=8117092k,nr_inodes=2029273,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/mapper/centos-root / xfs rw,relatime,attr2,inode64,noquota 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=33,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
<snipped>
/dev/sda1 /boot xfs rw,relatime,attr2,inode64,noquota 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
/dev/mapper/centos-root /var/lib/docker/devicemapper xfs rw,relatime,attr2,inode64,noquota 0 0
/dev/mapper/docker-253:0-202986118-73f6db02095526dbe8a3fd8db384dff5a2696051f9763a72ad6e53ff0ee6ebcf /var/lib/docker/devicemapper/mnt/73f6db02095526dbe8a3fd8db384dff5a2696051f9763a72ad6e53ff0ee6ebcf xfs rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0
/dev/mapper/docker-253:0-202986118-f1bde18188fccafd2c539bf1143a5c04d5e1e15d223fad6e86c2a4a7b6d345e8 /var/lib/docker/devicemapper/mnt/f1bde18188fccafd2c539bf1143a5c04d5e1e15d223fad6e86c2a4a7b6d345e8 xfs rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0
shm /var/lib/docker/containers/76527ed23b11542ea4c31e3360f5db5bcd1b46c5739193323822318dabce400b/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
shm /var/lib/docker/containers/71161447fbe0970c663b53a6e76bdb51a84dded0b6daaa09546f7c5d268fc34e/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
proc net:[4026532540] proc rw,nosuid,nodev,noexec,relatime 0 0
proc net:[4026532622] proc rw,nosuid,nodev,noexec,relatime 0 0
/dev/mapper/docker-253:0-202986118-abd59bc3dbcd725cfc2881aa4ddeb7db3f2601a552c8a65f30037323920aa653 /var/lib/docker/devicemapper/mnt/abd59bc3dbcd725cfc2881aa4ddeb7db3f2601a552c8a65f30037323920aa653 xfs rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0
shm /var/lib/docker/containers/28579998e994403e0282adf1c50fbe0ed64dde1cd106df0c7d1ecf0645c9d33e/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
proc net:[4026532704] proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=1626492k,mode=700 0 0
tmpfs /run/user/35542 tmpfs rw,nosuid,nodev,relatime,size=1626492k,mode=700,uid=35542,gid=20225 0 0
[root@ops01 ~]#