NCPA - check_disk permission denied docker overlay mounts

[Bala.Mutyam]

Hi,

Our NagiosXi reporting critial disk space alerts for Docker overlay mounts for NCPA agent. Please have a look?

Code: Select all

/usr/local/nagios/libexec/check_ncpa.py -H hostname -t 'token' -P 5693 -M 'plugins/check_disk' -q 'args=-l -w 20% -c 10%'

Alert: DISK CRITICAL - /var/lib/docker/overlay2/UID/merged is not accessible: Permission denied

Thanks

[benjaminsmith]

Hi Bala.Mutyam ,

Alert: DISK CRITICAL - /var/lib/docker/overlay2/UID/merged is not accessible: Permission denied

Definitely some type of permissions error here, so let's log into the remote host and make sure it's possible to run the plugin directly as the nagios user su - nagios, then run the plugin and post the output to the thread.

If the plugin runs successfully and you're getting the same error, let's review the permissions on that folder, post the output. Thanks.

Code: Select all

ls -l /var/lib/docker/overlay2/UID/merged

[Bala.Mutyam]

Hi,

Thanks for the update. Apologies for the delay, here is the output as requested:

Code: Select all

bash-4.2$ ./check_disk -l -w 20% -c 10%
DISK CRITICAL - /var/lib/docker/overlay2/d75a3e1b257d7de356e0c78b195c56ace60ca5b669db5b1f33e04d3a021ae58a/merged is not accessible: Permission denied
bash-4.2$ pwd
/usr/local/ncpa/plugins
bash-4.2$ whoami
nagios

Code: Select all

[root@hostname~]# ls -l /var/lib/docker/overlay2/d75a3e1b257d7de356e0c78b195c56ace60ca5b669db5b1f33e04d3a021ae58a/merged
total 100
drwxr-xr-x. 1 root root 4096 Jul 27  2018 bin
drwxr-xr-x. 2 root root 4096 Jun 26  2018 boot
drwxr-xr-x. 1 root root 4096 Aug 12  2019 dev
drwxr-xr-x. 2 root root 4096 Jul 27  2018 docker-entrypoint-initdb.d
lrwxrwxrwx. 1 root root   34 Jul 27  2018 entrypoint.sh -> usr/local/bin/docker-entrypoint.sh
drwxr-xr-x. 1 root root 4096 Aug 12  2019 etc
drwxr-xr-x. 2 root root 4096 Jun 26  2018 home
drwxr-xr-x. 1 root root 4096 Jul 16  2018 lib
drwxr-xr-x. 2 root root 4096 Jul 16  2018 lib64
drwxr-xr-x. 2 root root 4096 Jul 16  2018 media
drwxr-xr-x. 2 root root 4096 Jul 16  2018 mnt
drwxr-xr-x. 2 root root 4096 Jul 16  2018 opt
drwxr-xr-x. 2 root root 4096 Jun 26  2018 proc
drwx------. 1 root root 4096 Oct  4  2019 root
drwxr-xr-x. 1 root root 4096 Aug 12  2019 run
drwxr-xr-x. 2 root root 4096 Jul 16  2018 sbin
drwxr-xr-x. 2 root root 4096 Jul 16  2018 srv
drwxr-xr-x. 2 root root 4096 Jun 26  2018 sys
drwxrwxrwt. 1 root root 4096 Jun 10 20:19 tmp
drwxr-xr-x. 1 root root 4096 Jul 16  2018 usr
drwxr-xr-x. 1 root root 4096 Jul 16  2018 var

[lmiltchev]

Can you run the following two commands from the command line on the client (remote machine), and show the output?

Code: Select all

grep nag /etc/group
grep nagios /etc/sudoers

[Bala.Mutyam]

Here is output of the commands:

Code: Select all

grep nag /etc/group
nagios:x:155:nrpe,nagios
grep nagios /etc/sudoers - No results

[lmiltchev]

Add the following two lines to the /etc/sudoers file in order to allow nagios to check docker:

Code: Select all

nagios ALL=NOPASSWD: /usr/local/nagios/libexec/check_docker.py *
nagios ALL=NOPASSWD: /usr/bin/curl *

Also, you need to add nagios user to the docker group:

Code: Select all

usermod -a -G docker nagios

and restart your machine.

Let us know if this helped.