Page 1 of 1

Nagios Docker Stack Plugin works locally but not remote

Posted: Wed May 09, 2018 2:35 pm
by TheBadMan
I have a plugin to check the services running on docker.
I've gone through all of the suggestions from the dozens of posts I have seen here, but I'm still not able to get this to work.
The commands all run individually and the script runs as well, locally.
I added the nrpe user to the wheel and docker groups and set the wheel group in visudo to run anything without a password.

I modified /etc/passwd temporarily to allow the nrpe user to login with a shell. The script runs with or without sudo.

This part passes successfully:

Code: Select all

    /bin/docker > /dev/null 2>&1
    if [ $? -eq 1 ]; then
       echo "UNKNOWN - Missing docker binary"
       exit 3
    fi
This part fails. The command exists with a '1' and I can't get any output from the docker command. I've tried several things, including writing it to a file, but every time the file is blank.

Code: Select all

    /bin/docker stack ls | grep $STACK
    if [ $? -eq 1 ]; then
      echo "UNKNOWN - $STACK does not exist.  $(/usr/bin/sudo /bin/docker stack ls) \n  $(whoami)"
    exit 3
    fi
This is the output I'm getting:
UNKNOWN - ValidService does not exist. \n nrpe.
Any idea what permission I'm missing or is it something else?

Re: Nagios Docker Stack Plugin works locally but not remote

Posted: Thu May 10, 2018 3:37 pm
by kyang
Are you using a custom plugin you created or something from the Nagios Exchange or elsewhere?

This is the output you're getting from using NRPE?

This is just a reference to a check_docker plugin from the Nagios Exchange. (Unsure if you are using this or a custom plugin?)
https://github.com/timdaman/check_docker

Re: Nagios Docker Stack Plugin works locally but not remote

Posted: Fri May 11, 2018 9:43 am
by mcapra
The Docker daemon is particularly picky about what methods are used to communicate with it. I would highly recommend using the Docker SDK if you want to do Docker things with NRPE.

Having implemented similar orchestration, I've found popen (which NRPE uses) to be an incredibly unreliable way to interact with the Docker daemon.

That aside, I haven't encountered any problems invoking the docker command via NRPE:

Code: Select all

[root@capra_nag tmp]# /usr/local/nagios/libexec/check_nrpe -H 10.35.6.65 -c docker_test
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Here's my test script configured on the remote docker machine (associated with NRPE's docker_test command):

Code: Select all

echo $(docker ps 2>&1)
exit 0
I haven't done anything fancier than add the NRPE user (nagios) to the Docker group (docker). That and redirect STDERR to STDOUT in my calls. No sudo funny-business.

Re: Nagios Docker Stack Plugin works locally but not remote

Posted: Fri May 11, 2018 3:20 pm
by scottwilkerson
Thanks for adding that @mcapra