Page 1 of 2
Checking Docker Container with NCPA/NRDP
Posted: Wed Jan 17, 2018 12:58 pm
by cybergene
I installed NCPA on a remote host that have Docker installed. I want to check if one of the containers is running .
I'm using this
https://gist.github.com/ekristen/11254304 that work on the remote host:
Code: Select all
./docker_container.sh <container_id>
OK - <container_id> is running. IP: xxx.xx.x.x, StartedAt: 2018-01-11T14:43:17.538550851Z
but Nagios is showing an error:
Code: Select all
Current Status: UNKNOWN (for 0d 1h 57m 33s)
Status Information: /usr/local/ncpa/plugins/docker_container.sh: 25: [: <container_id>: unexpected operator
/usr/local/ncpa/plugins/docker_container.sh: 30: [: x/usr/bin/docker: unexpected operator
UNKNOWN - Unable to talk to the docker daemon
Since this is run as the Nagios user that doesn't have permissions to run Docker, I modified the scrip to run the docker commands as sudo (e.g. sudo docker inspect.. and so on) and it still runs good when I run it as Nagios user.
ncpa command under /usr/local/ncpa/etc/ncpa.conf.d/example
Code: Select all
[passive checks]
%HOSTNAME%|__HOST__ = system/agent_version
%HOSTNAME%|Disk Usage = disk/logical/|/used_percent --warning 80 --critical 90 --units Gi
%HOSTNAME%|CPU Usage = cpu/percent --warning 60 --critical 80 --aggregate avg
%HOSTNAME%|Swap Usage = memory/swap --warning 60 --critical 80 --units Gi
%HOSTNAME%|Memory Usage = memory/virtual --warning 80 --critical 90 --units Gi
%HOSTNAME%|Process Count = processes --warning 300 --critical 400
%HOSTNAME%|Docker = services --service docker --status running
%HOSTNAME%|ES Head Container = plugins/docker_container.sh/<container_id>
1) How can I test the passive check from Nagios?
2) What would be the issue here?
Thanks!
Re: Checking Docker Container with NCPA/NRDP
Posted: Wed Jan 17, 2018 1:23 pm
by mcapra
For more comprehensive monitoring of containers, I'd recommend
cadvisor paired with the
check_cadvisor plugin. Maybe heavy-handed for your use case, but I'm leaving this here for "the Googlers of the future"
cybergene wrote:
Since this is run as the Nagios user that doesn't have permissions to run Docker, I modified the scrip to run the docker commands as sudo
I would think you could also add the NCPA user to the
docker system group to solve this.
My initial hunch is NCPA is using a different shell than Bash to execute this command.
.sh extensions default to
/bin/sh per the config file's documentation:
https://github.com/NagiosEnterprises/nc ... #L271-L272
You could modify your NCPA configuration to default .sh files to /bin/bash instead of /bin/sh. Then restart NCPA.
Re: Checking Docker Container with NCPA/NRDP
Posted: Wed Jan 17, 2018 4:30 pm
by dwhitfield
@cybergene, did
@mcapra's suggestion work for you?
Re: Checking Docker Container with NCPA/NRDP
Posted: Thu Jan 18, 2018 9:57 am
by cybergene
Hi
@dwhitfield,
No, it didn't work. I'm still getting the the same error "Unable to talk to the docker daemon".
I tried reverting the script, i.e. removing the sudo from the docker commands. Now, it doesn't work with the nagios user in command line and I get "Unable to talk to the docker daemon".
Re: Checking Docker Container with NCPA/NRDP
Posted: Fri Jan 19, 2018 1:31 pm
by mcapra
But do you also see the same "unexpected operator" messages? Or just messages pertaining to the Docker daemon?
Have you also tried adding the NCPA user to the docker system group? Apologies if that seems repetitive, but it's not immediately clear from your last post what specific actions were taken.
Re: Checking Docker Container with NCPA/NRDP
Posted: Fri Jan 19, 2018 3:58 pm
by cybergene
I added nagios user to docker usergroup as in
https://docs.docker.com/engine/installa ... -root-user but it didn't work either.
It does work locally on the remote host but the response to Nagios Core is not working.
Re: Checking Docker Container with NCPA/NRDP
Posted: Fri Jan 19, 2018 5:27 pm
by swolf
Hi @cybergene,
The issue you're experiencing may be due to a bug in NCPA. We found recently (coincidentally, while building an official Docker integration) that NCPA may not correctly recognize secondary groups. This bug should be resolved in fairly short order.
In the meantime, if you need your Docker monitoring immediately, and want to use that specific script, you might have better luck with NRPE. It's not ideal, but we were able to verify that similar plugins work in that environment.
Sorry for any inconvenience on your end.
Re: Checking Docker Container with NCPA/NRDP
Posted: Fri Jan 19, 2018 5:33 pm
by cdienger
Thanks for the input
@swolf!
Re: Checking Docker Container with NCPA/NRDP
Posted: Mon Jan 22, 2018 8:11 am
by cybergene
Hi
@swolf,
Thank you for the input. I can't use NRPE on our servers but I can wait for the bug to be fixed.
@cdienger, is there a notification for the forum or some way I can get notified about this bug so I can update when it get fixed?
Thanks everyone!
Re: Checking Docker Container with NCPA/NRDP
Posted: Tue Jan 23, 2018 2:42 pm
by mcapra
I don't see a specific issue logged on the GitHub, but that would probably be the place to look for these sorts of changes:
https://github.com/NagiosEnterprises/ncpa