Checking Docker Container with NCPA/NRDP

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
cybergene
Posts: 92
Joined: Wed Aug 10, 2016 7:38 am
Location: Ottawa, Ontario, Canada

Checking Docker Container with NCPA/NRDP

Post by cybergene »

I installed NCPA on a remote host that have Docker installed. I want to check if one of the containers is running .
I'm using this https://gist.github.com/ekristen/11254304 that work on the remote host:

Code: Select all

./docker_container.sh <container_id>
OK - <container_id> is running. IP: xxx.xx.x.x, StartedAt: 2018-01-11T14:43:17.538550851Z
but Nagios is showing an error:

Code: Select all

Current Status:	  UNKNOWN   (for 0d 1h 57m 33s)
Status Information:	/usr/local/ncpa/plugins/docker_container.sh: 25: [: <container_id>: unexpected operator
/usr/local/ncpa/plugins/docker_container.sh: 30: [: x/usr/bin/docker: unexpected operator
UNKNOWN - Unable to talk to the docker daemon
Since this is run as the Nagios user that doesn't have permissions to run Docker, I modified the scrip to run the docker commands as sudo (e.g. sudo docker inspect.. and so on) and it still runs good when I run it as Nagios user.

ncpa command under /usr/local/ncpa/etc/ncpa.conf.d/example

Code: Select all

[passive checks]

%HOSTNAME%|__HOST__ = system/agent_version
%HOSTNAME%|Disk Usage = disk/logical/|/used_percent --warning 80 --critical 90 --units Gi
%HOSTNAME%|CPU Usage = cpu/percent --warning 60 --critical 80 --aggregate avg
%HOSTNAME%|Swap Usage = memory/swap --warning 60 --critical 80 --units Gi
%HOSTNAME%|Memory Usage = memory/virtual --warning 80 --critical 90 --units Gi
%HOSTNAME%|Process Count = processes --warning 300 --critical 400

%HOSTNAME%|Docker = services --service docker --status running
%HOSTNAME%|ES Head Container = plugins/docker_container.sh/<container_id>
1) How can I test the passive check from Nagios?
2) What would be the issue here?

Thanks!
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Checking Docker Container with NCPA/NRDP

Post by mcapra »

For more comprehensive monitoring of containers, I'd recommend cadvisor paired with the check_cadvisor plugin. Maybe heavy-handed for your use case, but I'm leaving this here for "the Googlers of the future" ;)
cybergene wrote: Since this is run as the Nagios user that doesn't have permissions to run Docker, I modified the scrip to run the docker commands as sudo
I would think you could also add the NCPA user to the docker system group to solve this.

My initial hunch is NCPA is using a different shell than Bash to execute this command. .sh extensions default to /bin/sh per the config file's documentation:
https://github.com/NagiosEnterprises/nc ... #L271-L272

You could modify your NCPA configuration to default .sh files to /bin/bash instead of /bin/sh. Then restart NCPA.
Former Nagios employee
https://www.mcapra.com/
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Checking Docker Container with NCPA/NRDP

Post by dwhitfield »

@cybergene, did @mcapra's suggestion work for you?
cybergene
Posts: 92
Joined: Wed Aug 10, 2016 7:38 am
Location: Ottawa, Ontario, Canada

Re: Checking Docker Container with NCPA/NRDP

Post by cybergene »

Hi @dwhitfield,

No, it didn't work. I'm still getting the the same error "Unable to talk to the docker daemon".

I tried reverting the script, i.e. removing the sudo from the docker commands. Now, it doesn't work with the nagios user in command line and I get "Unable to talk to the docker daemon".
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Checking Docker Container with NCPA/NRDP

Post by mcapra »

But do you also see the same "unexpected operator" messages? Or just messages pertaining to the Docker daemon?

Have you also tried adding the NCPA user to the docker system group? Apologies if that seems repetitive, but it's not immediately clear from your last post what specific actions were taken.
Former Nagios employee
https://www.mcapra.com/
cybergene
Posts: 92
Joined: Wed Aug 10, 2016 7:38 am
Location: Ottawa, Ontario, Canada

Re: Checking Docker Container with NCPA/NRDP

Post by cybergene »

I added nagios user to docker usergroup as in https://docs.docker.com/engine/installa ... -root-user but it didn't work either.
It does work locally on the remote host but the response to Nagios Core is not working.
User avatar
swolf
Developer
Posts: 294
Joined: Tue Jun 06, 2017 9:48 am

Re: Checking Docker Container with NCPA/NRDP

Post by swolf »

Hi @cybergene,

The issue you're experiencing may be due to a bug in NCPA. We found recently (coincidentally, while building an official Docker integration) that NCPA may not correctly recognize secondary groups. This bug should be resolved in fairly short order.

In the meantime, if you need your Docker monitoring immediately, and want to use that specific script, you might have better luck with NRPE. It's not ideal, but we were able to verify that similar plugins work in that environment.

Sorry for any inconvenience on your end.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Checking Docker Container with NCPA/NRDP

Post by cdienger »

Thanks for the input @swolf!
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
cybergene
Posts: 92
Joined: Wed Aug 10, 2016 7:38 am
Location: Ottawa, Ontario, Canada

Re: Checking Docker Container with NCPA/NRDP

Post by cybergene »

Hi @swolf,

Thank you for the input. I can't use NRPE on our servers but I can wait for the bug to be fixed.

@cdienger, is there a notification for the forum or some way I can get notified about this bug so I can update when it get fixed?

Thanks everyone!
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Checking Docker Container with NCPA/NRDP

Post by mcapra »

I don't see a specific issue logged on the GitHub, but that would probably be the place to look for these sorts of changes:
https://github.com/NagiosEnterprises/ncpa
Former Nagios employee
https://www.mcapra.com/
Locked