Page 1 of 2
check_docker.py
Posted: Tue Nov 05, 2019 2:51 pm
by exp-user1
Hello support team,
I have two running containers inside docker hosts. I am using check_docker.py plugin to do attached checks but the output is not ok.
CRITICAL: 100.000000% of containers (2/2) running (how critical if running normally)
CRITICAL: 0.000000% of containers (0/2) healthy (the two containers are healthy but output shows the opposite)
checking cpu and memory give the following error message.
(No output on stdout) stderr: Traceback (most recent call last):
File "/usr/local/nagios/libexec/check_docker.py", line 888, in <module>
_ = main()
File "/usr/local/nagios/libexec/check_docker.py", line 861, in main
(out, value) = do_check(ID_list) # Gives plugin output and a number/dict{str:int}
File "/usr/local/nagios/libexec/check_docker.py", line 849, in do_check
(out,value) = valid_checks[options.check_type][0](ID_list)
File "/usr/local/nagios/libexec/check_docker.py", line 690, in check_containers_memory
if 'usage' in json_object['memory_stats'].keys() and 'limit' in json_object['memory_stats'].keys():
TypeError: list indices must be integers, not str
Re: check_docker.py
Posted: Tue Nov 05, 2019 4:28 pm
by lmiltchev
Go to the CCM > Services, click on each "docker" service one-by-one, click on "Run Check Command" button, and show us the output (the actual command that you are running, along with the output of it). We will try to recreate the issue in-house.
Re: check_docker.py
Posted: Wed Nov 06, 2019 2:10 pm
by exp-user1
check cpu and mem are just showing full command without output.
Re: check_docker.py
Posted: Wed Nov 06, 2019 3:02 pm
by lmiltchev
On the first screenshot - I believe you are not using the thresholds correctly. If your critical threshold is 30%, and 2 out of 2 containers are running (100%), you would get a critical. 100% is greater than 30%. If you wanted to be warned when less than 30% of the containers are running, then you would use "30:". For more information on thresholds and ranges, see the Nagios Plugins Development Guidelines here:
https://nagios-plugins.org/doc/guidelin ... HOLDFORMAT
Here's a couple of examples, showing both scenarios:
Code: Select all
# /usr/local/nagios/libexec/check_docker.py -H http://192.168.18.222:2376/ --check-type 'containers_running' --all -t 0 -l --percentage -w '50' -c '30'
CRITICAL: 100.000000% of containers (2/2) running | total_usage=100.0%;50;30
Code: Select all
# /usr/local/nagios/libexec/check_docker.py -H http://192.168.18.222:2376/ --check-type 'containers_running' --all -t 0 -l --percentage -w '50:' -c '30:'
OK: 100.000000% of containers (2/2) running | total_usage=100.0%;50:;30:
On the other two screenshots - I do get an output, when running the same commands...
Code: Select all
# /usr/local/nagios/libexec/check_docker.py -H http://x.x.x.x:2376/ --check-type 'containers_cpu' --all -t 0 --networks-use-avg -- total-usage --total-average -w '10' -c '20'
OK: average_usage returned OK (value 0.0%), total_usage returned OK (value 0.0%), | average_usage=0.0%;10.0;20.0 total_usage=0.0%;10.0;20.0
# /usr/local/nagios/libexec/check_docker.py -H http://x.x.x.x:2376/ --check-type 'containers_memory' --all -t 0 --networks-use-avg --total-average --memory-unit GiB -w '10,30' -c '20,50'
OK: average_usage returned OK (value 0.00246047973633GiB), total_usage returned OK (value 0.00492095947266GiB), | average_usage=0.00246047973633GiB;10.0;20.0 total_usage=0.00492095947266GiB;30.0;50.0
Can you run the commands from the command line (not in the CCM), and add "-v" at the end of each command for increased verbosity? Hopefully, we will be able to debug the issue that way.
Re: check_docker.py
Posted: Wed Nov 06, 2019 4:36 pm
by exp-user1
You are right for threshold point, but this is generated by docker config wizard
.
and here is other check output:
./check_docker.py -H http://docker_ip:2376/containers/json?all=true --check-type 'containers_memory' --all -t 0 --networks-use-avg --total-average --memory-unit GiB -w '10,30' -c '20,50'
Traceback (most recent call last):
File "./check_docker.py", line 983, in <module>
_ = main()
File "./check_docker.py", line 956, in main
(out, values) = do_check(ID_list) # Gives plugin output and a number/dict{str:int}
File "./check_docker.py", line 943, in do_check
(out,value) = valid_checks[options.check_type][0](ID_list)
File "./check_docker.py", line 811, in check_containers_memory
if 'usage' in json_object['memory_stats'].keys() and 'limit' in json_object['memory_stats'].keys():
TypeError: list indices must be integers, not str
Re: check_docker.py
Posted: Wed Nov 06, 2019 5:05 pm
by lmiltchev
You are right for threshold point, but this is generated by docker config wizard
.
Well, in my wizard I see default thresholds as "50:" and "30:"...
example01.PNG
What is the version of the Docker wizard that you are currently using?
Code: Select all
Admin > System Extensions > Manage Config Wizards
Also, what is the version of the plugin?
Code: Select all
/usr/local/nagios/libexec/check_docker.py -V
I will have to escalate the issue to our developers as I am not able to recreate it in-house. I will let you know if we need any additional information. Thanks!
Re: check_docker.py
Posted: Thu Nov 07, 2019 12:58 pm
by exp-user1
Hi,
Here is docker config wizard version:
Monitor Docker containers via NCPA or through the cURL API.
1.0.3 Nagios Enterprises, LLC. Copyright © 2017-2018 Nagios Enterprises, LLC.
and the plugin version:
/usr/local/nagios/libexec/check_docker.py -V
OK: check_docker.py, Version 1.0.2. See --help for more info.
anyway, I found that I was using wrong API URL "http://docker_ip:2376/containers/json?all=true "
it should be only "http://docker_ip:2376/"
now just "container are Healthy" check is showing, would you check and clarify:
/usr/local/nagios/libexec/check_docker.py -H http://docker_ip:2376/ --check-type 'containers_healthy' --all -t 0 --percentage -l -w '50:' -c '30:'
CRITICAL: 0.000000% of containers (0/2) healthy | percentage=0.0%;50:;30:
Thanks!
Re: check_docker.py
Posted: Thu Nov 07, 2019 1:12 pm
by lmiltchev
There are newer versions of the wizard and the plugin (1.1.1) - FYI
anyway, I found that I was using wrong API URL "http://docker_ip:2376/containers/json?all=true "
it should be only "http://docker_ip:2376/"
Great! Is it safe to close this topic then and mark it as "resolved"? Thanks!
Re: check_docker.py
Posted: Fri Nov 08, 2019 9:46 am
by exp-user1
Could you please post the newer versions of wizard and plugin?
Thanks!
Re: check_docker.py
Posted: Fri Nov 08, 2019 10:00 am
by lmiltchev
You can just go to Admin > System Extensions > Manage Config Wizards > Check for Updates > Install Updates. Updating the version of the "Docker" wizard would update the plugin as well. Thanks!