check_docker.py

[exp-user1]

Hello support team,

I have two running containers inside docker hosts. I am using check_docker.py plugin to do attached checks but the output is not ok.

CRITICAL: 100.000000% of containers (2/2) running (how critical if running normally)

CRITICAL: 0.000000% of containers (0/2) healthy (the two containers are healthy but output shows the opposite)

checking cpu and memory give the following error message.

(No output on stdout) stderr: Traceback (most recent call last):
File "/usr/local/nagios/libexec/check_docker.py", line 888, in <module>
_ = main()
File "/usr/local/nagios/libexec/check_docker.py", line 861, in main
(out, value) = do_check(ID_list) # Gives plugin output and a number/dict{str:int}
File "/usr/local/nagios/libexec/check_docker.py", line 849, in do_check
(out,value) = valid_checks[options.check_type][0](ID_list)
File "/usr/local/nagios/libexec/check_docker.py", line 690, in check_containers_memory
if 'usage' in json_object['memory_stats'].keys() and 'limit' in json_object['memory_stats'].keys():
TypeError: list indices must be integers, not str

[lmiltchev]

Go to the CCM > Services, click on each "docker" service one-by-one, click on "Run Check Command" button, and show us the output (the actual command that you are running, along with the output of it). We will try to recreate the issue in-house.

[exp-user1]

check cpu and mem are just showing full command without output.

[lmiltchev]

On the first screenshot - I believe you are not using the thresholds correctly. If your critical threshold is 30%, and 2 out of 2 containers are running (100%), you would get a critical. 100% is greater than 30%. If you wanted to be warned when less than 30% of the containers are running, then you would use "30:". For more information on thresholds and ranges, see the Nagios Plugins Development Guidelines here:

https://nagios-plugins.org/doc/guidelin ... HOLDFORMAT

Here's a couple of examples, showing both scenarios:

Code: Select all

# /usr/local/nagios/libexec/check_docker.py -H http://192.168.18.222:2376/ --check-type 'containers_running' --all -t 0 -l --percentage -w '50' -c '30'
CRITICAL: 100.000000% of containers (2/2) running | total_usage=100.0%;50;30

Code: Select all

# /usr/local/nagios/libexec/check_docker.py -H http://192.168.18.222:2376/ --check-type 'containers_running' --all -t 0 -l --percentage -w '50:' -c '30:'
OK: 100.000000% of containers (2/2) running | total_usage=100.0%;50:;30:

On the other two screenshots - I do get an output, when running the same commands...

Code: Select all

# /usr/local/nagios/libexec/check_docker.py -H http://x.x.x.x:2376/ --check-type 'containers_cpu' --all -t 0 --networks-use-avg -- total-usage --total-average -w '10' -c '20'
OK: average_usage returned OK (value 0.0%), total_usage returned OK (value 0.0%),  | average_usage=0.0%;10.0;20.0 total_usage=0.0%;10.0;20.0

# /usr/local/nagios/libexec/check_docker.py -H http://x.x.x.x:2376/ --check-type 'containers_memory' --all -t 0 --networks-use-avg --total-average --memory-unit GiB -w '10,30' -c '20,50'
OK: average_usage returned OK (value 0.00246047973633GiB), total_usage returned OK (value 0.00492095947266GiB),  | average_usage=0.00246047973633GiB;10.0;20.0 total_usage=0.00492095947266GiB;30.0;50.0

Can you run the commands from the command line (not in the CCM), and add "-v" at the end of each command for increased verbosity? Hopefully, we will be able to debug the issue that way.

[exp-user1]

You are right for threshold point, but this is generated by docker config wizard .

and here is other check output:

./check_docker.py -H http://docker_ip:2376/containers/json?all=true --check-type 'containers_memory' --all -t 0 --networks-use-avg --total-average --memory-unit GiB -w '10,30' -c '20,50'
Traceback (most recent call last):
File "./check_docker.py", line 983, in <module>
_ = main()
File "./check_docker.py", line 956, in main
(out, values) = do_check(ID_list) # Gives plugin output and a number/dict{str:int}
File "./check_docker.py", line 943, in do_check
(out,value) = valid_checks[options.check_type][0](ID_list)
File "./check_docker.py", line 811, in check_containers_memory
if 'usage' in json_object['memory_stats'].keys() and 'limit' in json_object['memory_stats'].keys():
TypeError: list indices must be integers, not str

[lmiltchev]

You are right for threshold point, but this is generated by docker config wizard .

Well, in my wizard I see default thresholds as "50:" and "30:"...

example01.PNG

What is the version of the Docker wizard that you are currently using?

Code: Select all

Admin > System Extensions > Manage Config Wizards

Also, what is the version of the plugin?

Code: Select all

/usr/local/nagios/libexec/check_docker.py -V

I will have to escalate the issue to our developers as I am not able to recreate it in-house. I will let you know if we need any additional information. Thanks!

[exp-user1]

Hi,

Here is docker config wizard version:
Monitor Docker containers via NCPA or through the cURL API.
1.0.3 Nagios Enterprises, LLC. Copyright © 2017-2018 Nagios Enterprises, LLC.

and the plugin version:
/usr/local/nagios/libexec/check_docker.py -V
OK: check_docker.py, Version 1.0.2. See --help for more info.

anyway, I found that I was using wrong API URL "http://docker_ip:2376/containers/json?all=true "
it should be only "http://docker_ip:2376/"

now just "container are Healthy" check is showing, would you check and clarify:

/usr/local/nagios/libexec/check_docker.py -H http://docker_ip:2376/ --check-type 'containers_healthy' --all -t 0 --percentage -l -w '50:' -c '30:'
CRITICAL: 0.000000% of containers (0/2) healthy | percentage=0.0%;50:;30:

Thanks!

[lmiltchev]

There are newer versions of the wizard and the plugin (1.1.1) - FYI

anyway, I found that I was using wrong API URL "http://docker_ip:2376/containers/json?all=true "
it should be only "http://docker_ip:2376/"

Great! Is it safe to close this topic then and mark it as "resolved"? Thanks!

[exp-user1]

Could you please post the newer versions of wizard and plugin?

Thanks!

[lmiltchev]

You can just go to Admin > System Extensions > Manage Config Wizards > Check for Updates > Install Updates. Updating the version of the "Docker" wizard would update the plugin as well. Thanks!