UNKNOWN: cURL call failed' - Docker monitoring

[amane]

Hi,

We are monitoring the dockers with using NagiosXI. However 'UNKNOWN: cURL call failed' error occurred for below mentioned docker metrics in NagiosXI.
Docker CPU usage
Docker Memory Usage
Docker Running Count

Recently we have upgraded the Nagios XI version from 5.5.10 to 5.6.7.

Could you please help me to troubleshoot this issue?

Thanks,
amane

[mbellerue]

Just to clarify, these checks were running before you upgraded to 5.6.7. Is that correct?

Can you post the full check command that is running? Also, does this fail consistently, or is it an intermittent failure?

[amane]

Hi mbellerue,

Not sure if it was working before upgrading Nagios Instance to 5.6.7.

Please find check commands below for which we are getting 'UNKNOWN: cURL call failed' status.
./check_docker.py -H http://<Hostaddress>:1111/ --check-type 'containers_cpu' -C 'store-order-svc' -t 0 -w '75' -c '80'
./check_docker.py -H http://<Hostaddress>:1111/ --check-type 'containers_memory' -C 'store-order-svc' -t 0 -p -w '75' -c '80'
./check_docker.py -H http://<Hostaddress>:1111/ --check-type 'containers_running' -C 'store-order-svc' -t 0 -l -c '1:'

It's failing constantly.

Please find below the output for curl command from Nagios instance for docker.

[nagios@NAGIOSXI ~]$ curl -v -g http://Hostaddress:1111/
* About to connect() to Hostaddress port 1111 (#0)
* Trying 10.0.*.*...
* Connected to Hostaddress (10.0.*.*) port 1111 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: Hostaddress:1111
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer


Thanks,
amane

[mbellerue]

Okay, first thing's first, let's see if we can get more information from the plugin. Can you add --debug to one of your commands? That should give us some more output to work with.

[amane]

Hi mbellerue,

Please find below --debug output.

[nagios@NagiosXI libexec]$ ./check_docker.py -H http://<Hostaddress>:1111/ --check-type 'containers_cpu' -C 'store-order-svc' -t 0 -w '75' -c '80' --debug
containers
['store-order-svc']
End selection + type
hit containers_list_to_dict
hit is_docker_container_ID
hit talk_to_docker
curl 'http://<Hostaddress>:1111/containers/json?&filters={%22id%22:%5B%22store-order-svc%22%5D}&all=1' -g -f
ERR % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (56) Recv failure: Connection reset by peer

STDOUT
hit nagios_exit
UNKNOWN: cURL call failed


Thanks,
amane

[lmiltchev]

Can you run the following commands on the client (remote machine), and show the output?

Code: Select all

docker version
ps -ef | grep docker
netstat -apn | grep docker
docker container ls --all | grep 'store-order-svc'
grep nagios /etc/group

Also, run the command below on the Nagios XI server, and show the output:

Code: Select all

nmap <client ip> -p 1111
curl -f -g  http://<client ip>:1111/containers/json?&all=true

[amane]

Hi lmiltchev,

Please find below output for the given command.

root@dkr1:~# docker version
Client:
Version: 18.09.7
API version: 1.39
Go version: go1.10.4
Git commit: 2d0083d
Built: Fri Aug 16 14:19:38 2019
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.09.7
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 2d0083d
Built: Thu Aug 15 15:12:41 2019
OS/Arch: linux/amd64
Experimental: false
root@dkr1:~#

root@dkr1:~# ps -ef | grep docker
root 1297 1 0 Nov25 ? 00:00:55 /usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:1111
root 2193 1297 0 Nov25 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 10001 -container-ip <IP Address> -container-port 10001
root 2205 1273 0 Nov25 ? 00:00:04 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/39bf266a34301061bce71930ed4e843707a51695a84933357977f31d0d673fb2 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
root 14656 14593 0 09:11 pts/0 00:00:00 grep --color=auto docker


root@dkr1:~# docker container ls --all | grep 'store-order-svc'
39bf266a3430 <HostName>:8800/docker-hosted/store-order-svc:v9-release-1.1.3 "java -jar /app/orde…" 5 months ago Up 19 hours 127.0.0.1:10001->10001/tcp store-order-svc

root@dkr1:~# grep nagios /etc/group
nagios122:


[nagios@<NagiosXI> ~]$ nmap 10.23.201.85 -p 1111

Starting Nmap 6.47 ( http://nmap.org ) at 2019-12-02 03:56 EST
Nmap scan report for dkr1.1522.lowes.com (10.23.201.85)
Host is up (0.021s latency).
PORT STATE SERVICE
1111/tcp open lmsocialserver

Nmap done: 1 IP address (1 host up) scanned in 0.12 seconds

[nagios@<NagiosXI> ~]$ curl -f -g http://<IP Address>:1111/containers/json?&all=true
[1] 2601
[nagios@<NagiosXI> ~]$ [{"Id":"39bf266a34301061bce71930ed4e843707a51695a84933357977f31d0d673fb2","Names":["/store-order-svc"],"Image":"wsvmlnnep01.ec.lowes.com:8800/docker-hosted/store-order-svc:v9-release-1.1.3","ImageID":"sha256:c101f8f81f07540847cd58d24c7ab6c5ea586c24c8055e91d764bd138ff85dd1","Command":"java -jar /app/order-service.jar","Created":1561388289,"Ports":[{"IP":"127.0.0.1","PrivatePort":10001,"PublicPort":10001,"Type":"tcp"}],"Labels":{},"State":"running","Status":"Up 6 days","HostConfig":{"NetworkMode":"container-network"},"NetworkSettings":{"Networks":{"container-network":{"IPAMConfig":null,"Links":null,"Aliases":null,"NetworkID":"96bb406d447beb79585f5db4d1710ef9341edd019afc0238eb082bdf0ca80ed5","EndpointID":"55002ddd0091787939d890a383a78b9b16048fdf9d83e0860827f951b1c6fc3a","Gateway":"192.168.100.1","IPAddress":"192.168.100.2","IPPrefixLen":24,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"MacAddress":"02:42:c0:a8:64:02","DriverOpts":null}}},"Mounts":[]}]

[1]+ Done curl -f -g http://<Ip Address>:1111/containers/json?


Thanks,
amane

[swolf]

Hi @amane,

First, I just wanted to mention that the IP address and hostname of your server show up several times in the last reply. I mention this only because you did censor them in previous replies.

However, I did notice something strange related to that. Your nmap command is using a different IP address than what seems to be assigned to your host address ("10.0.*.*"). My guess is that the hostname corresponds to the web server that runs inside the container, where the IP corresponds to the machine running the container. If you're going to use a hostname, you need to make sure it points to the IP Address of the machine running the docker containers, not the containers themselves.

[amane]

Hi lmiltchev,

updated the output again, could you please verify.

root@dkr1:~# docker version
Client:
Version: 18.09.7
API version: 1.39
Go version: go1.10.4
Git commit: 2d0083d
Built: Fri Aug 16 14:19:38 2019
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.09.7
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 2d0083d
Built: Thu Aug 15 15:12:41 2019
OS/Arch: linux/amd64
Experimental: false
root@dkr1:~#

root@dkr1:~# ps -ef | grep docker
root 1297 1 0 Nov25 ? 00:00:55 /usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:1111
root 2193 1297 0 Nov25 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 10001 -container-ip <IP Address> -container-port 10001
root 2205 1273 0 Nov25 ? 00:00:04 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/39bf266a34301061bce71930ed4e843707a51695a84933357977f31d0d673fb2 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
root 14656 14593 0 09:11 pts/0 00:00:00 grep --color=auto docker


root@dkr1:~# docker container ls --all | grep 'store-order-svc'
39bf266a3430 wsvmlnnep01.ec.lowes.com:8800/docker-hosted/store-order-svc:v9-release-1.1.3 "java -jar /app/orde…" 5 months ago Up 19 hours 127.0.0.1:10001->10001/tcp store-order-svc

root@dkr1:~# grep nagios /etc/group
nagios122:


[nagios@lxappnagprd021 ~]$ nmap 10.23.201.85 -p 1111

Starting Nmap 6.47 ( http://nmap.org ) at 2019-12-09 07:19 EST
Nmap scan report for dkr1.1522.lowes.com (10.23.201.85)
Host is up (0.022s latency).
PORT STATE SERVICE
1111/tcp open lmsocialserver

Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds
[nagios@nagiosxi ~]$ curl -f -g http://10.23.201.85:1111/containers/json?&all=true
[1] 16344
[nagios@nagiosxi ~]$ [{"Id":"39bf266a34301061bce71930ed4e843707a51695a84933357977f31d0d673fb2","Names":["/store-order-svc"],"Image":"wsvmlnnep01.ec.lowes.com:8800/docker-hosted/store-order-svc:v9-release-1.1.3","ImageID":"sha256:c101f8f81f07540847cd58d24c7ab6c5ea586c24c8055e91d764bd138ff85dd1","Command":"java -jar /app/order-service.jar","Created":1561388289,"Ports":[{"IP":"127.0.0.1","PrivatePort":10001,"PublicPort":10001,"Type":"tcp"}],"Labels":{},"State":"running","Status":"Up 13 days","HostConfig":{"NetworkMode":"container-network"},"NetworkSettings":{"Networks":{"container-network":{"IPAMConfig":null,"Links":null,"Aliases":null,"NetworkID":"96bb406d447beb79585f5db4d1710ef9341edd019afc0238eb082bdf0ca80ed5","EndpointID":"55002ddd0091787939d890a383a78b9b16048fdf9d83e0860827f951b1c6fc3a","Gateway":"192.168.100.1","IPAddress":"192.168.100.2","IPPrefixLen":24,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"MacAddress":"02:42:c0:a8:64:02","DriverOpts":null}}},"Mounts":[]}]

[1]+ Done curl -f -g http://10.23.201.85:1111/containers/json?

Thanks & Regards,
amane

[lmiltchev]

Is 10.23.201.85 the IP address of the docker server or the IP address of the container that you are trying to monitor? Are you still seeing errors in the GUI?