check_ncpa timeout after ncpa agent upgrade to version 2.3.1

[bsanjay]

Hello Team,
We have recently updated the NCPA Agent on our client machines(monitored servers) version to 2.3.1 but after updating the agent checks started failing with message "service check time out" . FYI, these servers are in same network and was working fine before the update but after update we are unable to access NCPA Web page also. Please find the below info for your reference and do the needful,

->Port 5693 listening on Client Machine
->Client machine is reachable from nagios server
->It was working fine before NCPA Agent update

bsanjay@nagiosxi:[~]: nmap -p5693 lstpui.local

Starting Nmap 6.47 ( http://nmap.org ) at 2021-07-01 15:25 EDT
Nmap scan report for lstpui.local (192.168.90.12)
Host is up (0.00088s latency).
PORT STATE SERVICE
5693/tcp open unknown

Nmap done: 1 IP address (1 host up) scanned in 0.12 seconds
bsanjay@nagiosxi:[~]: /usr/local/nagios/libexec/check_ncpa.py -H lstpui.local -t MyNagiosXI -P 5693 -M cpu/percent -q 'aggregate=avg' -c 100 -w 97 -v
Connecting to: https://lstpui.local:5693/api/cpu/perce ... regate=avg
UNKNOWN: Execution exceeded timeout threshold of 60s
bsanjay@nagiosxi:[~]: ping lstpui.local
PING lstpui.local (192.168.90.12) 56(84) bytes of data.
64 bytes from lstpui.local (192.168.90.12): icmp_seq=1 ttl=61 time=1.10 ms
64 bytes from lstpui.local (192.168.90.12): icmp_seq=2 ttl=61 time=1.09 ms
^C
--- lstpui.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.091/1.097/1.103/0.006 ms


Best Regards,
bsanjay

[dchurch]

What is the output from the following command?

Code: Select all

/usr/local/nagios/libexec/check_ncpa.py -H lstpui.local -t MyNagiosXI -P 5693

I'm wondering if it's the host or specifically the plugin that's causing the timeout.

Is this a Linux host?

Can you please also PM me your ncpa.cfg?

[bsanjay]

Hello dchurch,
Yes, this is linux host and there are multiple linux hosts on which we are seeing the same issue after agent upgrade,
bsanjay@nagiosxi:[~]: /usr/local/nagios/libexec/check_ncpa.py -H lstpui.local -t MyNagiosXI -P 5693
Usage: check_ncpa.py [options]

Options:
-h, --help show this help message and exit
-H HOSTNAME, --hostname=HOSTNAME
The hostname to be connected to.
-M METRIC, --metric=METRIC
The metric to check, this is defined on client system.
This would also be the plugin name in the plugins
directory. Do not attach arguments to it, use the -a
directive for that. DO NOT INCLUDE the api/
instruction.
-P PORT, --port=PORT Port to use to connect to the client.
-w WARNING, --warning=WARNING
Warning value to be passed for the check.
-c CRITICAL, --critical=CRITICAL
Critical value to be passed for the check.

Best Regards,
bsanjay

[dchurch]

Sorry there was a mistake. What is the output from the following command?

Code: Select all

/usr/local/nagios/libexec/check_ncpa.py -H lstpui.local -t MyNagiosXI -P 5693 --list

Can you please also respond to my prior questions?

Is this a Linux host?

Can you please also PM me your ncpa.cfg?

[bsanjay]

Hello dchurch,
Below command also timeouts, please check the output of the command below.
bsanjay@nagiosxi:[~]: /usr/local/nagios/libexec/check_ncpa.py -H lstpui.local -t MyNagiosXI -P 5693 --list
UNKNOWN: Execution exceeded timeout threshold of 60s


Yes, this is linux host.

ncpa.cfg file attached

Note -
ncpa_listerner is running on host machine.
port 5693 is listening when checked from nagios server.
Tried to increase the timeout to 120 seconds but still getting the timeout error.
we are unable to access these servers over WEB UI also on URL https://lstpui.local:5693

Best Regards,
bsanjay

[dchurch]

Everything looks copacetic in your config. Literally all that's different from the default is the password.

There may still be network latency, a CPU-starved host, or a slow plugin.

I want to know how long the request takes. Can you try running this from the Nagios XI machine?

Code: Select all

time /usr/local/nagios/libexec/check_ncpa.py -H lstpui.local -T 0 -t MyNagiosXI -P 5693 --list

Perhaps it is network latency or possibly an SSL error. Can you try running the following commands from the Nagios XI machine and post the whole output?

Code: Select all

time openssl s_client -connect lstpui.local:5693 </dev/null

Then please log into the NCPA machine and do the same thing except this time to the loopback IP:

Code: Select all

time openssl s_client -connect 127.0.0.1:5693 </dev/null

Are there a lot of plugins installed in /usr/local/ncpa/plugins? Can you tar them up and send them over?

Can you please also send the contents of the NCPA logs? Under Linux it's under `/usr/local/ncpa/var/log/*.log`

[bsanjay]

Hello dchurch,
Please find below the required details,
bsanjay@nagiosxi:[~]: time /usr/local/nagios/libexec/check_ncpa.py -H lstpui.local -T 0 -t MyNagiosXI -P 5693 --list
UNKNOWN: An error occured connecting to API. (Connection error: '[Errno 104] Connection reset by peer')

real 1m43.403s
user 0m0.119s
sys 0m0.042s

bsanjay@nagiosxi:[~]: time openssl s_client -connect lstpui.local:5693 </dev/null
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 301 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1625656887
Timeout : 300 (sec)
Verify return code: 0 (ok)
---

real 1m43.746s
user 0m0.016s
sys 0m0.011s

[root@lstpui ~]# time openssl s_client -connect 127.0.0.1:5693 </dev/null
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1625658459
Timeout : 300 (sec)
Verify return code: 0 (ok)
---

There are no plugins inside ncpa plugins directory
[root@lstpui ~]# ls -lrth /usr/local/ncpa/plugins/
total 0

NCPA Logs
please check PM

NOTE - the actual server name & token is different which you can see in log file.

Best Regards,
bsanjay

[dchurch]

Can you try running this on the NCPA host and post the entire output?

Code: Select all

service ncpa_listener restart
time openssl s_client -connect 127.0.0.1:5693 </dev/null

[bsanjay]

Hi dchurch,
Please find the required details,

[root@lstpui ~]# service ncpa_listener restart
Restarting ncpa_listener (via systemctl): [ OK ]

[root@lstpui ~]# time openssl s_client -connect 127.0.0.1:5693 </dev/null
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1625747255
Timeout : 300 (sec)
Verify return code: 0 (ok)
---



real 1m43.173s
user 0m0.015s
sys 0m0.013s

[root@lstpui ~]# service ncpa_listener status
NCPA Listener: Service is running. (pid 5416)

Best Regards,
bsanjay

[ssax]

What does this output?

Code: Select all

curl -k -L -vvv 'https://lstpui.local:5693/'

What does /usr/local/ncpa/var/log/ncpa_listener.log show on the remote system? See any errors in /var/log/messages on the remote system that could be related? Do you have any security software installed on the remote system that could be impacting the connection?