NCPA hangs on request of 'API / services'

[mvndnburg]

Hi,

This is for NCPA 2.0.3 running on RHEL6, x86-64. Nagios XI, 5.4.0, also on RHEL 6.

I am performing a particular service check with NCPA on dozens of Linux hosts but on one host it is failing with UNKNOWN: Error occurred while running the plugin. Use the verbose flag for more details..

When I then log in on the NCPA GUI and select API / services the request does not complete (it's hanging), there is no error message in the GUI. Other API functions work normally.
The NCPA log displays the following:

File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
File "/root/ncpa/agent/listener/server.py", line 154, in token_auth_decoration
File "/root/ncpa/agent/listener/server.py", line 941, in api
File "/root/ncpa/agent/listener/services.py", line 236, in walk
File "/root/ncpa/agent/listener/services.py", line 17, in wrapper
File "/root/ncpa/agent/listener/services.py", line 156, in get_services_via_initctl
File "/root/ncpa/agent/listener/services.py", line 17, in wrapper
File "/root/ncpa/agent/listener/services.py", line 200, in get_services_via_initd
NameError: global name 'logging' is not defined
2017-05-02 12:23:46,489 8029 INFO 10.132.164.236 - - [2017-05-02 12:23:46] "GET /api/services HTTP/1.1" 500 3980 0.01380

What causes this? Is it related to this commit from a couple of weeks ago?

[tgriep]

It is hard to say that the change from a few weeks ago could be causing the issue without more information.
Can you login to the remote server and check the following NCPA log files for any errors and post them here?

Code: Select all

/usr/local/ncpa/var/log/ncpa_listener.log
/usr/local/ncpa/var/log/ncpa_passive.log

Have you tried restarting the NCPA listener to see if that fixes the issue?

[mvndnburg]

There is a misunderstanding:

It is hard to say that the change from a few weeks ago could be causing the issue

we did not actually implement that change. It is the only hit that I got when I gargled for the error message

I will upload the requested logs. We did restart the NCPA after the issue occurred.

[tgriep]

Sorry for the misunderstandng.
Can you post how you are running the service check so I can see if I can recreate the error?

[mvndnburg]

No worries.

I use the service command $USER1$/check_ncpa.py -H $HOSTADDRESS$ -t '<token>' -P 5693 -M 'service/snmpd/running' (this is the check_xi_ncpa_agent command).

Like I mentioned: the service check runs fine on 119 other hosts. Also: if I go to the NCPA GUI and call up the API / services page it (the NCPA) gives me the spinner icon and then hangs. Other API functions for this host work fine. The host is not particularly busy.

[tgriep]

Thanks for the command, I tried to recreate the issue and could not.

Lets turn on debugging in the NCPA Agent and check the log file after creating the issue.

Edit the following file

Code: Select all

/usr/local/nagios/ncpa/etc/ncpa.cfg

Under the following section

Code: Select all

[listener]

Change the loglevel to debug like below

Code: Select all

loglevel = debug

Save the file and restart the listener by running

Code: Select all

service ncpa_listener restart

Recreate the issue and post the ncpa_listener.log file again.

Is selinux running on that system?

[mvndnburg]

I have asked our infra guys to make the proposed changes.

Is selinux running on that system?

Yes, it is. And I know SELinux isn't formally supported by Nagios
The settings are managed globally and no other host has the issue.

I think it's probably more related to locally installed (or not installed) software but I don't have enough info about the NCPA product itself and it looks like some of the Py code is compiled into binaries? Makes troubleshooting harder.

[mvndnburg]

Recreate the issue and post the ncpa_listener.log file again.

Here you go:

2017-05-05 14:24:35,678 7070 INFO stopped
2017-05-05 14:24:42,116 7149 INFO started
2017-05-05 14:24:42,118 7149 INFO Using SSL version TLSv1_2
2017-05-05 14:24:47,090 7149 DEBUG Initializing WebSocket
2017-05-05 14:24:47,090 7149 DEBUG Validating WebSocket request
2017-05-05 14:24:47,093 7149 INFO 10.132.164.185 - - [2017-05-05 14:24:47] "GET /gui/ HTTP/1.1" 302 550 0.003907
2017-05-05 14:24:47,298 7149 DEBUG Initializing WebSocket
2017-05-05 14:24:47,299 7149 DEBUG Validating WebSocket request
2017-05-05 14:24:47,322 7149 INFO 10.132.164.185 - - [2017-05-05 14:24:47] "GET /login HTTP/1.1" 200 3068 0.023706
2017-05-05 14:25:04,751 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:04,751 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:04,751 7149 DEBUG Can only upgrade connection if using GET method.
2017-05-05 14:25:04,754 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:04] "POST /login HTTP/1.1" 302 623 0.003362
2017-05-05 14:25:04,964 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:04,964 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:04,973 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:04] "GET /gui/ HTTP/1.1" 200 6766 0.009201
2017-05-05 14:25:08,294 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:08,295 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:08,338 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:08] "GET /gui/api HTTP/1.1" 200 51778 0.043481
2017-05-05 14:25:08,526 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:08,526 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:08,641 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:08] "GET /api/ HTTP/1.1" 200 9646 0.115238
2017-05-05 14:25:13,249 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:13,249 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:13,258 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:13] "GET /api/memory HTTP/1.1" 200 569 0.008625
2017-05-05 14:25:13,272 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:13,272 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:13,280 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:13] "GET /api/memory HTTP/1.1" 200 569 0.008116
2017-05-05 14:25:17,309 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:17,310 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:17,327 7149 ERROR Exception on /api/services [GET]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
File "/root/ncpa/agent/listener/server.py", line 154, in token_auth_decoration
File "/root/ncpa/agent/listener/server.py", line 941, in api
File "/root/ncpa/agent/listener/services.py", line 236, in walk
File "/root/ncpa/agent/listener/services.py", line 17, in wrapper
File "/root/ncpa/agent/listener/services.py", line 156, in get_services_via_initctl
File "/root/ncpa/agent/listener/services.py", line 17, in wrapper
File "/root/ncpa/agent/listener/services.py", line 200, in get_services_via_initd
NameError: global name 'logging' is not defined
2017-05-05 14:25:17,331 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:17] "GET /api/services HTTP/1.1" 500 3980 0.022053
2017-05-05 14:25:17,332 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:17,332 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:17,344 7149 ERROR Exception on /api/services [GET]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
File "/root/ncpa/agent/listener/server.py", line 154, in token_auth_decoration
File "/root/ncpa/agent/listener/server.py", line 941, in api
File "/root/ncpa/agent/listener/services.py", line 236, in walk
File "/root/ncpa/agent/listener/services.py", line 17, in wrapper
File "/root/ncpa/agent/listener/services.py", line 156, in get_services_via_initctl
File "/root/ncpa/agent/listener/services.py", line 17, in wrapper
File "/root/ncpa/agent/listener/services.py", line 200, in get_services_via_initd
NameError: global name 'logging' is not defined
2017-05-05 14:25:17,345 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:17] "GET /api/services HTTP/1.1" 500 3980 0.013450

[tgriep]

I found a few issues on the NCPA Github site that are similar to what you are having and is could be a permission issue for the NCPA agent cannot read the init scripts to get the status of the running services.
One test is could you temporarily disable selinux and see if agent works after that.
If that doesn't work, please post it here.