Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

NCPA hangs on request of 'API / services'

https://support.nagios.com/forum/viewtopic.php?t=43696

Page 1 of 1

NCPA hangs on request of 'API / services'

Posted: Tue May 02, 2017 5:48 am
by mvndnburg
Hi,

This is for NCPA 2.0.3 running on RHEL6, x86-64. Nagios XI, 5.4.0, also on RHEL 6.

I am performing a particular service check with NCPA on dozens of Linux hosts but on one host it is failing with UNKNOWN: Error occurred while running the plugin. Use the verbose flag for more details..

When I then log in on the NCPA GUI and select API / services the request does not complete (it's hanging), there is no error message in the GUI. Other API functions work normally.
The NCPA log displays the following:
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
File "/root/ncpa/agent/listener/server.py", line 154, in token_auth_decoration
File "/root/ncpa/agent/listener/server.py", line 941, in api
File "/root/ncpa/agent/listener/services.py", line 236, in walk
File "/root/ncpa/agent/listener/services.py", line 17, in wrapper
File "/root/ncpa/agent/listener/services.py", line 156, in get_services_via_initctl
File "/root/ncpa/agent/listener/services.py", line 17, in wrapper
File "/root/ncpa/agent/listener/services.py", line 200, in get_services_via_initd
NameError: global name 'logging' is not defined
2017-05-02 12:23:46,489 8029 INFO 10.132.164.236 - - [2017-05-02 12:23:46] "GET /api/services HTTP/1.1" 500 3980 0.01380
What causes this? Is it related to this commit from a couple of weeks ago?

Re: NCPA hangs on request of 'API / services'

Posted: Tue May 02, 2017 11:43 am
by tgriep
It is hard to say that the change from a few weeks ago could be causing the issue without more information.
Can you login to the remote server and check the following NCPA log files for any errors and post them here?

Code: Select all

/usr/local/ncpa/var/log/ncpa_listener.log
/usr/local/ncpa/var/log/ncpa_passive.log
Have you tried restarting the NCPA listener to see if that fixes the issue?

Re: NCPA hangs on request of 'API / services'

Posted: Wed May 03, 2017 3:59 am
by mvndnburg
There is a misunderstanding:
It is hard to say that the change from a few weeks ago could be causing the issue
we did not actually implement that change. It is the only hit that I got when I gargled for the error message :)

I will upload the requested logs. We did restart the NCPA after the issue occurred.

Re: NCPA hangs on request of 'API / services'

Posted: Wed May 03, 2017 9:40 am
by tgriep
Sorry for the misunderstandng.
Can you post how you are running the service check so I can see if I can recreate the error?

Re: NCPA hangs on request of 'API / services'

Posted: Thu May 04, 2017 12:48 am
by mvndnburg
No worries.

I use the service command $USER1$/check_ncpa.py -H $HOSTADDRESS$ -t '<token>' -P 5693 -M 'service/snmpd/running' (this is the check_xi_ncpa_agent command).

Like I mentioned: the service check runs fine on 119 other hosts. Also: if I go to the NCPA GUI and call up the API / services page it (the NCPA) gives me the spinner icon and then hangs. Other API functions for this host work fine. The host is not particularly busy.

Re: NCPA hangs on request of 'API / services'

Posted: Thu May 04, 2017 9:28 am
by tgriep
Thanks for the command, I tried to recreate the issue and could not.

Lets turn on debugging in the NCPA Agent and check the log file after creating the issue.

Edit the following file

Code: Select all

/usr/local/nagios/ncpa/etc/ncpa.cfg
Under the following section

Code: Select all

[listener]
Change the loglevel to debug like below

Code: Select all

loglevel = debug
Save the file and restart the listener by running

Code: Select all

service ncpa_listener restart
Recreate the issue and post the ncpa_listener.log file again.

Is selinux running on that system?

Re: NCPA hangs on request of 'API / services'

Posted: Fri May 05, 2017 6:53 am
by mvndnburg
I have asked our infra guys to make the proposed changes.
Is selinux running on that system?
Yes, it is. And I know SELinux isn't formally supported by Nagios ;)
The settings are managed globally and no other host has the issue.

I think it's probably more related to locally installed (or not installed) software but I don't have enough info about the NCPA product itself and it looks like some of the Py code is compiled into binaries? Makes troubleshooting harder.

Re: NCPA hangs on request of 'API / services'

Posted: Fri May 05, 2017 8:19 am
by mvndnburg
tgriep wrote: Recreate the issue and post the ncpa_listener.log file again.
Here you go:
2017-05-05 14:24:35,678 7070 INFO stopped
2017-05-05 14:24:42,116 7149 INFO started
2017-05-05 14:24:42,118 7149 INFO Using SSL version TLSv1_2
2017-05-05 14:24:47,090 7149 DEBUG Initializing WebSocket
2017-05-05 14:24:47,090 7149 DEBUG Validating WebSocket request
2017-05-05 14:24:47,093 7149 INFO 10.132.164.185 - - [2017-05-05 14:24:47] "GET /gui/ HTTP/1.1" 302 550 0.003907
2017-05-05 14:24:47,298 7149 DEBUG Initializing WebSocket
2017-05-05 14:24:47,299 7149 DEBUG Validating WebSocket request
2017-05-05 14:24:47,322 7149 INFO 10.132.164.185 - - [2017-05-05 14:24:47] "GET /login HTTP/1.1" 200 3068 0.023706
2017-05-05 14:25:04,751 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:04,751 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:04,751 7149 DEBUG Can only upgrade connection if using GET method.
2017-05-05 14:25:04,754 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:04] "POST /login HTTP/1.1" 302 623 0.003362
2017-05-05 14:25:04,964 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:04,964 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:04,973 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:04] "GET /gui/ HTTP/1.1" 200 6766 0.009201
2017-05-05 14:25:08,294 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:08,295 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:08,338 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:08] "GET /gui/api HTTP/1.1" 200 51778 0.043481
2017-05-05 14:25:08,526 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:08,526 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:08,641 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:08] "GET /api/ HTTP/1.1" 200 9646 0.115238
2017-05-05 14:25:13,249 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:13,249 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:13,258 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:13] "GET /api/memory HTTP/1.1" 200 569 0.008625
2017-05-05 14:25:13,272 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:13,272 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:13,280 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:13] "GET /api/memory HTTP/1.1" 200 569 0.008116
2017-05-05 14:25:17,309 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:17,310 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:17,327 7149 ERROR Exception on /api/services [GET]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
File "/root/ncpa/agent/listener/server.py", line 154, in token_auth_decoration
File "/root/ncpa/agent/listener/server.py", line 941, in api
File "/root/ncpa/agent/listener/services.py", line 236, in walk
File "/root/ncpa/agent/listener/services.py", line 17, in wrapper
File "/root/ncpa/agent/listener/services.py", line 156, in get_services_via_initctl
File "/root/ncpa/agent/listener/services.py", line 17, in wrapper
File "/root/ncpa/agent/listener/services.py", line 200, in get_services_via_initd
NameError: global name 'logging' is not defined
2017-05-05 14:25:17,331 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:17] "GET /api/services HTTP/1.1" 500 3980 0.022053
2017-05-05 14:25:17,332 7149 DEBUG Initializing WebSocket
2017-05-05 14:25:17,332 7149 DEBUG Validating WebSocket request
2017-05-05 14:25:17,344 7149 ERROR Exception on /api/services [GET]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
File "/root/ncpa/agent/listener/server.py", line 154, in token_auth_decoration
File "/root/ncpa/agent/listener/server.py", line 941, in api
File "/root/ncpa/agent/listener/services.py", line 236, in walk
File "/root/ncpa/agent/listener/services.py", line 17, in wrapper
File "/root/ncpa/agent/listener/services.py", line 156, in get_services_via_initctl
File "/root/ncpa/agent/listener/services.py", line 17, in wrapper
File "/root/ncpa/agent/listener/services.py", line 200, in get_services_via_initd
NameError: global name 'logging' is not defined
2017-05-05 14:25:17,345 7149 INFO 10.132.164.185 - - [2017-05-05 14:25:17] "GET /api/services HTTP/1.1" 500 3980 0.013450

Re: NCPA hangs on request of 'API / services'

Posted: Fri May 05, 2017 9:11 am
by tgriep
I found a few issues on the NCPA Github site that are similar to what you are having and is could be a permission issue for the NCPA agent cannot read the init scripts to get the status of the running services.
One test is could you temporarily disable selinux and see if agent works after that.
If that doesn't work, please post it here.
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited