Swap failure post NCPA 2.4.1

[shoreypu]

Hello,

We're attempting to upgrade from NCPA v2.40 to v2.4.1, and our checks for swap are failing. All other NCPA checks are working. I checked the agent and it shows the following errors.
{ "swap":
{ "total": "Error retrieving child: 'PdhAddEnglishCounterW failed. Performance counters may be disabled.'",
"free": "Error retrieving child: 'PdhAddEnglishCounterW failed. Performance counters may be disabled.'",
"percent": "Error retrieving child: 'PdhAddEnglishCounterW failed. Performance counters may be disabled.'",
"used": "Error retrieving child: 'PdhAddEnglishCounterW failed. Performance counters may be disabled.'" } }

If we remove 2.4.1 and reinstall 2.4.0, the checks/metrics start working/report successfully. I didn't find this as a bug with 2.4.1, however, I did see a similar report for NCPA v3.0 Beta.

We are currently testing on Windows Server 2019.

Thanks in advance....John

[jmichaelson]

Hi John, Can you find a relevant snippet of the log (or the whole works if necessary)? A quick compare on GitHub of the code changes doesn't show anything that should be causing the issue, so the logs would be helpful.

They can be found in:

Code: Select all

C:\Program Files (x86)\Nagios\NCPA\var\log

[shoreypu]

Attaching the ncpa_listener.log

Here is the most recent entry.
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\flask\app.py", line 1817, in wsgi_app
File "C:\Python27\lib\site-packages\flask\app.py", line 1477, in full_dispatch_request
File "C:\Python27\lib\site-packages\flask\app.py", line 1381, in handle_user_exception
File "C:\Python27\lib\site-packages\flask\app.py", line 1475, in full_dispatch_request
File "C:\Python27\lib\site-packages\flask\app.py", line 1461, in dispatch_request
File "C:\Users\Administrator\Documents\repos\ncpa\agent\listener\server.py", line 310, in token_auth_decoration
File "C:\Users\Administrator\Documents\repos\ncpa\agent\listener\server.py", line 1121, in api
File "C:\Users\Administrator\Documents\repos\ncpa\agent\listener\nodes.py", line 99, in run_check
File "C:\Users\Administrator\Documents\repos\ncpa\agent\listener\nodes.py", line 284, in get_values
File "C:\Users\Administrator\Documents\repos\ncpa\agent\listener\psapi.py", line 158, in <lambda>
File "C:\Python27\lib\site-packages\psutil\__init__.py", line 1987, in swap_memory
File "C:\Python27\lib\site-packages\psutil\_pswindows.py", line 257, in swap_memory
RuntimeError: PdhAddEnglishCounterW failed. Performance counters may be disabled.

[namdosan1409]

The factory default for this setting is 120 seconds. Failed Pings Before Reboot. The device will be rebooted after this number of ping failures. car games

[swolf]

Attaching the ncpa_listener.log

Here is the most recent entry.
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\flask\app.py", line 1817, in wsgi_app
File "C:\Python27\lib\site-packages\flask\app.py", line 1477, in full_dispatch_request
File "C:\Python27\lib\site-packages\flask\app.py", line 1381, in handle_user_exception
File "C:\Python27\lib\site-packages\flask\app.py", line 1475, in full_dispatch_request
File "C:\Python27\lib\site-packages\flask\app.py", line 1461, in dispatch_request
File "C:\Users\Administrator\Documents\repos\ncpa\agent\listener\server.py", line 310, in token_auth_decoration
File "C:\Users\Administrator\Documents\repos\ncpa\agent\listener\server.py", line 1121, in api
File "C:\Users\Administrator\Documents\repos\ncpa\agent\listener\nodes.py", line 99, in run_check
File "C:\Users\Administrator\Documents\repos\ncpa\agent\listener\nodes.py", line 284, in get_values
File "C:\Users\Administrator\Documents\repos\ncpa\agent\listener\psapi.py", line 158, in <lambda>
File "C:\Python27\lib\site-packages\psutil\__init__.py", line 1987, in swap_memory
File "C:\Python27\lib\site-packages\psutil\_pswindows.py", line 257, in swap_memory
RuntimeError: PdhAddEnglishCounterW failed. Performance counters may be disabled.

This looked like something I'd seen on the Github issue list, but then I checked and you also posted the issue over there. I posted this log over on that issue as well. I wasn't able to reproduce this on either of the Windows devices I have on hand, so this will probably require some digging with the maintainer of that project.

[bbahn]

This issue is because the psutil library can't find the \\Paging File(_Total)\\% Usage performance counter.

The registry item Counter under the Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib\009 subkey should contain your base performance counters, of which the paging file usage should be a part. Given that your machine is unable to find this counter, it is likely that these are corrupted in some manner. You can try rebuilding your performance counters with:

Code: Select all

cd c:\windows\system32
lodctr /R

If the problem persists, your performance counters may be disabled. You can check performance counters by:

1. Open the Registry Editor (regedit.exe).
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfProc\Performance.
3. If there is a Disable Performance Counters value, set it to 0.
4. Restart your computer.

[shoreypu]

It looks like we needed a combination of rebuilding the performance counters and setting Disable Performance Counters in the registry to resolve the issue. I'm not sure if simply rebuilding the performance counts followed by a reboot will resolve this as well. If I can get another server to fail, I will test that option and report back. We tested upgrading on another Server 2019 server, but that one worked post-upgrade.

Thanks for all your help in resolving this. It will now allow us to move ahead with upgrading all of our agents.