check_nrpe -> check_pdh issue

[stefanmadsen]

Hi,

so today we discovered that some of our counters are not returning the actual values for nagios to proces.

the following configuration in nagios Xi for a check_pdh check is as following:
define service {
host_name pfanpsql06
service_description SQL Memory Manager - SQL Cache Memory KB
check_command check_nrpe_nossl_3args_4kpayload!check_pdh! -a "counter:Memory Manager SQL Cache Memory (KB)=\\MSSQL\$$MSSQL2014:Memory Manager\SQL Cache Memory(KB)"!!!!!!
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
register 1
}

where the command check_nrpe_nossl_3args_4kpayload is defined by:
define command {
command_name check_nrpe_nossl_3args_4kpayload
command_line $USER1$/check_nrpe -2 -P 4096 -H $HOSTADDRESS$ -n -t 30 -c $ARG1$ $ARG2$ $ARG3$
}

if we call this method directly from the Linux box we get (note that we have to remove the single $ break in 'MSSQL\$$'):
/usr/local/nagios/libexec/check_nrpe -2 -P 4096 -H pfanpsql06 -n -t 30 -c check_pdh -a "counter:Memory Manager SQL Cache Memory (KB)=\\MSSQL\$MSSQL2014:Memory Manager\SQL Cache Memory (KB)"
OK: Memory Manager SQL Cache Memory (KB) = 86888|'Memory Manager SQL Cache Memory (KB)_value'=86888;0;0

if we attempt to call this method as it is defined via nagiosxi CCM - run check command, we get:
[[email protected] ~]$ /usr/local/nagios/libexec/check_nrpe -2 -P 4096 -H pfanpsql06 -n -t 30 -c check_pdh -a "counter:Memory Manager SQL Cache Memory (KB)=\\MSSQL\$$MSSQL2014:Memory Manager\SQL Cache Memory (KB)" Failed to poll counter: Memory Manager SQL Cache Memory (KB) PdhAddCounter failed: c0000bb8: The specified object was not found on the computer.
- which is caused by the double $, hence if we use a single $ then it works fine.

now the funny bit.
- if we save the command with double $$ it actually runs, but ALWAYS returns 0 !!!!
- if we save the command with a single $ it fails (unable to find counter)

yet if we call the command at the same time directly from the console we get actual values - that is other than 0.
what gives?

note that we are running NRPE v3
apparently its only specific counters we seem to have issues with, and it could be related to the $$ breaks, or possibly the check_pdh function returning something nagios cant properly consume. any thoughts?

[stefanmadsen]

had an idea, perhaps it was the formatting of the return values ?
so tried to fiddle with it the detaul syntax and perf config to no avail

for instance:
/usr/local/nagios/libexec/check_nrpe -2 -P 4096 -H pfanpsql06 -n -t 30 -c check_pdh -a "counter:Memory Manager SQL Cache Memory (KB)=\MSSQL\$MSSQL2014:Memory Manager\SQL Cache Memory (KB)" 'perf-config=*(unit:kb)' 'detail-syntax=${alias} = ${value_f}'

yields:
OK: Memory Manager SQL Cache Memory (KB) = 148304|'Memory Manager SQL Cache Memory (KB)_value'=148304kb;0;0

just fine, but still in nagios the value is simply 0

[stefanmadsen]

and to add slightly more confusion - the VERY similar counter
/usr/local/nagios/libexec/check_nrpe -2 -P 4096 -H pfanpsql06 -n -t 30 -c check_pdh -a "counter:SQLDB Data File size(KB)=\\MSSQL\$MSSQL2014:Databases(_Total)\\Data File(s) Size (KB)"

yields:
OK: SQLDB Data File size(KB) = 3.93845e+009|'SQLDB Data File size(KB)_value'=3938450368;0;0

and the value is matched inside nagiosxi correctly as stated in this line.

[tgriep]

When an argument for the check_nrpe command has a dollar sign in it, they sometimes can be troublesome when they need to be escaped.
What I have found to work consistently is to escape the $ with another $ in double quotes like the example below.

Code: Select all

"counter:Memory Manager SQL Cache Memory (KB)=\MSSQL$"$"MSSQL2014:Memory Manager\SQL Cache Memory (KB)"

Also, you may not have to escape the backslash in the command as it may not be needed.

Another thing, the Test Check Command button may not work for all commands and may return false information.
When it it used, it tries to add escaping to the command and sometimes it doesn't do it correctly and creates an false error.

[stefanmadsen]

thanks for the idea, but doesnt really seem to Work :

i tried various permutations:
check_nrpe_nossl_3args_4kpayload!check_pdh! -a "counter:Memory Manager SQL Cache Memory (KB)=\MSSQL\$"$"MSSQL2014:Memory Manager\SQL Cache Memory (KB)"!!!!!!
check_nrpe_nossl_3args_4kpayload!check_pdh! -a "counter:Memory Manager SQL Cache Memory (KB)=\MSSQL$"$"MSSQL2014:Memory Manager\SQL Cache Memory (KB)"!!!!!!
check_nrpe_nossl_3args_4kpayload!check_pdh! -a "counter:Memory Manager SQL Cache Memory (KB)=\MSSQL"$"$MSSQL2014:Memory Manager\SQL Cache Memory (KB)"!!!!!!
check_nrpe_nossl_3args_4kpayload!check_pdh! -a "counter:Memory Manager SQL Cache Memory (KB)=\MSSQL$'$'MSSQL2014:Memory Manager\SQL Cache Memory (KB)"!!!!!!
check_nrpe_nossl_3args_4kpayload!check_pdh! -a "counter:Memory Manager SQL Cache Memory (KB)=\MSSQL\$'$'MSSQL2014:Memory Manager\SQL Cache Memory (KB)"!!!!!!

hell - even with more breaking...
check_nrpe_nossl_3args_4kpayload!check_pdh! -a "counter:Memory Manager SQL Cache Memory (KB)=\MSSQL\$\$MSSQL2014:Memory Manager\SQL Cache Memory (KB)"!!!!!!

still reporting 0 in nagiosxi

[stefanmadsen]

note that :
check_nrpe_nossl_3args_4kpayload!check_pdh! -a "counter:Memory Manager SQL Cache Memory (KB)=\MSSQL\$"$"MSSQL2014:Memory Manager\SQL Cache Memory (KB)"!!!!!!

plays nice with nagiosxi's test command. - but still yielding 0 as result when the check is run via scheduler

Udklip.JPG

Udklip1.JPG

[tgriep]

Can you login to the Windows server that is hosting MSSQL server as an administrator, run perfmon to open up the performance monitor application and add that counter to the Performance Monitor screen.
Verify that the counter has valid data and then right click the counter and select Properties, screen capture the counter and post it here.

SQL-Memory.PNG

What version of NSClient++ is installed on the Windows server?
Can you post your nsclient.ini file from that server?

[WillemDH]

Fyi, for me \$$ worked in the check_service command
https://support.nagios.com/forum/viewto ... it=+dollar

But I guess it isn't the same as check_pdh.

[tgriep]

Thanks @WillemDH for the help.

The $"$" works for me.

Code: Select all

check_pdh!-a "counter:Memory Manager SQL Cache Memory (KB)=\MSSQL$"$"SQLEXPRESS:Memory Manager\SQL Cache Memory (KB)"

SQL-Memory-XI.PNG

[stefanmadsen]

I will have to get back to you regarding the RDP / counter visual confirmation - as i dont have access to this server.

however, as you can plainly see above, the counter exists, if i pull it directly from the nagios/Linux commandline, - hell we're even getting it now via the builtin nagiosxi command tester.

we are running nscp 5.0.46 and running NRPEv3

Code: Select all

# If you want to fill this file with all available options run the following command:
#   nscp settings --generate --add-defaults --load-all
# If you want to activate a module and bring in all its options use:
#   nscp settings --activate-module <MODULE NAME> --add-defaults
# For details run: nscp settings --help


; TODO
[/settings/default]

; Undocumented key
password = ...

; Undocumented key
allowed hosts = 127.0.0.1,10.225.99.25,10.225.99.26

; Section for NRPE (NRPEServer.dll) (check_nrpe) protocol options.
[/settings/NRPE/server]

payload length=4096

; COMMAND ARGUMENT PROCESSING - This option determines whether or not the we will allow clients to specify arguments to commands that are executed.
allow arguments = 1

; COMMAND ALLOW NASTY META CHARS - This option determines whether or not the we will allow clients to specify nasty (as in |`&><'"\[]{}) characters in arguments.
allow nasty characters = 1

; PORT NUMBER - Port to use for NRPE.
port = 5666

; ENABLE SSL ENCRYPTION - This option controls if SSL should be enabled.
use ssl = 0

; ALLOW INSECURE CHIPHERS and ENCRYPTION - Only enable this if you are using legacy check_nrpe client.
insecure = false

; EXTENDED RESPONSE - Send more then 1 return packet to allow response to go beyond payload size (requires modified client if legacy is true this defaults to false).
extended response = 1

verify mode = none

[/settings/NRPE/client/targets/default]

; RETRIES - Number of times to retry sending.
;retries = 3

; ENABLE SSL ENCRYPTION - This option controls if SSL should be enabled.
;use ssl = 0

; TIMEOUT - Timeout when reading/writing packets to/from sockets.
;timeout = 30

[/settings/external scripts/scripts/default]

; IGNORE PERF DATA - Do not parse performance data from the output
;ignore perfdata = 0

[/modules]

; CheckDisk - CheckDisk can check various file and disk related things.
CheckDisk = 1

; CheckEventLog - Check for errors and warnings in the event log.
CheckEventLog = 1

; CheckExternalScripts - Execute external scripts
CheckExternalScripts = 1

; CheckHelpers - Various helper function to extend other checks.
CheckHelpers = 1

; CheckNSCP - Use this module to check the healt and status of NSClient++ it self
CheckNSCP = 1

; CheckSystem - Various system related checks, such as CPU load, process state, service state memory usage and PDH counters.
CheckSystem = enabled

; CheckWMI - Check status via WMI
CheckWMI = 1

; NRPEServer - A server that listens for incoming NRPE connection and processes incoming requests.
NRPEServer = 1

; NSClientServer - A server that listens for incoming check_nt connection and processes incoming requests.
NSClientServer = 1

; Section for NSClient (NSClientServer.dll) (check_nt) protocol options.
[/settings/NSClient/server]

; PERFORMANCE DATA - Send performance data back to Nagios (set this to 0 to remove all performance data).
performance data = 1

; PORT NUMBER - Port to use for check_nt.
port = 12489

; ENABLE SSL ENCRYPTION - This option controls if SSL should be enabled.
use ssl = 0