CHECK_NRPE: Received 0 bytes

[gormank]

I've noticed that when I get timeout alerts, that nsclient.log has log entries that correspond complaining about a file in a path that doesn't exist. I'm trying to figure out why I get these alerts consistently and why there's a nonsensical error message at the same time...

When this happens the check is retried after some seconds and succeeds.

Code: Select all

[Mon Nov 14 13:59:18 2016] SERVICE ALERT: hostname FS_Win_IO UNKNOWN SOFT 1 CHECK_NRPE: Received 0 bytes from daemon.
[Mon Nov 14 14:50:08 2016] SERVICE ALERT: hostname FS_Win_IO UNKNOWN SOFT 1 CHECK_NRPE: Received 0 bytes from daemon.
[Mon Nov 14 15:41:18 2016] SERVICE ALERT: hostname FS_Win_IO UNKNOWN SOFT 1 CHECK_NRPE: Received 0 bytes from daemon.
[Mon Nov 14 16:32:18 2016] SERVICE ALERT: hostname FS_Win_IO UNKNOWN SOFT 1 CHECK_NRPE: Received 0 bytes from daemon.
[Mon Nov 14 17:23:18 2016] SERVICE ALERT: hostname FS_Win_IO UNKNOWN SOFT 1 CHECK_NRPE: Received 0 bytes from daemon.
[Mon Nov 14 17:57:38 2016] SERVICE ALERT: hostname CPU_Win_Queue UNKNOWN SOFT 1 CHECK_NRPE: Received 0 bytes from daem
[Mon Nov 14 18:48:38 2016] SERVICE ALERT: hostname CPU_Win_Queue UNKNOWN SOFT 1 CHECK_NRPE: Received 0 bytes from daem
[Mon Nov 14 19:39:48 2016] SERVICE ALERT: hostname CPU_Win_Queue UNKNOWN SOFT 1 CHECK_NRPE: Received 0 bytes from daem
[Mon Nov 14 20:14:18 2016] SERVICE ALERT: hostname FS_Win_IO UNKNOWN SOFT 1 CHECK_NRPE: Received 0 bytes from daemon.
[Mon Nov 14 21:05:08 2016] SERVICE ALERT: hostname FS_Win_IO UNKNOWN SOFT 1 CHECK_NRPE: Received 0 bytes from daemon.

Code: Select all

2016-11-14 13:59:09: error:D:\source\nscp\include\socket/connection.hpp:149: Failed to send data: The file handle supplied is not valid
2016-11-14 14:50:19: error:D:\source\nscp\include\socket/connection.hpp:149: Failed to send data: The file handle supplied is not valid
2016-11-14 15:41:31: error:D:\source\nscp\include\socket/connection.hpp:149: Failed to send data: The file handle supplied is not valid
2016-11-14 16:32:34: error:D:\source\nscp\include\socket/connection.hpp:149: Failed to send data: The file handle supplied is not valid
2016-11-14 17:23:36: error:D:\source\nscp\include\socket/connection.hpp:149: Failed to send data: The file handle supplied is not valid
2016-11-14 17:57:41: error:D:\source\nscp\include\socket/connection.hpp:149: Failed to send data: The file handle supplied is not valid
2016-11-14 18:48:55: error:D:\source\nscp\include\socket/connection.hpp:149: Failed to send data: The file handle supplied is not valid
2016-11-14 19:40:05: error:D:\source\nscp\include\socket/connection.hpp:149: Failed to send data: The file handle supplied is not valid
2016-11-14 20:14:14: error:D:\source\nscp\include\socket/connection.hpp:149: Failed to send data: The file handle supplied is not valid
2016-11-14 21:05:33: error:D:\source\nscp\include\socket/connection.hpp:149: Failed to send data: The file handle supplied is not valid

[avandemore]

The error is generated by NSClient which we don't write or maintain so you should try asking there.

However one thing that immediately jumps out is the path:
error:D:\source\nscp\include\socket/connection.hpp:149
One slash in there is not like the others.

Please upload the NSClient configuration file and we can look at it. What version is it?

[gormank]

Great. My management is having a fit over that response. Are you trying to drive customers away?

The source dir doesn't exist, and the connection.hpp file doesn't exist, so the error isn't very informative, but may lead to a solution.

[avandemore]

We are trying to provide you with the most information possible so you can resolve your issue. As stated:

The error is generated by NSClient which we don't write or maintain

I also asked you to provide your configuration so we can check it's contents. Even though we aren't the best place to ask, we can give a best effort.

[gormank]

Sorry,
The version is 0.4.3.143.
I did some searching and it seems the command is timing out on the nsclient side. The nsclient site after more searching said they had updaed the error message to make it more sensible.

Code: Select all

# If you want to fill this file with all avalible options run the following command:
#   nscp settings --generate --add-defaults --load-all
# If you want to activate a module and bring in all its options use:
#   nscp settings --activate-module <MODULE NAME> --add-defaults
# For details run: nscp settings --help

; Undocumented section
[/settings/default]

; PASSWORD - Password used to authenticate against server
password = removed

; ALLOWED HOSTS - A coma separated list of allowed hosts. You can use netmasks (/ syntax) or * to create ranges.
allowed hosts = 10.133.134.0/24,10.136.243.0/24

; Undocumented section
[/settings/NRPE/server]
allow arguments = true
allow nasty characters = true

; VERIFY MODE - Comma separated list of verification flags to set on the SSL socket.  default-workarounds	Various workarounds for what I understand to be broken ssl implementations no-sslv2	Do not use the SSLv2 protocol. no-sslv3	Do not use the SSLv3 protocol. no-tlsv1	Do not use the TLSv1 protocol. single-dh-use	Always create a new key when using temporary/ephemeral DH parameters. This option must be used to prevent small subgroup attacks, when the DH parameters were not generated using "strong" primes (e.g. when using DSA-parameters).   
ssl options = no-sslv2,no-sslv3

; VERIFY MODE - Comma separated list of verification flags to set on the SSL socket.  none	The server will not send a client certificate request to the client, so the client will not send a certificate. peer	The server sends a client certificate request to the client and the certificate returned (if any) is checked. fail-if-no-cert	if the client did not return a certificate, the TLS/SSL handshake is immediately terminated. This flag must be used together with peer. peer-cert	Alias for peer and fail-if-no-cert. workarounds	Various bug workarounds. single	Always create a new key when using tmp_dh parameters. client-once	Only request a client certificate on the initial TLS/SSL handshake. This flag must be used together with verify-peer   
verify mode = none

; ALLOW INSECURE CHIPHERS and ENCRYPTION - Only enable this if you are using legacy check_nrpe client.
insecure = true

; Undocumented section
[/modules]

; NRPEServer - A server that listens for incoming NRPE connection and processes incoming requests.
NRPEServer = 1

; CheckSystem - Various system related checks, such as CPU load, process state, service state memory usage and PDH counters.
CheckSystem = 1

; NSClientServer - A server that listens for incoming check_nt connection and processes incoming requests.
NSClientServer = 1

; CheckExternalScripts - Execute external scripts
CheckExternalScripts = 1

; CheckHelpers - Various helper function to extend other checks.
CheckHelpers = 1

; NSCAClient - NSCA client can be used both from command line and from queries to submit passive checks via NSCA
NSCAClient = 1

; CheckEventLog - Check for errors and warnings in the event log.
CheckEventLog = 1

; CheckNSCP - Use this module to check the healt and status of NSClient++ it self
CheckNSCP = 1

; CheckDisk - CheckDisk can check various file and disk related things.
CheckDisk = 1

; A list of templates for wrapped scripts.
%SCRIPT% will be replaced by the actual script an %ARGS% will be replaced by any given arguments.
[/settings/external scripts/wrappings]

; POWERSHELL WRAPPING - 
ps1 = cmd /c echo scripts\\%SCRIPT% %ARGS%; exit($lastexitcode) | powershell.exe -command -

; BATCH FILE WRAPPING - 
bat = scripts\\%SCRIPT% %ARGS%

; VISUAL BASIC WRAPPING - 
vbs = cscript.exe //T:115 //NoLogo scripts\\lib\\wrapper.vbs %SCRIPT% %ARGS%

[/settings/external scripts]
allow arguments = true
allow nasty characters = true
allow arguments = 1

[/settings/external scripts/scripts]
check_mcafee_datdate = cscript.exe //T:115 //NoLogo scripts\\check_mcafee_datdate.vbs 14 30
virus_detected = cscript.exe //T:115 //NoLogo scripts\\Virus_detected.vbs
check_process2 = cscript.exe //T:115 //NoLogo scripts\\check_process2.vbs $ARGS"$
check_cpu_queue = cscript.exe //T:115 //NoLogo scripts\\check_cpu_queue.vbs $ARGS"$
winfsio = cscript.exe //T:115 //NoLogo scripts\\winFSIO.vbs
check_networkinterfaces = scripts\\check_networkinterfaces.bat

; A list of aliases available.  An alias is an internal command that has been predefined to provide a single command without arguments. Be careful so you don't create loops (ie check_loop=check_a, check_a=check_loop)
[/settings/external scripts/alias]

; alias_volumes_loose - Alias for alias_volumes_loose. To configure this item add a section called: /settings/external scripts/alias/alias_volumes_loose
alias_volumes_loose = check_drivesize

; alias_volumes - Alias for alias_volumes. To configure this item add a section called: /settings/external scripts/alias/alias_volumes
alias_volumes = check_drivesize

; alias_sched_all - Alias for alias_sched_all. To configure this item add a section called: /settings/external scripts/alias/alias_sched_all
alias_sched_all = check_tasksched show-all "syntax=${title}: ${exit_code}" "crit=exit_code ne 0"

; alias_process_stopped - Alias for alias_process_stopped. To configure this item add a section called: /settings/external scripts/alias/alias_process_stopped
alias_process_stopped = check_process "process=$ARG1$" "crit=state != 'stopped'"

; alias_service - Alias for alias_service. To configure this item add a section called: /settings/external scripts/alias/alias_service
alias_service = check_service

; alias_process_hung - Alias for alias_process_hung. To configure this item add a section called: /settings/external scripts/alias/alias_process_hung
alias_process_hung = check_process "filter=is_hung" "crit=count>0"

; alias_process_count - Alias for alias_process_count. To configure this item add a section called: /settings/external scripts/alias/alias_process_count
alias_process_count = check_process "process=$ARG1$" "warn=count > $ARG2$" "crit=count > $ARG3$"

; alias_process - Alias for alias_process. To configure this item add a section called: /settings/external scripts/alias/alias_process
alias_process = check_process "process=$ARG1$" "crit=state != 'started'"

; alias_mem - Alias for alias_mem. To configure this item add a section called: /settings/external scripts/alias/alias_mem
alias_mem = check_memory

; alias_file_size - Alias for alias_file_size. To configure this item add a section called: /settings/external scripts/alias/alias_file_size
alias_file_size = check_files "path=$ARG1$" "crit=size > $ARG2$" "top-syntax=${list}" "detail-syntax=${filename] ${size}" max-dir-depth=10

; alias_disk - Alias for alias_disk. To configure this item add a section called: /settings/external scripts/alias/alias_disk
alias_disk = check_drivesize

; alias_cpu_ex - Alias for alias_cpu_ex. To configure this item add a section called: /settings/external scripts/alias/alias_cpu_ex
alias_cpu_ex = check_cpu "warn=load > $ARG1$" "crit=load > $ARG2$" time=5m time=1m time=30s

; alias_file_age - Alias for alias_file_age. To configure this item add a section called: /settings/external scripts/alias/alias_file_age
alias_file_age = check_files "path=$ARG1$" "crit=written > $ARG2$" "top-syntax=${list}" "detail-syntax=${filename] ${written}" max-dir-depth=10

; alias_cpu - Alias for alias_cpu. To configure this item add a section called: /settings/external scripts/alias/alias_cpu
alias_cpu = check_cpu

; alias_event_log - Alias for alias_event_log. To configure this item add a section called: /settings/external scripts/alias/alias_event_log
;alias_event_log = check_eventlog
alias_event_log = CheckEventLog file=security file=application MaxWarn=1 MaxCrit=1 "filter=level in ('error', 'Audit Failure') AND generated > -5m"

; alias_service_ex - Alias for alias_service_ex. To configure this item add a section called: /settings/external scripts/alias/alias_service_ex
alias_service_ex = check_service "exclude=Net Driver HPZ12" "exclude=Pml Driver HPZ12" exclude=stisvc

; alias_up - Alias for alias_up. To configure this item add a section called: /settings/external scripts/alias/alias_up
alias_up = check_uptime

; alias_disk_loose - Alias for alias_disk_loose. To configure this item add a section called: /settings/external scripts/alias/alias_disk_loose
alias_disk_loose = check_drivesize

; alias_sched_task - Alias for alias_sched_task. To configure this item add a section called: /settings/external scripts/alias/alias_sched_task
alias_sched_task = check_tasksched show-all "filter=title eq '$ARG1$'" "detail-syntax=${title} (${exit_code})" "crit=exit_code ne 0"

; alias_sched_long - Alias for alias_sched_long. To configure this item add a section called: /settings/external scripts/alias/alias_sched_long
alias_sched_long = check_tasksched "filter=status = 'running'" "detail-syntax=${title} (${most_recent_run_time})" "crit=most_recent_run_time < -$ARG1$"

[avandemore]

See http://docs.nsclient.org/0.4.3/faq/index.html sections 1.7 and 1.10.
Add this to the nsclient configuration and restart it:

Code: Select all

[/settings/log]
file name = nsclient.log
level = debug

After a troublesome event has occurred, please supply the nsclient.log file.

[tgriep]

Can you login to the XI server, run the following commands and post the output?

Code: Select all

nmap -Pn -p 5666 xxx.xxx.xxx.xxx 
/usr/local/nagios/libexec/check_nrpe -H xxx.xxx.xxx.xxx

Replace xxx.xxx.xxx.xxx with the IP address of the host.

Also, can you post how the checks are configured in XI that are giving you issues?
Thanks.

[gormank]

Nmap won't tell anything because the commands fail once in a while, and are therefore valid.
I didn't write the VBS, since I haven't written anything in VB in 20 years... I can still read it but it hurts.

Code: Select all

# nmap Pn -p 5666 10.133.134.58

Starting Nmap 5.51 ( http://nmap.org ) at 2016-11-15 22:05 UTC
Failed to resolve given hostname/IP: Pn.  Note that you can't use '/mask' AND '1-4,7,100-' style IP ranges
Failed to find device eth3 which was referenced in /proc/net/route
Nmap scan report for txslm2mwapp005 (10.133.134.58)
Host is up (0.00035s latency).
rDNS record for 10.133.134.58: txslm2mwapp005.m2m.local
PORT     STATE SERVICE
5666/tcp open  nrpe
MAC Address: 9C:B6:54:89:35:EC (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 0.12 seconds

Code: Select all

define service {
        service_description             FS_Win_IO
        use                             1VZW_default_service
        hostgroup_name                  1VZW_Windows_Physical,1VZW_Windows_Virtual
        display_name                    FS_Win_IO
        check_command                   check_nrpe!winfsio!!!!!!!
        register                        1
        }

Code: Select all

define service {
        service_description             CPU_Win_Queue
        use                             1VZW_default_service
        hostgroup_name                  1VZW_Windows_Physical,1VZW_Windows_Virtual
        check_command                   check_nrpe!check_cpu_queue!-a 4 6!!!!!!
        _xiwizard                       windowsserver
        register                        1
        }

# cat /usr/local/nagios/windows/scripts/winFSIO.vbs

Code: Select all

strComputer = "."
Set objWMIService = GetObject("winmgmts:" _
    & "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
set objRefresher = CreateObject("WbemScripting.SWbemRefresher")
Set colDisks = objRefresher.AddEnum _
    (objWMIService, "win32_perfformatteddata_perfdisk_logicaldisk"). _
        objectSet
objRefresher.Refresh
For Each objDisk in colDisks
        If objDisk.Name = "_Total" Then
        Wscript.Echo "Total Summary: " & objDisk.SplitIOPerSec & " IO/s, Write/s " & objDisk.DiskWritesPerSec & " Reads/s: " & objDisk.DiskReadsPerSec & " Transfers/s: " & objDisk.DiskTransfersPerSec & "| split_total=" & objDisk.SplitIOPerSec & ";;; tps_total=" & objDisk.DiskTransfersPerSec & ";;;total_writes/s=" & objDisk.DiskWritesPerSec & ";;;total_read/s=" & objDisk.DiskReadsPerSec
        End if
Next

# cat /usr/local/nagios/windows/scripts/check_cpu_queue.vbs

Code: Select all

' Monitor Operating System Performance
' CPU Queue monitor
' Tests the cpu queuelength and if the queue is over 4 more than 6 times than warn

If NOT WScript.Arguments.Count = 2 Then
    WScript.Echo "This script has two arguments. 1)CPU queue threshold ; 2) How many times over the threshold"
        WScript.Quit 3
Else
        TestQueueOF =  CInt(WScript.Arguments.Item(0))
        TestQueueXtimes =  CInt(WScript.Arguments.Item(1))
end if
strComputer = "."
Set objWMIService = GetObject("winmgmts:" _
    & "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
set objRefresher = CreateObject("WbemScripting.SWbemRefresher")
Set colItems = objRefresher.AddEnum _
    (objWMIService, "Win32_PerfFormattedData_PerfOS_System").objectSet
objRefresher.Refresh
ReturnCode = 0
ProcQLength = 0
path = "c:\temp\cpucounter.txt"
Set fso = CreateObject("Scripting.FileSystemObject")
FileExists = fso.FileExists(path)
count = 0
'WScript.Echo "testqueue=" & TestQueueOF
'WScript.Echo "count should= " & TestQueueXtimes
For Each objItem in colItems
                'Wscript.Echo "Processor Queue Length: " & _
                '       objItem.ProcessorQueueLength
                        ProcQLength = objItem.ProcessorQueueLength
                        objRefresher.Refresh
                        'WScript.Echo ProcQLength
                        If TestQueueOF < ProcQLength  Then
                                If FileExists Then
                                        set fsofile = fso.OpenTextFile(path)
                                        count = CInt(fsofile.ReadAll)
                                        count = count + 1
                                        IF count > TestQueueXtimes Then
                                                ReturnCode = 1
                                        End if
                                        Set fsofile = fso.CreateTextFile(path,True)
                                        fsofile.Write count
                                        fsofile.close
                                Else
                                        'WScript.Echo "should be here"
                                        count = 1
                                        Set fsofile = fso.CreateTextFile(path,True)
                                        fsofile.Write count
                                        fsofile.close

                                End IF
                        Else
                        IF FileExists Then fso.DeleteFile(path)

                        WScript.Echo "OK - CPU queue length = " & ProcQLength & " | CPUQueue=" & ProcQLength
                        WScript.Quit 0
                        End If
Next
IF ReturnCode Then
        WScript.Echo "Warning - CPU queue length = " & ProcQLength & " | CPUQueue=" & ProcQLength
        WScript.Quit 1
Else
        WScript.Echo "OK - CPU queue length = " & ProcQLength & " | CPUQueue=" & ProcQLength
        WScript.Quit 0

[avandemore]

From a shell on the windows box what does cscript.exe //T:115 //NoLogo scripts\\winFSIO.vbs do?

[gormank]

C:\Program Files\NSClient++>cscript.exe /T:115 /NoLogo scripts\winFSIO.vbs
Total Summary: 143 IO/s, Write/s 287 Reads/s: 54 Transfers/s: 342| split_total=143;;; tps_total=342;;;total_writes/s=287
;;;total_read/s=54

C:\Program Files\NSClient++>cscript.exe /T:115 /NoLogo scripts\winFSIO.vbs
Total Summary: 0 IO/s, Write/s 0 Reads/s: 0 Transfers/s: 0| split_total=0;;; tps_total=0;;;total_writes/s=0;;;total_read
/s=0