CHECK_NRPE: Received 0 bytes from daemon

[sajid4u2c]

Hi All,

I have around 10 servers for which Nagios monitoring was working well till yesterday.
All of sudden, the sevrice checks are being failed with the return message as "

CHECK_NRPE: Received 0 bytes from daemon .Check the remote server logs for error messages"

When I ran just the basic command of NRPE test from NAgios server, the output is also same.

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H hostaddress

CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages

When I check the nsclient.log file, it is empty.

I checked for the port 5666 at destination server from nagios server, and it is working fine.

PORT 5666 OK: hostaddress/5666 is responding

Iam not sure if this is to be related to something that has been done at network side.

Nagios version : 5.2.7
Check_nrpe : 2.12
NSclient++ : 0.3.9.322

Could you please suggest If i missed any more troubleshooting steps.

[cdienger]

Can you provide the nsclient.ini file from one of the servers having an issue? Make sure to remove any sensitive information(like passwords) from it.

[sajid4u2c]

Hi cdienger,

Please find the NSCLient++.ini file

Code: Select all

[modules]
NRPEListener.dll
CheckSystem.dll
CheckDisk.dll
;# NSCLIENT++ MODULES
;# A list with DLLs to load at startup.
;  You will need to enable some of these for NSClient++ to work.
; ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !
; *                                                               *
; * N O T I C E ! ! ! - Y O U   H A V E   T O   E D I T   T H I S *
; *                                                               *
; ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !
;FileLogger.dll
;CheckSystem.dll
;CheckDisk.dll
;NSClientListener.dll
;NRPEListener.dll
;SysTray.dll
;CheckEventLog.dll
;CheckHelpers.dll
;CheckWMI.dll
;CheckNSCP.dll
;
; Script to check external scripts and/or internal aliases.
;CheckExternalScripts.dll
;
; NSCA Agent if you enable this NSClient++ will talk to NSCA hosts repeatedly (so dont enable unless you want to use NSCA)
;NSCAAgent.dll
;
; LUA script module used to write your own "check deamon".
;LUAScript.dll
;
; RemoteConfiguration IS AN EXTREM EARLY IDEA SO DONT USE FOR PRODUCTION ENVIROMNEMTS!
;RemoteConfiguration.dll
; Check other hosts through NRPE extreme beta and probably a bit dangerous! :)
;NRPEClient.dll
; Extreamly early beta of a task-schedule checker
;CheckTaskSched.dll

[crash]
; Archive crash dump files if a crash is detected
;archive=1

; Submit crash reports to a crash report server (this overrrides archive)
;submit=0

; Restart service if a crash is detected
restart=1

[Settings]
;# OBFUSCATED PASSWORD
;  This is the same as the password option but here you can store the password in an obfuscated manner.
;  *NOTICE* obfuscation is *NOT* the same as encryption, someone with access to this file can still figure out the 
;  password. Its just a bit harder to do it at first glance.
;obfuscated_password=Jw0KAUUdXlAAUwASDAAB
;
;# PASSWORD
;  This is the password (-s) that is required to access NSClient remotely. If you leave this blank everyone will be able to access the daemon remotly.
;password=secret-password
;
;# ALLOWED HOST ADDRESSES
;  This is a comma-delimited list of IP address of hosts that are allowed to talk to the all daemons.
;  If leave this blank anyone can access the deamon remotly (NSClient still requires a valid password).
;  The syntax is host or ip/mask so 192.168.0.0/24 will allow anyone on that subnet access
;allowed_hosts=127.0.0.1/32
;
;# USE THIS FILE
;  Use the INI file as opposed to the registry if this is 0 and the use_reg in the registry is set to 1 
;  the registry will be used instead.
use_file=1
allowed_hosts="here we have multiple server entries separated with ,"
;
; # USE SHARED MEMORY CHANNELS
;  This is the "new" way for using the system tray based on an IPC framework on top shared memmory channels and events.
;  It is brand new and (probably has bugs) so dont enable this unless for testing!
;  If set to 1 shared channels will be created and system tray icons created and such and such...
;shared_session=0


[log]
;# LOG DEBUG
;  Set to 1 if you want debug message printed in the log file (debug messages are always printed to stdout when run with -test)
;debug=1
;
;# LOG FILE
;  The file to print log statements to
;file=nsclient.log
;
;# LOG DATE MASK
;  The format to for the date/time part of the log entry written to file.
;date_mask=%Y-%m-%d %H:%M:%S
;
;# LOG ROOT FOLDER
;  The root folder to use for logging.
;  exe = the folder where the executable is located
;  local-app-data = local application data (probably a better choice then the old default)
;root_folder=exe


[NSClient]
;# ALLOWED HOST ADDRESSES
;  This is a comma-delimited list of IP address of hosts that are allowed to talk to NSClient deamon.
;  If you leave this blank the global version will be used instead.
;allowed_hosts=
;
;# NSCLIENT PORT NUMBER
;  This is the port the NSClientListener.dll will listen to.
;port=12489
;
;# BIND TO ADDRESS
;  Allows you to bind server to a specific local address. This has to be a dotted ip adress not a hostname.
;  Leaving this blank will bind to all avalible IP adresses.
;bind_to_address=
;
;# SOCKET TIMEOUT
;  Timeout when reading packets on incoming sockets. If the data has not arrived withint this time we will bail out.
;socket_timeout=30

[NRPE]
;# NRPE PORT NUMBER
;  This is the port the NRPEListener.dll will listen to.
port=5666
;
;# COMMAND TIMEOUT
;  This specifies the maximum number of seconds that the NRPE daemon will allow plug-ins to finish executing before killing them off.
command_timeout=60
;
;# COMMAND ARGUMENT PROCESSING
;  This option determines whether or not the NRPE daemon will allow clients to specify arguments to commands that are executed.
allow_arguments=1
;
;# COMMAND ALLOW NASTY META CHARS
;  This option determines whether or not the NRPE daemon will allow clients to specify nasty (as in |`&><'"\[]{}) characters in arguments.
allow_nasty_meta_chars=1
;
;# USE SSL SOCKET
;  This option controls if SSL should be used on the socket.
use_ssl=1
;
;# BIND TO ADDRESS
;  Allows you to bind server to a specific local address. This has to be a dotted ip adress not a hostname.
;  Leaving this blank will bind to all avalible IP adresses.
; bind_to_address=
;
;# ALLOWED HOST ADDRESSES
;  This is a comma-delimited list of IP address of hosts that are allowed to talk to NRPE deamon.
;  If you leave this blank the global version will be used instead.
;allowed_hosts=
;
;# SCRIPT DIRECTORY
;  All files in this directory will become check commands.
;  *WARNING* This is undoubtedly dangerous so use with care!
;script_dir=scripts\
;
;# SOCKET TIMEOUT
;  Timeout when reading packets on incoming sockets. If the data has not arrived withint this time we will bail out.
socket_timeout=60

[Check System]
;# CPU BUFFER SIZE
;  Can be anything ranging from 1s (for 1 second) to 10w for 10 weeks. Notice that a larger buffer will waste memory 
;  so don't use a larger buffer then you need (ie. the longest check you do +1).
;CPUBufferSize=1h
;
;# CHECK RESOLUTION
;  The resolution to check values (currently only CPU).
;  The value is entered in 1/10:th of a second and the default is 10 (which means ones every second)
;CheckResolution=10
;
;# CHECK ALL SERVICES
;  Configure how to check services when a CheckAll is performed.
;  ...=started means services in that class *has* to be running.
;  ...=stopped means services in that class has to be stopped.
;  ...=ignored means services in this class will be ignored.
;check_all_services[SERVICE_BOOT_START]=ignored
;check_all_services[SERVICE_SYSTEM_START]=ignored
;check_all_services[SERVICE_AUTO_START]=started
;check_all_services[SERVICE_DEMAND_START]=ignored
;check_all_services[SERVICE_DISABLED]=stopped

[External Script]
;# COMMAND TIMEOUT
;  This specifies the maximum number of seconds that the NRPE daemon will allow plug-ins to finish executing before killing them off.
;command_timeout=60
;
;# COMMAND ARGUMENT PROCESSING
;  This option determines whether or not the NRPE daemon will allow clients to specify arguments to commands that are executed.

allow_arguments=1

;
;# COMMAND ALLOW NASTY META CHARS
;  This option determines whether or not the NRPE daemon will allow clients to specify nasty (as in |`&><'"\[]{}) characters in arguments.

allow_nasty_meta_chars=1
;
;# SCRIPT DIRECTORY
;  All files in this directory will become check commands.
;  *WARNING* This is undoubtedly dangerous so use with care!
;script_dir=c:\my\script\dir

[Script Wrappings]
vbs=cscript.exe //T:30 //NoLogo scripts\lib\wrapper.vbs %SCRIPT% %ARGS%
ps1=cmd /c echo scripts\%SCRIPT% %ARGS%; exit($lastexitcode) | powershell.exe -command -
bat=scripts\%SCRIPT% %ARGS%

[External Scripts]
;check_es_long=scripts\long.bat
;check_es_ok=scripts\ok.bat
;check_es_nok=scripts\nok.bat
;check_vbs_sample=cscript.exe //T:30 //NoLogo scripts\check_vb.vbs
;check_powershell_warn=cmd /c echo scripts\powershell.ps1 | powershell.exe -command -

[External Alias]
alias_cpu=checkCPU warn=80 crit=90 time=5m time=1m time=30s
alias_cpu_ex=checkCPU warn=$ARG1$ crit=$ARG2$ time=5m time=1m time=30s
alias_mem=checkMem MaxWarn=80% MaxCrit=90% ShowAll=long type=physical type=virtual type=paged type=page
alias_up=checkUpTime MinWarn=1d MinWarn=1h

alias_disk=CheckDriveSize MinWarn=10% MinCrit=5% CheckAll FilterType=FIXED
alias_disk_loose=CheckDriveSize MinWarn=10% MinCrit=5% CheckAll FilterType=FIXED ignore-unreadable
alias_volumes=CheckDriveSize MinWarn=10% MinCrit=5% CheckAll=volumes FilterType=FIXED
alias_volumes_loose=CheckDriveSize MinWarn=10% MinCrit=5% CheckAll=volumes FilterType=FIXED ignore-unreadable 

alias_service=checkServiceState CheckAll
alias_service_ex=checkServiceState CheckAll "exclude=Net Driver HPZ12" "exclude=Pml Driver HPZ12" exclude=stisvc
alias_process=checkProcState "$ARG1$=started"
alias_process_stopped=checkProcState "$ARG1$=stopped"
alias_process_count=checkProcState MaxWarnCount=$ARG2$ MaxCritCount=$ARG3$ "$ARG1$=started"
alias_process_hung=checkProcState MaxWarnCount=1 MaxCritCount=1 "$ARG1$=hung"

alias_event_log=CheckEventLog file=application file=system MaxWarn=1 MaxCrit=1 "filter=generated gt -2d AND severity NOT IN ('success', 'informational') AND source != 'SideBySide'" truncate=800 unique descriptions "syntax=%severity%: %source%: %message% (%count%)"

alias_file_size=CheckFiles "filter=size > $ARG2$" "path=$ARG1$" MaxWarn=1 MaxCrit=1 "syntax=%filename% %size%" max-dir-depth=10
alias_file_age=checkFile2 filter=out "file=$ARG1$" filter-written=>1d MaxWarn=1 MaxCrit=1 "syntax=%filename% %write%"

alias_sched_all=CheckTaskSched "filter=exit_code ne 0" "syntax=%title%: %exit_code%" warn=>0
alias_sched_long=CheckTaskSched "filter=status = 'running' AND most_recent_run_time < -$ARG1$" "syntax=%title% (%most_recent_run_time%)" warn=>0
alias_sched_task=CheckTaskSched "filter=title eq '$ARG1$' AND exit_code ne 0" "syntax=%title% (%most_recent_run_time%)" warn=>0

alias_updates=check_updates -warning 0 -critical 0

check_ok=CheckOK Everything is fine!

[Wrapped Scripts]
;check_test_vbs=check_test.vbs /arg1:1 /arg2:1 /variable:1
;check_test_ps1=check_test.ps1 arg1 arg2
;check_test_bat=check_test.bat arg1 arg2
;check_battery=check_battery.vbs
;check_printer=check_printer.vbs
;check_updates=check_updates.vbs


; [includes]
;# The order when used is "reversed" thus the last included file will be "first"
;# Included files can include other files (be carefull only do basic recursive checking)
;
; myotherfile.ini
; real.ini


[NSCA Agent]
;# CHECK INTERVALL (in seconds)
;   How often we should run the checks and submit the results.
;interval=5
;
;# ENCRYPTION METHOD
;   This option determines the method by which the send_nsca client will encrypt the packets it sends 
;   to the nsca daemon. The encryption method you choose will be a balance between security and 
;   performance, as strong encryption methods consume more processor resources.
;   You should evaluate your security needs when choosing an encryption method.
;
; Note: The encryption method you specify here must match the decryption method the nsca daemon uses 
;       (as specified in the nsca.cfg file)!!
; Values:
;	0 = None	(Do NOT use this option)
;	1 = Simple XOR  (No security, just obfuscation, but very fast)
;   2 = DES
;   3 = 3DES (Triple DES)
;	4 = CAST-128
;	6 = xTEA
;	8 = BLOWFISH
;	9 = TWOFISH
;	11 = RC2
;	14 = RIJNDAEL-128 (AES)
;	20 = SERPENT
;encryption_method=14
;
;# ENCRYPTION PASSWORD
;  This is the password/passphrase that should be used to encrypt the sent packets. 
;password=
;
;# BIND TO ADDRESS
;  Allows you to bind server to a specific local address. This has to be a dotted ip adress not a hostname.
;  Leaving this blank will bind to "one" local interface.
; -- not supported as of now -- 
;bind_to_address=
;
;# LOCAL HOST NAME
;  The name of this host (if empty "computername" will be used.
;hostname=
;
;# NAGIOS SERVER ADDRESS
;  The address to the nagios server to submit results to.
;nsca_host=192.168.0.1
;
;# NAGIOS SERVER PORT
;  The port to the nagios server to submit results to.
;nsca_port=5667
;

;# CHECK COMMAND LIST
;  The checks to run everytime we submit results back to nagios
;  Any command(alias/key) starting with a host_ is sent as HOST_COMMAND others are sent as SERVICE_COMMANDS
;  where the alias/key is used as service name.
;
[NSCA Commands]
;my_cpu_check=checkCPU warn=80 crit=90 time=20m time=10s time=4
;my_mem_check=checkMem MaxWarn=80% MaxCrit=90% ShowAll type=page
;my_svc_check=checkServiceState CheckAll exclude=wampmysqld exclude=MpfService
;host_check=check_ok

;# REMOTE NRPE PROXY COMMANDS
;  A list of commands that check other hosts.
;  Used by the NRPECLient module
[NRPE Client Handlers]
check_other=-H 192.168.0.1 -p 5666 -c remote_command -a arguments

;# LUA SCRIPT SECTION
;  A list of all Lua scripts to load.
;[LUA Scripts]
;scripts\test.lua

[cdienger]

The configuration looks good and works on my test machine. Try enabling logging by adding the FileLogger module to the config file and see if we can get more info from the log:

Code: Select all

[modules]
NRPEListener.dll
FileLogger.dll
CheckSystem.dll
CheckDisk.dll

make sure to restart the nsclient++ service after making the change.

[sajid4u2c]

Please find the below log entry in nsclient.log when i test the connection:

2019-03-20 18:27:35: error:modules\NRPEListener\NRPEListener.cpp:290: Unauthorize access from: [IP address]

replaced the actual IP address with text

[ssax]

Under your [NRPE] section, please change this:

Code: Select all

    ;allowed_hosts=

To this:

Code: Select all

allowed_hosts=127.0.0.1,YOUR_XI_SERVER_IP

Then restart NSClient++ and test again.

[sajid4u2c]

adding 127.0.0.1 , did not work.

However I have added the IP which was thrown in Nsclient.log in the allowed_hosts list and it started working.

The same IP address was shown in Nsclient.log of all the remaining servers. did the same in all and problem is solved now.

But I still wonder what that IP is. when I checked with our security team they said it is some natted IP.
Though it is not reachable from nagios server.

Thanks @cdienger !!

[scottwilkerson]

adding 127.0.0.1 , did not work.

However I have added the IP which was thrown in Nsclient.log in the allowed_hosts list and it started working.

The same IP address was shown in Nsclient.log of all the remaining servers. did the same in all and problem is solved now.

But I still wonder what that IP is. when I checked with our security team they said it is some natted IP.
Though it is not reachable from nagios server.

Thanks @cdienger !!

Glad to hear the issue is resolved!

Locking thread