Page 3 of 4
Re: CHECK_NRPE: Error - Could not connect to . Check system
Posted: Tue Apr 09, 2019 1:56 pm
by npolovenko
Also, on the Solaris server please run this command and show me the output:
ls -l /usr/local/nagios/libexec/
Re: CHECK_NRPE: Error - Could not connect to . Check system
Posted: Tue Apr 09, 2019 2:46 pm
by anilgupta
Save the file and restart the nrpe service on the Solaris server one more time.
Executed the action plan as suggested.
Also, on the Solaris server please run this command and show me the output:
Please find below the output
Code: Select all
bash-3.2# uname -a
SunOS csbtipdcapmdw07 5.10 Generic_150400-63 sun4v sparc sun4v
bash-3.2# ls -l /usr/local/nagios/libexec/
total 29690
-rwxr-xr-x 1 wassrv wassrv 844 Sep 4 2013 check_alertlog
-rwxr-xr-x 1 wassrv wassrv 1860 May 16 2012 check_app_disk
-rwxr-xr-x 1 wassrv wassrv 267532 Sep 11 2012 check_apt
-rwxr-xr-x 1 wassrv wassrv 2245 Sep 11 2012 check_breeze
-rwxr-xr-x 1 wassrv wassrv 113688 Sep 11 2012 check_by_ssh
-rwxr-xr-x 1 wassrv wassrv 167508 Sep 11 2012 check_clamd
-rwxr-xr-x 1 wassrv wassrv 85956 Sep 11 2012 check_cluster
-rwxr-xr-x 1 wassrv wassrv 107492 Sep 11 2012 check_dig
-rwxr-xr-x 1 wassrv wassrv 328424 Sep 11 2012 check_disk
-rwxr-xr-x 1 wassrv wassrv 9148 Sep 11 2012 check_disk_smb
-rwxr-xr-x 1 wassrv wassrv 113656 Sep 11 2012 check_dns
-rwxr-xr-x 1 wassrv wassrv 2670 Oct 20 2015 check_ds
-rwxr-xr-x 1 wassrv wassrv 65824 Sep 11 2012 check_dummy
-rwxr-xr-x 1 wassrv wassrv 3056 Sep 11 2012 check_file_age
-rwxr-xr-x 1 wassrv wassrv 6318 Sep 11 2012 check_flexlm
-rwxr-xr-x 1 wassrv wassrv 167508 Sep 11 2012 check_ftp
-rwxr-xr-x 1 wassrv wassrv 1815 May 24 2012 check_gosecadmin
-rwxr-xr-x 1 wassrv wassrv 797 Nov 13 2012 check_gosecenrol
-rwxr-xr-x 1 wassrv wassrv 839 Jun 21 2013 check_gosecenrol_log
-rwxr-xr-x 1 wassrv wassrv 407748 Sep 11 2012 check_http
-rwxr-xr-x 1 wassrv wassrv 1510 May 6 2012 check_https
-rwxr-xr-x 1 wassrv wassrv 15137 Sep 11 2012 check_ifoperstatus
-rwxr-xr-x 1 wassrv wassrv 12601 Sep 11 2012 check_ifstatus
-rwxr-xr-x 1 wassrv wassrv 167508 Sep 11 2012 check_imap
-rwxr-xr-x 1 wassrv wassrv 6890 Sep 11 2012 check_ircd
-rwxr-xr-x 1 wassrv wassrv 167508 Sep 11 2012 check_jabber
-rwxr-xr-x 1 wassrv wassrv 110804 Sep 11 2012 check_ldap
-rwxr-xr-x 1 wassrv wassrv 110804 Sep 11 2012 check_ldaps
-rwxr-xr-x 1 wassrv wassrv 2317 May 14 2013 check_load
-rwxr-xr-x 1 wassrv wassrv 6320 Apr 30 2013 check_log
-rwxr-xr-x 1 wassrv wassrv 6014 Nov 19 2012 check_log.orig
-rwxr-xr-x 1 wassrv wassrv 20287 Sep 11 2012 check_mailq
-rwxr-xr-x 1 wassrv wassrv 89868 Sep 11 2012 check_mrtg
-rwxr-xr-x 1 wassrv wassrv 89268 Sep 11 2012 check_mrtgtraf
-rwxr-xr-x 1 wassrv wassrv 95840 Sep 11 2012 check_nagios
-rwxr-xr-x 1 wassrv wassrv 167508 Sep 11 2012 check_nntp
-rwxr-xr-x 1 wassrv wassrv 167508 Sep 11 2012 check_nntps
-rwxrwxr-x 1 wassrv wassrv 71476 Sep 6 2013 check_nrpe
-rwxr-xr-x 1 wassrv wassrv 113188 Sep 11 2012 check_nt
-rwxr-xr-x 1 wassrv wassrv 116256 Sep 11 2012 check_ntp
-rwxr-xr-x 1 wassrv wassrv 110856 Sep 11 2012 check_ntp_peer
-rwxr-xr-x 1 wassrv wassrv 108720 Sep 11 2012 check_ntp_time
-rwxr-xr-x 1 wassrv wassrv 132300 Sep 11 2012 check_nwstat
-rwxr-xr-x 1 wassrv wassrv 1697 May 6 2012 check_oam
-rwxr-xr-x 1 wassrv wassrv 820 Jan 8 2013 check_oid_log
-rwxr-xr-x 1 wassrv wassrv 2079 Oct 20 2015 check_oim
-rwxr-xr-x 1 wassrv wassrv 1949 Nov 14 2012 check_oimnull
-rwxr-xr-x 1 wassrv wassrv 1591 Sep 26 2012 check_ora
-rwxr-xr-x 1 wassrv wassrv 8325 Sep 11 2012 check_oracle
-rwxr-xr-x 1 wassrv wassrv 1442 May 3 2012 check_ovd
-rwxr-xr-x 1 wassrv wassrv 102688 Apr 30 2012 check_overcr
-rwxr-xr-x 1 wassrv wassrv 100200 Sep 11 2012 check_pgsql
-rwxr-xr-x 1 wassrv wassrv 117912 Sep 11 2012 check_ping
-rwxr-xr-x 1 wassrv wassrv 167508 Sep 11 2012 check_pop
-rwxr-xr-x 1 wassrv wassrv 293360 Sep 11 2012 check_procs
-rwxr-xr-x 1 wassrv wassrv 100104 Sep 11 2012 check_real
-rwxr-xr-x 1 wassrv wassrv 9584 Sep 11 2012 check_rpc
-rwxr-xr-x 1 wassrv wassrv 1414 Sep 11 2012 check_sensors
-rwxr-xr-x 1 wassrv wassrv 167508 Sep 11 2012 check_simap
-rwxr-xr-x 1 wassrv wassrv 353292 Sep 11 2012 check_smtp
-rwxr-xr-x 1 wassrv wassrv 167508 Sep 11 2012 check_spop
-rwxr-xr-x 1 wassrv wassrv 96764 Sep 11 2012 check_ssh
-rwxr-xr-x 1 wassrv wassrv 167508 Sep 11 2012 check_ssmtp
-rwxr-xr-x 1 wassrv wassrv 167508 Sep 11 2012 check_tcp
-rwxr-xr-x 1 wassrv wassrv 99328 Sep 11 2012 check_time
-rwxr-xr-x 1 wassrv wassrv 167508 Sep 11 2012 check_udp
-rwxr-xr-x 1 wassrv wassrv 108308 Sep 11 2012 check_ups
-rwxr-xr-x 1 wassrv wassrv 83308 Sep 11 2012 check_users
-rwxr-xr-x 1 wassrv wassrv 2939 Sep 11 2012 check_wave
-rwxr-xr-x 1 wassrv wassrv 1461 May 6 2012 check_weblogic
-rw-r--r-- 1 wassrv wassrv 143872 Sep 11 2012 checks.tar
-rw-r--r-- 1 wassrv wassrv 276 Mar 14 2013 enrolscriptenv
drwxr-xr-x 2 wassrv wassrv 4 Dec 20 2012 eventhandlers
-rwxr-xr-x 1 wassrv wassrv 96260 Sep 11 2012 negate
-rw-r--r-- 1 wassrv wassrv 176 May 29 2012 newmail
-rwxr-xr-x 1 wassrv wassrv 2526 Jun 30 2015 notify.pl
-rwxr-xr-x 1 wassrv wassrv 504 Jan 25 2013 restart-oid
-rw-r--r-- 1 wassrv wassrv 315 Jan 8 2013 scriptenv
-rw-r--r-- 1 wassrv wassrv 6656000 Sep 6 2013 scripts.tar
-rwxr-xr-x 1 wassrv wassrv 1563 Apr 30 2013 sendmail
-rw-r--r-- 1 wassrv wassrv 584 Dec 20 2012 setenv
-rwxr-xr-x 1 wassrv wassrv 92524 Sep 11 2012 urlize
-rwxr-xr-x 1 wassrv wassrv 1938 Sep 11 2012 utils.pm
-rwxr-xr-x 1 wassrv wassrv 2728 Oct 17 2013 utils.sh
bash-3.2#
After the nrpre restart I do not see any change in error message.
Code: Select all
[guptaa@cacbigdcapmdw51 log]$ date;/usr/local/nagios/libexec/check_nrpe -H 10.92.34.66 -t 30 -c check_disk -a '-w 20% -c 10% -p /'
Tue Apr 9 15:43:07 EDT 2019
CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected).
Re: CHECK_NRPE: Error - Could not connect to . Check system
Posted: Tue Apr 09, 2019 2:54 pm
by anilgupta
nrpe.cfg file after the change, just for your reference.
Code: Select all
bash-3.2# uname -a
SunOS csbtipdcapmdw07 5.10 Generic_150400-63 sun4v sparc sun4v
bash-3.2# cat nrpe.cfg
#############################################################################
# Sample NRPE Config File
# Written by: Ethan Galstad ([email protected])
#
# Last Modified: 11-23-2007
#
# NOTES:
# This is a sample configuration file for the NRPE daemon. It needs to be
# located on the remote host that is running the NRPE daemon, not the host
# from which the check_nrpe client is being executed.
#############################################################################
# LOG FACILITY
# The syslog facility that should be used for logging purposes.
log_facility=daemon
# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID
# number. The file is only written if the NRPE daemon is started by the root
# user and is running in standalone mode.
pid_file=/var/tmp/nrpe.pid
# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-priviledged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
server_port=5666
# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface
# and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
server_address=10.92.34.66
# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
nrpe_user=wassrv
# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
nrpe_group=wassrv
# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon. Network addresses with a bit mask
# (i.e. 192.168.1.0/24) are also supported. Hostname wildcards are not currently
# supported.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address. I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
#allowed_hosts=127.0.0.1,10.92.34.51,10.92.34.49,10.92.34.53,10.92.34.52,cacbigdcapmdw51.mgmt.cihs.gov.on.ca
allowed_hosts=127.0.0.1,10.92.34.51,10.92.34.49,10.92.34.53,10.92.34.52,cacbigdcapmdw51.mgmt.cihs.gov.on.ca, 10.194.78.208, 10.194.78.206
# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed. This option only works
# if the daemon was configured with the --enable-command-args configure script
# option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments
dont_blame_nrpe=0
# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the
# command line from the command definition.
#
# *** THIS EXAMPLE MAY POSE A POTENTIAL SECURITY RISK, SO USE WITH CAUTION! ***
# Usage scenario:
# Execute restricted commmands using sudo. For this to work, you need to add
# the nagios user to your /etc/sudoers. An example entry for alllowing
# execution of the plugins from might be:
#
# nagios ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
#
# This lets the nagios user run all commands in that directory (and only them)
# without asking for a password. If you do this, make sure you don't give
# random users write access to that directory or its contents!
# command_prefix=/usr/bin/sudo
# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the
# syslog facility.
# Values: 0=debugging off, 1=debugging on
debug=0
# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# allow plugins to finish executing before killing them off.
command_timeout=60
# CONNECTION TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# wait for a connection to be established before exiting. This is sometimes
# seen where a network problem stops the SSL being established even though
# all network sessions are connected. This causes the nrpe daemons to
# accumulate, eating system resources. Do not set this too low.
connection_timeout=300
# WEEK RANDOM SEED OPTION
# This directive allows you to use SSL even if your system does not have
# a /dev/random or /dev/urandom (on purpose or because the necessary patches
# were not applied). The random number generator will be seeded from a file
# which is either a file pointed to by the environment valiable $RANDFILE
# or $HOME/.rnd. If neither exists, the pseudo random number generator will
# be initialized and a warning will be issued.
# Values: 0=only seed from /dev/[u]random, 1=also seed from weak randomness
#allow_weak_random_seed=1
# INCLUDE CONFIG FILE
# This directive allows you to include definitions from an external config file.
#include=<somefile.cfg>
# INCLUDE CONFIG DIRECTORY
# This directive allows you to include definitions from config files (with a
# .cfg extension) in one or more directories (with recursion).
#include_dir=<somedirectory>
#include_dir=<someotherdirectory>
# COMMAND DEFINITIONS
# Command definitions that this daemon will run. Definitions
# are in the following format:
#
# command[<command_name>]=<command_line>
#
# When the daemon receives a request to return the results of <command_name>
# it will execute the command specified by the <command_line> argument.
#
# Unlike Nagios, the command line cannot contain macros - it must be
# typed exactly as it should be executed.
#
# Note: Any plugins that are used in the command lines must reside
# on the machine that this daemon is running on! The examples below
# assume that you have plugins installed in a /usr/local/nagios/libexec
# directory. Also note that you will have to modify the definitions below
# to match the argument format the plugins expect. Remember, these are
# examples only!
# The following examples use hardcoded command arguments...
command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_app_disk]=/usr/local/nagios/libexec/check_app_disk -w 80 -c 90
command[check_load]=/usr/local/nagios/libexec/check_load -w 6 -c 10
command[check_oid_log]=/usr/local/nagios/libexec/check_oid_log
command[check_gosecenrol_log]=/usr/local/nagios/libexec/check_gosecenrol_log
command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
bash-3.2#
Re: CHECK_NRPE: Error - Could not connect to . Check system
Posted: Tue Apr 09, 2019 3:50 pm
by anilgupta
@npolovenko,
Could you please suggest alternate way to expedite the troubleshooting?
I work 8:30 AM till 4:30PM EST. I get less time to work on the issue.
The production deployment is waiting on fix of this issue.
Please suggest.
Re: CHECK_NRPE: Error - Could not connect to . Check system
Posted: Tue Apr 09, 2019 4:21 pm
by npolovenko
@anilgupta, Let's try the following. In the nrpe.cfg file change the check_disk command to look like this:
command[check_disk]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /
Save the file, restart the nrpe agent again and run these commands from the nagios server.
/usr/local/nagios/libexec/check_nrpe -H 10.92.34.66 -t 30 -c check_disk
/usr/local/nagios/libexec/check_nrpe -2 -H 10.92.34.66 -t 30 -c check_disk
And show me the output.
For faster assistance you can escalate this issue to a support ticket:
https://support.nagios.com/tickets/
That way we'll be able to organize a screen sharing session to resolve this.
Re: CHECK_NRPE: Error - Could not connect to . Check system
Posted: Wed Apr 10, 2019 10:46 am
by anilgupta
And show me the output.
After making the change, the command is working as expected.
Code: Select all
[guptaa@cacbigdcapmdw51 libexec]$ /usr/local/nagios/libexec/check_nrpe -H 10.92.34.66 -t 30 -c check_disk
DISK OK - free space: / 35654 MB (71% inode=99%);| /=14338MB;39993;44992;0;49992
[guptaa@cacbigdcapmdw51 libexec]$ /usr/local/nagios/libexec/check_nrpe -2 -H 10.92.34.66 -t 30 -c check_disk
DISK OK - free space: / 35654 MB (71% inode=99%);| /=14338MB;39993;44992;0;49992
[guptaa@cacbigdcapmdw51 libexec]$
Additionally, I created command similar to "check_nrpe" with command_line as "
Code: Select all
$USER1$/check_nrpe -H $HOSTADDRESS$ -t 30 -c $ARG1$
". Now this new check_nrpe is working for Check_Disk.
Till here, we are satisfied.
However, we are encountering below issues:
1) I need to monitor CPU on the same server. I did below :
(a) added
Code: Select all
"command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats -w 70,40,30 -c 90,60,40"
in nrpe.cfg
(b) Restarted nrpe
(c) Below command shows "Unable to read output"
Code: Select all
/usr/local/nagios/libexec/check_nrpe -H 10.92.34.66 -t 30 -c check_cpu_stats
NRPE: Unable to read output
I don't see CPU related file under
on target Solaris server. Could that be the reason for this error, please suggest.
2) Host Status is showing down with status information as "CRITICAL - 10.92.34.66: rta nan, lost 100%"
Re: CHECK_NRPE: Error - Could not connect to . Check system
Posted: Wed Apr 10, 2019 12:12 pm
by npolovenko
@anilgupta
After making the change, the command is working as expected.
Great. I think the previous version of the command didn't work because you have the following option disabled in the nrpe.cfg:
dont_blame_nrpe=0
You should change it to:
dont_blame_nrpe=1
And restart the nrpe agent.
I don't see CPU related file under /usr/local/nagios/libexec/ on target Solaris server. Could that be the reason for this error, please suggest.
Yep, that's the problem. Because all this command does is it points to the plugin installed locally on the solaris server.
command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats -w 70,40,30 -c 90,60,40
You can download the actual plugin from here:
And place it into the /usr/local/nagios/libexec/ folder on the Solaris server. Use WinSCP or FileZilla to upload files.
After that you should give executable permissions to the plugin:
chmod 777 /usr/local/nagios/libexec/check_cpu_stats
I'm not 100% sure if this plugin is fully compatible with the version of your server. After you upload it and set permission run this command in the command line, on the Solaris server and make sure the plugin works OK:
/usr/local/nagios/libexec/check_cpu_stats -w 70,40,30 -c 90,60,40
Re: CHECK_NRPE: Error - Could not connect to . Check system
Posted: Wed Apr 10, 2019 2:49 pm
by anilgupta
Thanks npolovenko!
Great. I think the previous version of the command didn't work because you have the following option disabled in the nrpe.cfg:
I enabled the option by changing
changed the command as:
Code: Select all
command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
After nrpe restart, I get below message for the check_disk
Code: Select all
[guptaa@cacbigdcapmdw51 ~]$ /usr/local/nagios/libexec/check_nrpe -H 10.92.34.66 -t 30 -c check_disk -a '-w 20% -c 10% -p /'
CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected).
Yep, that's the problem. Because all this command does is it points to the plugin installed locally on the solaris server.
I downloaded the file under libexec and executed chmod 777 against the file. The result is not as expected.
Code: Select all
[guptaa@cacbigdcapmdw51 ~]$ /usr/local/nagios/libexec/check_nrpe -H 10.92.34.66 -t 30 -c check_cpu_stats -a '-w 70,40,30 -c 90,60,40'
CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected).
Please suggest.
Re: CHECK_NRPE: Error - Could not connect to . Check system
Posted: Wed Apr 10, 2019 3:06 pm
by npolovenko
@anilgupta, Please put the following option somewhere in the middle of the nrpe.cfg file:
allow_bash_command_substitution=1
And restart the nrpe agent. Try running the check_disk command again.
It could also be that this particular nrpe agent is old and was initially configured not to accept arguments.
As far as check_cpu_stats, please run the command I suggested in the previous message:
/usr/local/nagios/libexec/check_cpu_stats -w 70,40,30 -c 90,60,40
And let me know if it works. This command needs to be executed on the Solaris server.
Re: CHECK_NRPE: Error - Could not connect to . Check system
Posted: Thu Apr 11, 2019 8:34 am
by anilgupta
And restart the nrpe agent. Try running the check_disk command again.
It could also be that this particular nrpe agent is old and was initially configured not to accept arguments.
After adding
in nrpe.cfg and restarting nrpe. I see same error message, please see attached nrpe.cfg
Code: Select all
[guptaa@cacbigdcapmdw51 ~]$ /usr/local/nagios/libexec/check_nrpe -H 10.92.34.66 -t 30 -c check_disk -a '-w 20% -c 10% -p /'
CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected).
As far as check_cpu_stats, please run the command I suggested in the previous message:
I get the output when I run it on Solaris server.
Code: Select all
bash-3.2# /usr/local/nagios/libexec/check_cpu_stats -w 70,40,30 -c 90,60,40
CPU STATISTICS OK : user=0% system=0%, iowait=0%, idle=100%, nice=0.00%, steal=0.00% | CpuUser=0%;70;90;0; CpuSystem=0%;40;60;0; CpuIowait=0%;30;40;0; CpuIdle=100%;0;0;0; CpuNice=0.0%;0;0;0; CpuSteal=0.0%;0;0;0;
bash-3.2#
however, it is breaking from Nagios Server.
Code: Select all
[guptaa@cacbigdcapmdw51 ~]$ /usr/local/nagios/libexec/check_nrpe -H 10.92.34.66 -t 30 -c check_cpu_stats -a '-w 70,40,30 -c 90,60,40'
CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected).