CHECK_NRPE: Received 0 bytes from daemon. Check the remote

[maddev]

I seem to have stuck at a typical issue which might be a piece of cake for most of you.

I am trying to monitor a RHEL5 machine using Nagios XI 2014.
After adding the server through wizard, I am getting the below error as status information for all services of that host and status is marked as unknown.
CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages.

I tried the below,
1) On Nagios XI server

Code: Select all

# /usr/local/nagios/libexec/check_nrpe -H remoterhel5box
NRPE v2.15

2) On the remote redhat machine

Code: Select all

# netstat -at | grep nrpe
tcp        0      0 *:nrpe                      *:*                         LISTEN

Code: Select all

# lsof -i:5666
COMMAND   PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
xinetd  27527 root    5u  IPv4 44504170      0t0  TCP *:nrpe (LISTEN)

Code: Select all

# service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:5667
2    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:5666

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Code: Select all

# cat /etc/services | grep 566[6-8]
nrpe    5666/tcp
nsca    5667/tcp
ndo2db  5668/tcp

Code: Select all

# /usr/local/nagios/libexec/check_nrpe -H localhost
NRPE v2.15

Code: Select all

# /usr/local/nagios/libexec/check_nrpe -H localhost -c check_load
OK - load average: 0.12, 0.26, 0.31|load1=0.120;15.000;30.000;0; load5=0.260;10.000;25.000;0; load15=0.310;5.000;20.000;0;

Code: Select all

# tail /var/log/messages
Apr 20 09:34:30 auspcrpbak02 xinetd[27527]: START: nrpe pid=28084 from=10.168.185.20
Apr 20 09:34:30 auspcrpbak02 nrpe[28084]: Error: Request contained command arguments!
Apr 20 09:34:30 auspcrpbak02 nrpe[28084]: Client request was invalid, bailing out...
Apr 20 09:34:30 auspcrpbak02 xinetd[27527]: EXIT: nrpe status=0 pid=28084 duration=0(sec)
Apr 20 09:34:55 auspcrpbak02 xinetd[27527]: START: nrpe pid=28092 from=10.168.185.20
Apr 20 09:34:55 auspcrpbak02 nrpe[28092]: Error: Request contained command arguments!
Apr 20 09:34:55 auspcrpbak02 nrpe[28092]: Client request was invalid, bailing out...
Apr 20 09:34:55 auspcrpbak02 xinetd[27527]: EXIT: nrpe status=0 pid=28092 duration=0(sec)
Apr 20 09:34:55 auspcrpbak02 xinetd[27527]: START: nrpe pid=28094 from=127.0.0.1
Apr 20 09:34:55 auspcrpbak02 xinetd[27527]: EXIT: nrpe status=0 pid=28094 duration=0(sec)

[lmiltchev]

On the client, open the "/usr/local/nagios/etc/nrpe.cfg" in a text editor and change the following line from:

Code: Select all

dont_blame_nrpe=0

to:

Code: Select all

dont_blame_nrpe=1

Save, exit and restart NRPE.

If it's running under xinetd:

Code: Select all

service xinetd restart

Hope this helps.

[maddev]

Hi lmiltchev,

I have set this to 1 and restarted already. Still no good.

Code: Select all

# grep blame /usr/local/nagios/etc/nrpe.cfg
dont_blame_nrpe=1
# command arguments *AND* the dont_blame_nrpe directive in this

[jolson]

Please post your entire nrpe.cfg, excluding any sensitive information:

Code: Select all

cat /usr/local/nagios/etc/nrpe.cfg

Do any of the commands listed in nrpe.cfg work properly? You could try running the following from Nagios XI:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H remoterhel5box -c check_users
/usr/local/nagios/libexec/check_nrpe -H remoterhel5box -c check_load

I am guessing you run into the same exception as before?

[maddev]

Hi jolson,

Please find below the contents of nrpe.cfg on my remote node

# more /usr/local/nagios/etc/nrpe.cfg

Code: Select all

#############################################################################
# Sample NRPE Config File
# Written by: Ethan Galstad ([email protected])
#
# Last Modified: 11-23-2007
#
# NOTES:
# This is a sample configuration file for the NRPE daemon.  It needs to be
# located on the remote host that is running the NRPE daemon, not the host
# from which the check_nrpe client is being executed.
#############################################################################


# LOG FACILITY
# The syslog facility that should be used for logging purposes.

log_facility=daemon



# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID
# number.  The file is only written if the NRPE daemon is started by the root
# user and is running in standalone mode.

pid_file=/var/run/nrpe.pid



# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-priviledged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

server_port=5666



# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface
# and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

#server_address=127.0.0.1



# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_user=nagios



# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_group=nagios



# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address.  I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

# allowed_hosts=127.0.0.1
allowed_hosts=<ip addr of my Nagios XI server>


# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed.  This option only works
# if the daemon was configured with the --enable-command-args configure script
# option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=1



# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the
# command line from the command definition.
#
# *** THIS EXAMPLE MAY POSE A POTENTIAL SECURITY RISK, SO USE WITH CAUTION! ***
# Usage scenario:
# Execute restricted commmands using sudo.  For this to work, you need to add
# the nagios user to your /etc/sudoers.  An example entry for alllowing
# execution of the plugins from might be:
#
# nagios          ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
#
# This lets the nagios user run all commands in that directory (and only them)
# without asking for a password.  If you do this, make sure you don't give
# random users write access to that directory or its contents!

# command_prefix=/usr/bin/sudo



# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the
# syslog facility.
# Values: 0=debugging off, 1=debugging on

debug=0



# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# allow plugins to finish executing before killing them off.

command_timeout=60



# CONNECTION TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# wait for a connection to be established before exiting. This is sometimes
# seen where a network problem stops the SSL being established even though
# all network sessions are connected. This causes the nrpe daemons to
# accumulate, eating system resources. Do not set this too low.

connection_timeout=300



# WEEK RANDOM SEED OPTION
# This directive allows you to use SSL even if your system does not have
# a /dev/random or /dev/urandom (on purpose or because the necessary patches
# were not applied). The random number generator will be seeded from a file
# which is either a file pointed to by the environment valiable $RANDFILE
# or $HOME/.rnd. If neither exists, the pseudo random number generator will
# be initialized and a warning will be issued.
# Values: 0=only seed from /dev/[u]random, 1=also seed from weak randomness

#allow_weak_random_seed=1



# INCLUDE CONFIG FILE
# This directive allows you to include definitions from an external config file.

#include=<somefile.cfg>



# INCLUDE CONFIG DIRECTORY
# This directive allows you to include definitions from config files (with a
# .cfg extension) in one or more directories (with recursion).

#include_dir=<somedirectory>
include_dir=/usr/local/nagios/etc/nrpe



# COMMAND DEFINITIONS
# Command definitions that this daemon will run.  Definitions
# are in the following format:
#
# command[<command_name>]=<command_line>
#
# When the daemon receives a request to return the results of <command_name>
# it will execute the command specified by the <command_line> argument.
#
# Unlike Nagios, the command line cannot contain macros - it must be
# typed exactly as it should be executed.
#
# Note: Any plugins that are used in the command lines must reside
# on the machine that this daemon is running on!  The examples below
# assume that you have plugins installed in a /usr/local/nagios/libexec
# directory.  Also note that you will have to modify the definitions below
# to match the argument format the plugins expect.  Remember, these are
# examples only!


# The following examples use hardcoded command arguments...

#command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
#command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
#command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1
#command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
#command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200


# The following examples allow user-supplied arguments and can
# only be used if the NRPE daemon was compiled with support for
# command arguments *AND* the dont_blame_nrpe directive in this
# config file is set to '1'.  This poses a potential security risk, so
# make sure you read the SECURITY file before doing this.

#command[check_users]=/usr/local/nagios/libexec/check_users -w $ARG1$ -c $ARG2$
#command[check_load]=/usr/local/nagios/libexec/check_load -w $ARG1$ -c $ARG2$
#command[check_disk]=/usr/local/nagios/libexec/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
#command[check_procs]=/usr/local/nagios/libexec/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$

I am able to successfully execute the below from my Nagios XI server; where as the commands mentioned by you are throwing syntax/usage error

Code: Select all

# /usr/local/nagios/libexec/check_nrpe -H remoterhel5box
NRPE v2.15

Code: Select all

#/usr/local/nagios/libexec/check_nrpe -H remoterhel5box -c check_users


Usage:
check_users -w <users> -c <users>

Code: Select all

#/usr/local/nagios/libexec/check_nrpe -H remoterhel5box -c check_load
check_load: Could not parse arguments
Usage:
check_load [-r] -w WLOAD1,WLOAD5,WLOAD15 -c CLOAD1,CLOAD5,CLOAD15

[lmiltchev]

The "check_users" command is commented out in the "nrpe.cfg" file. If you used our Linux agent installer, most probably the commands that you are using are defined in the "common.cfg" file. Can you show us this file? On RHEL/CentOS it should be located at "/usr/local/nagios/etc/nrpe/common.cfg". Thanks!

[maddev]

here's the content of /usr/local/nagios/etc/nrpe/common.cfg

Code: Select all

# cd /usr/local/nagios/etc/nrpe
# cat common.cfg

### GENERIC SERVICES ###
command[check_init_service]=sudo /usr/local/nagios/libexec/check_init_service $ARG1$
command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$

### MISC SYSTEM METRICS ###
#command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_users]=/usr/local/nagios/libexec/check_users $ARG1$
command[check_load]=/usr/local/nagios/libexec/check_load $ARG1$
command[check_swap]=/usr/local/nagios/libexec/check_swap $ARG1$
command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats.sh $ARG1$
command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$

### SYSTEM UPDATES ###
command[check_yum]=/usr/local/nagios/libexec/check_yum
command[check_apt]=/usr/local/nagios/libexec/check_apt

### DISK ###
command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
command[check_ide_smart]=/usr/local/nagios/libexec/check_ide_smart $ARG1$

### PROCESSES ###
command[check_all_procs]=/usr/local/nagios/libexec/custom_check_procs
command[check_procs]=/usr/local/nagios/libexec/check_procs $ARG1$

### OPEN FILES ###
command[check_open_files]=/usr/local/nagios/libexec/check_open_files.pl $ARG1$

### NETWORK CONNECTIONS ###
command[check_netstat]=/usr/local/nagios/libexec/check_netstat.pl -p $ARG1$ $ARG2$

[jolson]

Could you try running:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H remoterhel5box -c check_services -a init

I believe that NRPE is working properly, and we need to give the client some arguments.

You can also alter the commands in your /usr/local/nagios/etc/nrpe/common.cfg. Some example command definitions are as follows:
command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200
command[check_for_elasticsearch_proc]=/usr/local/nagios/libexec/check_procs -a elasticsearch -w 1:1 -c 1:1
command[check_for_logstash_proc]=/usr/local/nagios/libexec/check_procs -a logstash -w 2:2 -c 2:2

[maddev]

Hi Jolson,

I was able to solve this by configuring the binary with –enable-command-args and then following the steps of NRPE agent installation from source guide available in Nagios XI site.

Thank you very much for working along to troubleshoot this.

[tmcdonald]

I'll be closing this thread now, but feel free to open another if you need anything in the future!