Page 3 of 7

Re: RHEL 6.3 & NRPE Issues

Posted: Mon Sep 15, 2014 2:44 pm
by eloyd
What is /etc/resolv.conf now? If it's empty, or if there is no "nameserver" line, try adding:

Code: Select all

nameserver 8.8.8.8
and see if that resolves your host issues.

The check_nrpe not working means something else. And I'm in the middle of my own work right now, so if Spencer beats me to it, so be it. Otherwise I'll be back later to take a look.

Re: RHEL 6.3 & NRPE Issues

Posted: Mon Sep 15, 2014 4:59 pm
by sreinhardt
I completely agree with eloyd here, one comment is that unless your system is actually running bind or some other form of dns server, you would never want to point to yourself for resolution, just going to cause issues. Either 8.8.8.8 or an internal dns server would be just fine in resolv.conf.

What do you have for allowed hosts in /etc/xinet.d/nrpe? Actually just post the whole file please, along with /usr/local/nrpe/etc/nrpe.cfg or /usr/local/nagios/etc/nrpe.cfg(depending on which you have). Do you have both 127.0.0.1 and localhost? Seems funny to ask, but sometimes little changes like that can make all the difference for nrpe.

Re: RHEL 6.3 & NRPE Issues

Posted: Tue Sep 16, 2014 7:47 am
by 00_kl250
Ok, so I added our local dns server and googles 8.8.8.8 in the resolv.conf file and that took care of the netstat -at | grep nrpe issue:


[root@ ~]# netstat -at | grep nrpe
tcp 0 0 *:nrpe *:* LISTEN


However, still having issues when runing:

[root@ ~]# /usr/local/nagios/libexec/check_nrpe -H localhost -n

Code: Select all

CHECK_NRPE: Error receiving data from daemon.

less /etc/xinetd.d/nrpe

# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
        flags           = REUSE
        socket_type     = stream
        port            = 5666
        wait            = no
        user            = nagios
        group           = nagios
        server          = /usr/local/nagios/bin/nrpe
        server_args     = -c /usr/local/nagios/etc/nrpe.cfg --inetd
        log_on_failure  += USERID
        disable         = no
        only_from       = 192.168.0.29
}
/usr/local/nagios/etc

Code: Select all

#############################################################################
# Sample NRPE Config File
# Written by: Ethan Galstad ([email protected])
#
# Last Modified: 11-23-2007
#
# NOTES:
# This is a sample configuration file for the NRPE daemon.  It needs to be
# located on the remote host that is running the NRPE daemon, not the host
# from which the check_nrpe client is being executed.
#############################################################################


# LOG FACILITY
# The syslog facility that should be used for logging purposes.

log_facility=daemon



# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID
# number.  The file is only written if the NRPE daemon is started by the root
# user and is running in standalone mode.

pid_file=/var/run/nrpe.pid



# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-priviledged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

server_port=5666



# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface
# and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

#server_address=127.0.0.1



# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_user=nagios



# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_group=nagios



# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon. Network addresses with a bit mask
# (i.e. 192.168.1.0/24) are also supported. Hostname wildcards are not currently
# supported.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address.  I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1



# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed.  This option only works
# if the daemon was configured with the --enable-command-args configure script
# option.
# *** ENABLING THIS OPTION IS A SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=0



# BASH COMMAND SUBTITUTION
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments that contain bash command substitutions of the form
# $(...).  This option only works if the daemon was configured with both
# the --enable-command-args and --enable-bash-command-substitution configure
# script options.
#
# *** ENABLING THIS OPTION IS A HIGH SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow bash command substitutions,
#         1=allow bash command substitutions

allow_bash_command_substitution=0



# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the
# command line from the command definition.
#
# *** THIS EXAMPLE MAY POSE A POTENTIAL SECURITY RISK, SO USE WITH CAUTION! ***
# Usage scenario:
# Execute restricted commmands using sudo.  For this to work, you need to add
# the nagios user to your /etc/sudoers.  An example entry for alllowing
# execution of the plugins from might be:
#
# nagios          ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
#
# This lets the nagios user run all commands in that directory (and only them)
# without asking for a password.  If you do this, make sure you don't give
# random users write access to that directory or its contents!

# command_prefix=/usr/bin/sudo



# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the
# syslog facility.
# Values: 0=debugging off, 1=debugging on

debug=0



# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# allow plugins to finish executing before killing them off.

command_timeout=60



# CONNECTION TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# wait for a connection to be established before exiting. This is sometimes
# seen where a network problem stops the SSL being established even though
# accumulate, eating system resources. Do not set this too low.

connection_timeout=300



# WEEK RANDOM SEED OPTION
# This directive allows you to use SSL even if your system does not have
# a /dev/random or /dev/urandom (on purpose or because the necessary patches
# were not applied). The random number generator will be seeded from a file
# which is either a file pointed to by the environment valiable $RANDFILE
# or $HOME/.rnd. If neither exists, the pseudo random number generator will
# be initialized and a warning will be issued.
# Values: 0=only seed from /dev/[u]random, 1=also seed from weak randomness

#allow_weak_random_seed=1



# INCLUDE CONFIG FILE
# This directive allows you to include definitions from an external config file.

#include=<somefile.cfg>



# INCLUDE CONFIG DIRECTORY
# This directive allows you to include definitions from config files (with a
# .cfg extension) in one or more directories (with recursion).

#include_dir=<somedirectory>
#include_dir=<someotherdirectory>



# COMMAND DEFINITIONS
# Command definitions that this daemon will run.  Definitions
# are in the following format:
#
# command[<command_name>]=<command_line>
#
# When the daemon receives a request to return the results of <command_name>
# it will execute the command specified by the <command_line> argument.
#
# Unlike Nagios, the command line cannot contain macros - it must be
# typed exactly as it should be executed.
#
# Note: Any plugins that are used in the command lines must reside
# on the machine that this daemon is running on!  The examples below
# assume that you have plugins installed in a /usr/local/nagios/libexec
# directory.  Also note that you will have to modify the definitions below
# to match the argument format the plugins expect.  Remember, these are
# examples only!


# The following examples use hardcoded command arguments...

command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200


# The following examples allow user-supplied arguments and can
# only be used if the NRPE daemon was compiled with support for
# command arguments *AND* the dont_blame_nrpe directive in this
# config file is set to '1'.  This poses a potential security risk, so
# make sure you read the SECURITY file before doing this.

#command[check_users]=/usr/local/nagios/libexec/check_users -w $ARG1$ -c $ARG2$
#command[check_load]=/usr/local/nagios/libexec/check_load -w $ARG1$ -c $ARG2$
#command[check_disk]=/usr/local/nagios/libexec/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
#command[check_procs]=/usr/local/nagios/libexec/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$

Thanks again!

Re: RHEL 6.3 & NRPE Issues

Posted: Tue Sep 16, 2014 7:53 am
by 00_kl250
BTW, I edited the /etc/xinetd.d/nrpe file to include 127.0.0.1 along with my nagios monitoring server of 192.168.0.29 to see if that fixes the the issue, but it didn't

# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
flags = REUSE
socket_type = stream
port = 5666
wait = no
user = nagios
group = nagios
server = /usr/local/nagios/bin/nrpe
server_args = -c /usr/local/nagios/etc/nrpe.cfg --inetd
log_on_failure += USERID
disable = no
only_from = 127.0.0.1, 192.168.0.29
}

[root@ etc]# /usr/local/nagios/libexec/check_nrpe -H localhost -n
CHECK_NRPE: Error receiving data from daemon.

Re: RHEL 6.3 & NRPE Issues

Posted: Tue Sep 16, 2014 1:14 pm
by millisa
Are you able to see anything listening on tcp port 5666?

Code: Select all

nmap localhost
or

Code: Select all

nmap 192.168.0.29
(should show you all the ports that are viewable)
If you don't have nmap, on centos it comes out of the base repos:

Code: Select all

yum install nmap
(this is slightly different than lsof - lsof would show what's being listened on, but wouldn't necessarily account for iptables or something else blocking access to the port). If you don't see 5666 come up, then we probably need to look at iptables since you've shown with lsof that it should be listening on 5666.

The one other bit that jumps out at me is that this is RHEL6.3. I remember somewhere around 6.3 there was an errata update for the selinux-policy packages that corrected one of the issues you'd hit with NRPE. (I believe it was this one. If you have selinux enabled, you may want to consider updating (6.4 came out in February 2013, 6.5 was November of 2013). Unless it's been reconfigured, I believe it logs to /var/log/audit/audit.log, it's probably worth checking when you run the check.

Re: RHEL 6.3 & NRPE Issues

Posted: Tue Sep 16, 2014 5:24 pm
by abrist
You are restarting xinetd after these edits, correct?

Code: Select all

service xinetd restart
00_kl250 wrote:[root@ etc]# /usr/local/nagios/libexec/check_nrpe -H localhost -n
CHECK_NRPE: Error receiving data from daemon.
You will receive this error if there is an ssl mismatch. Try rerunning the command without the -n:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H localhost

Re: RHEL 6.3 & NRPE Issues

Posted: Tue Sep 16, 2014 7:26 pm
by 00_kl250
Yep i've been restarting the xinetd service after any changes.

Just restarted it and ran it again:

[root@TDMAURCMESERV32 ~]# service xinetd restart
Stopping xinetd: [ OK ]
Starting xinetd: [ OK ]
[root@TDMAURCMESERV32 ~]# /usr/local/nagios/libexec/check_nrpe -H localhost
connect to address ::1 port 5666: Connection refused
connect to address 127.0.0.1 port 5666: Connection refused
connect to host localhost port 5666: Connection refused[root@TDMAURCMESERV32 ~]#

I install nmap and ran the command and get the following:


Starting Nmap 5.51 ( http://nmap.org ) at 2014-09-16 19:24 CDT
Failed to find device bond0 which was referenced in /proc/net/route
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0000030s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 988 closed ports
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
25/tcp open smtp
111/tcp open rpcbind
7000/tcp open afs3-fileserver
7001/tcp open afs3-callback
7002/tcp open afs3-prserver
7007/tcp open afs3-bos
7019/tcp open unknown
9000/tcp open cslistener
9998/tcp open distinct32
9999/tcp open abyss

I don't see port 5666

Re: RHEL 6.3 & NRPE Issues

Posted: Wed Sep 17, 2014 4:46 pm
by sreinhardt
Agreed, I don't see 5666 either, which should show regardless if xinetd and nrpe are allowing that host access. Let's check iptables and see if you might be blocked there:

Code: Select all

iptables -L -n

Re: RHEL 6.3 & NRPE Issues

Posted: Wed Sep 17, 2014 4:51 pm
by eloyd
May I suggest:

Code: Select all

iptables -L -v -n -x
To show the counters more fully? If you have a rule that's never being hit, then you need to look at iptables ordering.

Re: RHEL 6.3 & NRPE Issues

Posted: Wed Sep 17, 2014 4:55 pm
by sreinhardt
Fair enough, thanks for the tip! OP please send back the output from that when you have a chance.