check_nrpe : No route to host

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

check_nrpe : No route to host

Post by donnyforbes »

I am trying to deploy a monitored server in test environment before I do production. I am following this guide http://www.tecmint.com/how-to-add-linux ... ed-server/ I have installed Nagios Core on Centos 7 Machine and the machine being monitored is a RHEL 7.3 Machine. I can't pass remote-host's nrpe check. Although both machines can talk to each other using ssh, when testing it says "no route to host on port 5666" see the complete message below.

When I run the following command from my nagios server here is what I get.

[root@nag ~]# /usr/local/nagios/libexec/check_nrpe -H 192.168.1.42
connect to address 192.168.1.42 port 5666: No route to host
connect to host 192.168.1.42 port 5666: No route to host

From the nagios server I am able to use ssh to the remote host
[root@nag ~]# ssh 192.168.1.xxx /usr/local/nagios/libexec/check_procs
root@192.168.1.xxx password:
PROCS OK: 617 processes | procs=617;;;0;

On the remote server I do the following:
firewall-cmd --zone=public --add-port=5666/tcp --permanent
then verify the port 5666/tcp is open

[root@huey ~]# netstat -na | grep 5666
tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN
tcp6 0 0 :::5666 :::* LISTEN

Also, I verify that nrpe is running

[root@huey ~]# ps -ef |grep nrpe
root 17336 21113 0 08:20 pts/1 00:00:00 grep --color=auto nrpe
nagios 37879 1 0 Jun15 ? 00:00:00 /usr/local/nagios/bin/nrpe -c /usr/local/nagios/etc/nrpe.cfg -f

At this point I am out of ideas and not sure what to do.
Nagios Server: CentOS Linux release 7.0.1406 (Core)

Remote Host: Red Hat Enterprise Linux Server release 7.3 (Maipo)

Any help would be great at this time ...
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: check_nrpe : No route to host

Post by mcapra »

Double-check the URL in your OP. The link looks like it got truncated. Knowing what steps were taken might be massively helpful.

Can you share the contents of the following file from your remote RHEL machine:

Code: Select all

/usr/local/nagios/etc/nrpe.cfg
Could you also try an nmap from the Nagios Core machine to the remote RHEL machine? Just because the port is listening on the RHEL machine doesn't mean the Nagios Core machine can talk over that port.

Might look something like this:

Code: Select all

[root@nagios ~]# nmap -p 5666 your_host_here

Starting Nmap 6.40 ( http://nmap.org ) at 2017-06-16 10:48 CDT
Nmap scan report for your_host_here (some_ip_here)
Host is up (0.000044s latency).
PORT     STATE  SERVICE
5666/tcp open nrpe

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds
Though if the state is closed or filtered, there may be network equipment between the Nagios machine and the RHEL machine that is blocking such traffic.
Former Nagios employee
https://www.mcapra.com/
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: check_nrpe : No route to host

Post by donnyforbes »

Hi so I yum install nmap got it installed and did what you asked from the nagios Core machine here is the output

[root@nag ~]# nmap -p 5666 192.168.1.xxx

Starting Nmap 6.40 ( http://nmap.org ) at 2017-06-16 10:11 PDT
Nmap scan report for my_host_here (192.168.1.xxx)
Host is up (0.098s latency).
PORT STATE SERVICE
5666/tcp filtered nrpe
MAC Address: 94:18:82:80:CD:B8 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 0.82 seconds

Is this what you were looking for?
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: check_nrpe : No route to host

Post by dwhitfield »

It looks like you *may* have a firewall in the way.

Can you share the contents of the following file from your remote RHEL machine: /usr/local/nagios/etc/nrpe.cfg
Also, please share /etc/xinetd.d/nrpe

Are you able to telnet on port 5666?
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: check_nrpe : No route to host

Post by donnyforbes »

Code: Select all

#############################################################################
# Sample NRPE Config File
# Written by: Ethan Galstad (nagios@nagios.org) # # Last Modified: 2016-05-10 # # NOTES:
# This is a sample configuration file for the NRPE daemon.  It needs to be # located on the remote host that is running the NRPE daemon, not the host # from which the check_nrpe client is being executed.
#############################################################################


# LOG FACILITY
# The syslog facility that should be used for logging purposes.

log_facility=daemon



# LOG FILE
# If a log file is specified in this option, nrpe will write to # that file instead of using syslog.

#log_file=${prefix}/var/nrpe.log



# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the # syslog facility.
# Values: 0=debugging off, 1=debugging on

debug=0



# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID # number.  The file is only written if the NRPE daemon is started by the root # user and is running in standalone mode.

pid_file=/usr/local/nagios/var/nrpe.pid



# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-privileged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

server_port=5666



# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface # and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

#server_address=127.0.0.1



# LISTEN QUEUE SIZE
# Listen queue size (backlog) for serving incoming connections.
# You may want to increase this value under high load.

#listen_queue_size=5



# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_user=nagios



# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_group=nagios



# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames # that are allowed to talk to the NRPE daemon. Network addresses with a bit mask # (i.e. 192.168.1.0/24) are also supported. Hostname wildcards are not currently # supported.
#
# Note: The daemon only does rudimentary checking of the client's IP # address.  I would highly recommend adding entries in your /etc/hosts.allow # file to allow only the specified host to connect to the port # you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1,::1



# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients # to specify arguments to commands that are executed.  This option only works # if the daemon was configured with the --enable-command-args configure script # option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! *** # Read the SECURITY file for information on some of the security implications # of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=0



# BASH COMMAND SUBSTITUTION
# This option determines whether or not the NRPE daemon will allow clients # to specify arguments that contain bash command substitutions of the form # $(...).  This option only works if the daemon was configured with both # the --enable-command-args and --enable-bash-command-substitution configure # script options.
#
# *** ENABLING THIS OPTION IS A HIGH SECURITY RISK! *** # Read the SECURITY file for information on some of the security implications # of enabling this variable.
#
# Values: 0=do not allow bash command substitutions,
#         1=allow bash command substitutions

allow_bash_command_substitution=0



# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the # command line from the command definition.
#
# *** THIS EXAMPLE MAY POSE A POTENTIAL SECURITY RISK, SO USE WITH CAUTION! *** # Usage scenario:
# Execute restricted commmands using sudo.  For this to work, you need to add # the nagios user to your /etc/sudoers.  An example entry for allowing # execution of the plugins from might be:
#
# nagios          ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
#
# This lets the nagios user run all commands in that directory (and only them) # without asking for a password.  If you do this, make sure you don't give # random users write access to that directory or its contents!

# command_prefix=/usr/bin/sudo



# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will # allow plugins to finish executing before killing them off.

command_timeout=60



# CONNECTION TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will # wait for a connection to be established before exiting. This is sometimes # seen where a network problem stops the SSL being established even though # all network sessions are connected. This causes the nrpe daemons to # accumulate, eating system resources. Do not set this too low.

connection_timeout=300



# WEAK RANDOM SEED OPTION
# This directive allows you to use SSL even if your system does not have # a /dev/random or /dev/urandom (on purpose or because the necessary patches # were not applied). The random number generator will be seeded from a file # which is either a file pointed to by the environment valiable $RANDFILE # or $HOME/.rnd. If neither exists, the pseudo random number generator will # be initialized and a warning will be issued.
# Values: 0=only seed from /dev/[u]random, 1=also seed from weak randomness

#allow_weak_random_seed=1



# SSL/TLS OPTIONS
# These directives allow you to specify how to use SSL/TLS.

# SSL VERSION
# This can be any of: SSLv2 (only use SSLv2), SSLv2+ (use any version),
#        SSLv3 (only use SSLv3), SSLv3+ (use SSLv3 or above), TLSv1 (only use
#        TLSv1), TLSv1+ (use TLSv1 or above), TLSv1.1 (only use TLSv1.1),
#        TLSv1.1+ (use TLSv1.1 or above), TLSv1.2 (only use TLSv1.2),
#        TLSv1.2+ (use TLSv1.2 or above)
# If an "or above" version is used, the best will be negotiated. So if both # ends are able to do TLSv1.2 and use specify SSLv2, you will get TLSv1.2.
# If you are using openssl 1.1.0 or above, the SSLv2 options are not available.

#ssl_version=SSLv2+

# SSL USE ADH
# This is for backward compatibility and is DEPRECATED. Set to 1 to enable # ADH or 2 to require ADH. 1 is currently the default but will be changed # in a later version.

#ssl_use_adh=1

# SSL CIPHER LIST
# This lists which ciphers can be used. For backward compatibility, this # defaults to 'ssl_cipher_list=ALL:!MD5:@STRENGTH' in this version but # will be changed to something like the example below in a later version of NRPE.

#ssl_cipher_list=ALL:!MD5:@STRENGTH
#ssl_cipher_list=ALL:!aNULL:!eNULL:!SSLv2:!LOW:!EXP:!RC4:!MD5:@STRENGTH

# SSL Certificate and Private Key Files

#ssl_cacert_file=/etc/ssl/servercerts/ca-cert.pem
#ssl_cert_file=/etc/ssl/servercerts/nagios-cert.pem
#ssl_privatekey_file=/etc/ssl/servercerts/nagios-key.pem

# SSL USE CLIENT CERTS
# This options determines client certificate usage.
# Values: 0 = Don't ask for or require client certificates (default)
#         1 = Ask for client certificates
#         2 = Require client certificates

#ssl_client_certs=0

# SSL LOGGING
# This option determines which SSL messages are send to syslog. OR values # together to specify multiple options.

# Values: 0x00 (0)  = No additional logging (default)
#         0x01 (1)  = Log startup SSL/TLS parameters
#         0x02 (2)  = Log remote IP address
#         0x04 (4)  = Log SSL/TLS version of connections
#         0x08 (8)  = Log which cipher is being used for the connection
#         0x10 (16) = Log if client has a certificate
#         0x20 (32) = Log details of client's certificate if it has one
#         -1 or 0xff or 0x2f = All of the above

#ssl_logging=0x00



# NASTY METACHARACTERS
# This option allows you to override the list of characters that cannot # be passed to the NRPE daemon.

# nasty_metachars="|`&><'\\[]{};\r\n"


# INCLUDE CONFIG FILE
# This directive allows you to include definitions from an external config file.

#include=<somefile.cfg>



# INCLUDE CONFIG DIRECTORY
# This directive allows you to include definitions from config files (with a # .cfg extension) in one or more directories (with recursion).

#include_dir=<somedirectory>
#include_dir=<someotherdirectory>



# COMMAND DEFINITIONS
# Command definitions that this daemon will run.  Definitions # are in the following format:
#
# command[<command_name>]=<command_line>
#
# When the daemon receives a request to return the results of <command_name> # it will execute the command specified by the <command_line> argument.
#
# Unlike Nagios, the command line cannot contain macros - it must be # typed exactly as it should be executed.
#
# Note: Any plugins that are used in the command lines must reside # on the machine that this daemon is running on!  The examples below # assume that you have plugins installed in a /usr/local/nagios/libexec # directory.  Also note that you will have to modify the definitions below # to match the argument format the plugins expect.  Remember, these are # examples only!


# The following examples use hardcoded command arguments...

command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10 command[check_load]=/usr/local/nagios/libexec/check_load -r -w .15,.10,.05 -c .30,.25,.20 command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1 command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200


# The following examples allow user-supplied arguments and can # only be used if the NRPE daemon was compiled with support for # command arguments *AND* the dont_blame_nrpe directive in this # config file is set to '1'.  This poses a potential security risk, so # make sure you read the SECURITY file before doing this.

#command[check_users]=/usr/local/nagios/libexec/check_users -w $ARG1$ -c $ARG2$ #command[check_load]=/usr/local/nagios/libexec/check_load -w $ARG1$ -c $ARG2$ #command[check_disk]=/usr/local/nagios/libexec/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$ #command[check_procs]=/usr/local/nagios/libexec/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$

Code: Select all

# default: off
# description: NRPE (Nagios Remote Plugin Executor) service nrpe {
    disable         = yes
    socket_type     = stream
    port            = 5666
    wait            = no
    user            = nagios
    group           = nagios
    server          = /usr/local/nagios/bin/nrpe
    server_args     = -c /usr/local/nagios/etc/nrpe.cfg --inetd
    only_from       = 192.168.1.xxx (this is my nag server)
    log_on_success  =
}
We don't use telnet, however I did install it on the Centos server and got this.

Code: Select all

[root@nag ~]# telnet 192.168.1.xxx 5666
Trying 192.168.1.xxx...
telnet: connect to address 192.168.1.xxx: No route to host
[root@nag ~]#
Thanks
Last edited by dwhitfield on Fri Jun 16, 2017 12:48 pm, edited 1 time in total.
Reason: code blocks FTW
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: check_nrpe : No route to host

Post by dwhitfield »

You may want to set dont_blame_nrpe to 1. I don't think that's your issue though.

What's the output of traceroute 192.168.1.42? Not sure if that's a dummy IP, but that's what was in your first post.

Have you spoken to your network or security team to see if they have any idea what's filtering port 5666?

Also, what's the output of firewall-cmd --get-zones? Please put the output in a code block. The "Code" button is the fifth from the left on the post input screen (between Quote and List).
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: check_nrpe : No route to host

Post by donnyforbes »

Here is the output from traceroute from the server running nag

[root@nag ~]# traceroute 192.168.1.xxx
traceroute to 192.168.1.42 (192.168.1.xxx), 30 hops max, 60 byte packets
1 server_domain_name(192.168.1.xxx) 1.848 ms !X 1.844 ms !X 1.823 ms !X
[root@nag ~]#

Traceroute from Remote Host to Nag Server

[root@huey ~]# traceroute 192.168.1.xxx
traceroute to 192.168.1.xxx (192.168.1.xxx), 30 hops max, 60 byte packets
1 nag (192.168.1.xxx) 0.345 ms !X 0.340 ms !X 0.348 ms !X
[root@huey ~]#


This is from the remote host.

[root@huey ~]# firewall-cmd --zone=public --add-port=5666/tcp --permanent
Warning: ALREADY_ENABLED: 5666:tcp
success
[root@huey ~]#

Is this what you were asking for?
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: check_nrpe : No route to host

Post by dwhitfield »

Did you compile from source and also install our agent or use yum repos? xinetd says it's disabled, but you having something listening on 5666.

You could try just enabling /etc/xinetd.d/nrpe but I suspect that's not going to help anything.

Until 5666 says open instead of filtered, you are likely to have problems, so I think your best bet is to speak with someone on your networking team.
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: check_nrpe : No route to host

Post by donnyforbes »

I installed the source. I used the following documentation NRPE DOCUMENTATION that gets installed on the server.
What do you suspect I do? As you can see the ports are open on the remote hosts?
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: check_nrpe : No route to host

Post by donnyforbes »

I tired this
You could try just enabling /etc/xinetd.d/nrpe but I suspect that's not going to help anything.

this did service xinetd restart & systemctl reload xinetd

Then tried again from the nag server and got the same below.

[root@nag ~]# /usr/local/nagios/libexec/check_nrpe -H 192.168.1.42
connect to address 192.168.1.42 port 5666: No route to host
connect to host 192.168.1.42 port 5666: No route to host

any idea or thoughts on this?
Locked