check_nrpe : No route to host
-
- Posts: 357
- Joined: Tue Jun 13, 2017 2:17 pm
check_nrpe : No route to host
I am trying to deploy a monitored server in test environment before I do production. I am following this guide http://www.tecmint.com/how-to-add-linux ... ed-server/ I have installed Nagios Core on Centos 7 Machine and the machine being monitored is a RHEL 7.3 Machine. I can't pass remote-host's nrpe check. Although both machines can talk to each other using ssh, when testing it says "no route to host on port 5666" see the complete message below.
When I run the following command from my nagios server here is what I get.
[root@nag ~]# /usr/local/nagios/libexec/check_nrpe -H 192.168.1.42
connect to address 192.168.1.42 port 5666: No route to host
connect to host 192.168.1.42 port 5666: No route to host
From the nagios server I am able to use ssh to the remote host
[root@nag ~]# ssh 192.168.1.xxx /usr/local/nagios/libexec/check_procs
root@192.168.1.xxx password:
PROCS OK: 617 processes | procs=617;;;0;
On the remote server I do the following:
firewall-cmd --zone=public --add-port=5666/tcp --permanent
then verify the port 5666/tcp is open
[root@huey ~]# netstat -na | grep 5666
tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN
tcp6 0 0 :::5666 :::* LISTEN
Also, I verify that nrpe is running
[root@huey ~]# ps -ef |grep nrpe
root 17336 21113 0 08:20 pts/1 00:00:00 grep --color=auto nrpe
nagios 37879 1 0 Jun15 ? 00:00:00 /usr/local/nagios/bin/nrpe -c /usr/local/nagios/etc/nrpe.cfg -f
At this point I am out of ideas and not sure what to do.
Nagios Server: CentOS Linux release 7.0.1406 (Core)
Remote Host: Red Hat Enterprise Linux Server release 7.3 (Maipo)
Any help would be great at this time ...
When I run the following command from my nagios server here is what I get.
[root@nag ~]# /usr/local/nagios/libexec/check_nrpe -H 192.168.1.42
connect to address 192.168.1.42 port 5666: No route to host
connect to host 192.168.1.42 port 5666: No route to host
From the nagios server I am able to use ssh to the remote host
[root@nag ~]# ssh 192.168.1.xxx /usr/local/nagios/libexec/check_procs
root@192.168.1.xxx password:
PROCS OK: 617 processes | procs=617;;;0;
On the remote server I do the following:
firewall-cmd --zone=public --add-port=5666/tcp --permanent
then verify the port 5666/tcp is open
[root@huey ~]# netstat -na | grep 5666
tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN
tcp6 0 0 :::5666 :::* LISTEN
Also, I verify that nrpe is running
[root@huey ~]# ps -ef |grep nrpe
root 17336 21113 0 08:20 pts/1 00:00:00 grep --color=auto nrpe
nagios 37879 1 0 Jun15 ? 00:00:00 /usr/local/nagios/bin/nrpe -c /usr/local/nagios/etc/nrpe.cfg -f
At this point I am out of ideas and not sure what to do.
Nagios Server: CentOS Linux release 7.0.1406 (Core)
Remote Host: Red Hat Enterprise Linux Server release 7.3 (Maipo)
Any help would be great at this time ...
Re: check_nrpe : No route to host
Double-check the URL in your OP. The link looks like it got truncated. Knowing what steps were taken might be massively helpful.
Can you share the contents of the following file from your remote RHEL machine:
Could you also try an nmap from the Nagios Core machine to the remote RHEL machine? Just because the port is listening on the RHEL machine doesn't mean the Nagios Core machine can talk over that port.
Might look something like this:
Though if the state is closed or filtered, there may be network equipment between the Nagios machine and the RHEL machine that is blocking such traffic.
Can you share the contents of the following file from your remote RHEL machine:
Code: Select all
/usr/local/nagios/etc/nrpe.cfg
Might look something like this:
Code: Select all
[root@nagios ~]# nmap -p 5666 your_host_here
Starting Nmap 6.40 ( http://nmap.org ) at 2017-06-16 10:48 CDT
Nmap scan report for your_host_here (some_ip_here)
Host is up (0.000044s latency).
PORT STATE SERVICE
5666/tcp open nrpe
Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
-
- Posts: 357
- Joined: Tue Jun 13, 2017 2:17 pm
Re: check_nrpe : No route to host
Hi so I yum install nmap got it installed and did what you asked from the nagios Core machine here is the output
[root@nag ~]# nmap -p 5666 192.168.1.xxx
Starting Nmap 6.40 ( http://nmap.org ) at 2017-06-16 10:11 PDT
Nmap scan report for my_host_here (192.168.1.xxx)
Host is up (0.098s latency).
PORT STATE SERVICE
5666/tcp filtered nrpe
MAC Address: 94:18:82:80:CD:B8 (Unknown)
Nmap done: 1 IP address (1 host up) scanned in 0.82 seconds
Is this what you were looking for?
[root@nag ~]# nmap -p 5666 192.168.1.xxx
Starting Nmap 6.40 ( http://nmap.org ) at 2017-06-16 10:11 PDT
Nmap scan report for my_host_here (192.168.1.xxx)
Host is up (0.098s latency).
PORT STATE SERVICE
5666/tcp filtered nrpe
MAC Address: 94:18:82:80:CD:B8 (Unknown)
Nmap done: 1 IP address (1 host up) scanned in 0.82 seconds
Is this what you were looking for?
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: check_nrpe : No route to host
It looks like you *may* have a firewall in the way.
Can you share the contents of the following file from your remote RHEL machine: /usr/local/nagios/etc/nrpe.cfg
Also, please share /etc/xinetd.d/nrpe
Are you able to telnet on port 5666?
Can you share the contents of the following file from your remote RHEL machine: /usr/local/nagios/etc/nrpe.cfg
Also, please share /etc/xinetd.d/nrpe
Are you able to telnet on port 5666?
-
- Posts: 357
- Joined: Tue Jun 13, 2017 2:17 pm
Re: check_nrpe : No route to host
Code: Select all
#############################################################################
# Sample NRPE Config File
# Written by: Ethan Galstad (nagios@nagios.org) # # Last Modified: 2016-05-10 # # NOTES:
# This is a sample configuration file for the NRPE daemon. It needs to be # located on the remote host that is running the NRPE daemon, not the host # from which the check_nrpe client is being executed.
#############################################################################
# LOG FACILITY
# The syslog facility that should be used for logging purposes.
log_facility=daemon
# LOG FILE
# If a log file is specified in this option, nrpe will write to # that file instead of using syslog.
#log_file=${prefix}/var/nrpe.log
# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the # syslog facility.
# Values: 0=debugging off, 1=debugging on
debug=0
# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID # number. The file is only written if the NRPE daemon is started by the root # user and is running in standalone mode.
pid_file=/usr/local/nagios/var/nrpe.pid
# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-privileged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
server_port=5666
# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface # and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
#server_address=127.0.0.1
# LISTEN QUEUE SIZE
# Listen queue size (backlog) for serving incoming connections.
# You may want to increase this value under high load.
#listen_queue_size=5
# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
nrpe_user=nagios
# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
nrpe_group=nagios
# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames # that are allowed to talk to the NRPE daemon. Network addresses with a bit mask # (i.e. 192.168.1.0/24) are also supported. Hostname wildcards are not currently # supported.
#
# Note: The daemon only does rudimentary checking of the client's IP # address. I would highly recommend adding entries in your /etc/hosts.allow # file to allow only the specified host to connect to the port # you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
allowed_hosts=127.0.0.1,::1
# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients # to specify arguments to commands that are executed. This option only works # if the daemon was configured with the --enable-command-args configure script # option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! *** # Read the SECURITY file for information on some of the security implications # of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments
dont_blame_nrpe=0
# BASH COMMAND SUBSTITUTION
# This option determines whether or not the NRPE daemon will allow clients # to specify arguments that contain bash command substitutions of the form # $(...). This option only works if the daemon was configured with both # the --enable-command-args and --enable-bash-command-substitution configure # script options.
#
# *** ENABLING THIS OPTION IS A HIGH SECURITY RISK! *** # Read the SECURITY file for information on some of the security implications # of enabling this variable.
#
# Values: 0=do not allow bash command substitutions,
# 1=allow bash command substitutions
allow_bash_command_substitution=0
# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the # command line from the command definition.
#
# *** THIS EXAMPLE MAY POSE A POTENTIAL SECURITY RISK, SO USE WITH CAUTION! *** # Usage scenario:
# Execute restricted commmands using sudo. For this to work, you need to add # the nagios user to your /etc/sudoers. An example entry for allowing # execution of the plugins from might be:
#
# nagios ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
#
# This lets the nagios user run all commands in that directory (and only them) # without asking for a password. If you do this, make sure you don't give # random users write access to that directory or its contents!
# command_prefix=/usr/bin/sudo
# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will # allow plugins to finish executing before killing them off.
command_timeout=60
# CONNECTION TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will # wait for a connection to be established before exiting. This is sometimes # seen where a network problem stops the SSL being established even though # all network sessions are connected. This causes the nrpe daemons to # accumulate, eating system resources. Do not set this too low.
connection_timeout=300
# WEAK RANDOM SEED OPTION
# This directive allows you to use SSL even if your system does not have # a /dev/random or /dev/urandom (on purpose or because the necessary patches # were not applied). The random number generator will be seeded from a file # which is either a file pointed to by the environment valiable $RANDFILE # or $HOME/.rnd. If neither exists, the pseudo random number generator will # be initialized and a warning will be issued.
# Values: 0=only seed from /dev/[u]random, 1=also seed from weak randomness
#allow_weak_random_seed=1
# SSL/TLS OPTIONS
# These directives allow you to specify how to use SSL/TLS.
# SSL VERSION
# This can be any of: SSLv2 (only use SSLv2), SSLv2+ (use any version),
# SSLv3 (only use SSLv3), SSLv3+ (use SSLv3 or above), TLSv1 (only use
# TLSv1), TLSv1+ (use TLSv1 or above), TLSv1.1 (only use TLSv1.1),
# TLSv1.1+ (use TLSv1.1 or above), TLSv1.2 (only use TLSv1.2),
# TLSv1.2+ (use TLSv1.2 or above)
# If an "or above" version is used, the best will be negotiated. So if both # ends are able to do TLSv1.2 and use specify SSLv2, you will get TLSv1.2.
# If you are using openssl 1.1.0 or above, the SSLv2 options are not available.
#ssl_version=SSLv2+
# SSL USE ADH
# This is for backward compatibility and is DEPRECATED. Set to 1 to enable # ADH or 2 to require ADH. 1 is currently the default but will be changed # in a later version.
#ssl_use_adh=1
# SSL CIPHER LIST
# This lists which ciphers can be used. For backward compatibility, this # defaults to 'ssl_cipher_list=ALL:!MD5:@STRENGTH' in this version but # will be changed to something like the example below in a later version of NRPE.
#ssl_cipher_list=ALL:!MD5:@STRENGTH
#ssl_cipher_list=ALL:!aNULL:!eNULL:!SSLv2:!LOW:!EXP:!RC4:!MD5:@STRENGTH
# SSL Certificate and Private Key Files
#ssl_cacert_file=/etc/ssl/servercerts/ca-cert.pem
#ssl_cert_file=/etc/ssl/servercerts/nagios-cert.pem
#ssl_privatekey_file=/etc/ssl/servercerts/nagios-key.pem
# SSL USE CLIENT CERTS
# This options determines client certificate usage.
# Values: 0 = Don't ask for or require client certificates (default)
# 1 = Ask for client certificates
# 2 = Require client certificates
#ssl_client_certs=0
# SSL LOGGING
# This option determines which SSL messages are send to syslog. OR values # together to specify multiple options.
# Values: 0x00 (0) = No additional logging (default)
# 0x01 (1) = Log startup SSL/TLS parameters
# 0x02 (2) = Log remote IP address
# 0x04 (4) = Log SSL/TLS version of connections
# 0x08 (8) = Log which cipher is being used for the connection
# 0x10 (16) = Log if client has a certificate
# 0x20 (32) = Log details of client's certificate if it has one
# -1 or 0xff or 0x2f = All of the above
#ssl_logging=0x00
# NASTY METACHARACTERS
# This option allows you to override the list of characters that cannot # be passed to the NRPE daemon.
# nasty_metachars="|`&><'\\[]{};\r\n"
# INCLUDE CONFIG FILE
# This directive allows you to include definitions from an external config file.
#include=<somefile.cfg>
# INCLUDE CONFIG DIRECTORY
# This directive allows you to include definitions from config files (with a # .cfg extension) in one or more directories (with recursion).
#include_dir=<somedirectory>
#include_dir=<someotherdirectory>
# COMMAND DEFINITIONS
# Command definitions that this daemon will run. Definitions # are in the following format:
#
# command[<command_name>]=<command_line>
#
# When the daemon receives a request to return the results of <command_name> # it will execute the command specified by the <command_line> argument.
#
# Unlike Nagios, the command line cannot contain macros - it must be # typed exactly as it should be executed.
#
# Note: Any plugins that are used in the command lines must reside # on the machine that this daemon is running on! The examples below # assume that you have plugins installed in a /usr/local/nagios/libexec # directory. Also note that you will have to modify the definitions below # to match the argument format the plugins expect. Remember, these are # examples only!
# The following examples use hardcoded command arguments...
command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10 command[check_load]=/usr/local/nagios/libexec/check_load -r -w .15,.10,.05 -c .30,.25,.20 command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1 command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200
# The following examples allow user-supplied arguments and can # only be used if the NRPE daemon was compiled with support for # command arguments *AND* the dont_blame_nrpe directive in this # config file is set to '1'. This poses a potential security risk, so # make sure you read the SECURITY file before doing this.
#command[check_users]=/usr/local/nagios/libexec/check_users -w $ARG1$ -c $ARG2$ #command[check_load]=/usr/local/nagios/libexec/check_load -w $ARG1$ -c $ARG2$ #command[check_disk]=/usr/local/nagios/libexec/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$ #command[check_procs]=/usr/local/nagios/libexec/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
Code: Select all
# default: off
# description: NRPE (Nagios Remote Plugin Executor) service nrpe {
disable = yes
socket_type = stream
port = 5666
wait = no
user = nagios
group = nagios
server = /usr/local/nagios/bin/nrpe
server_args = -c /usr/local/nagios/etc/nrpe.cfg --inetd
only_from = 192.168.1.xxx (this is my nag server)
log_on_success =
}
Code: Select all
[root@nag ~]# telnet 192.168.1.xxx 5666
Trying 192.168.1.xxx...
telnet: connect to address 192.168.1.xxx: No route to host
[root@nag ~]#
Last edited by dwhitfield on Fri Jun 16, 2017 12:48 pm, edited 1 time in total.
Reason: code blocks FTW
Reason: code blocks FTW
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: check_nrpe : No route to host
You may want to set dont_blame_nrpe to 1. I don't think that's your issue though.
What's the output of traceroute 192.168.1.42? Not sure if that's a dummy IP, but that's what was in your first post.
Have you spoken to your network or security team to see if they have any idea what's filtering port 5666?
Also, what's the output of firewall-cmd --get-zones? Please put the output in a code block. The "Code" button is the fifth from the left on the post input screen (between Quote and List).
What's the output of traceroute 192.168.1.42? Not sure if that's a dummy IP, but that's what was in your first post.
Have you spoken to your network or security team to see if they have any idea what's filtering port 5666?
Also, what's the output of firewall-cmd --get-zones? Please put the output in a code block. The "Code" button is the fifth from the left on the post input screen (between Quote and List).
-
- Posts: 357
- Joined: Tue Jun 13, 2017 2:17 pm
Re: check_nrpe : No route to host
Here is the output from traceroute from the server running nag
[root@nag ~]# traceroute 192.168.1.xxx
traceroute to 192.168.1.42 (192.168.1.xxx), 30 hops max, 60 byte packets
1 server_domain_name(192.168.1.xxx) 1.848 ms !X 1.844 ms !X 1.823 ms !X
[root@nag ~]#
Traceroute from Remote Host to Nag Server
[root@huey ~]# traceroute 192.168.1.xxx
traceroute to 192.168.1.xxx (192.168.1.xxx), 30 hops max, 60 byte packets
1 nag (192.168.1.xxx) 0.345 ms !X 0.340 ms !X 0.348 ms !X
[root@huey ~]#
This is from the remote host.
[root@huey ~]# firewall-cmd --zone=public --add-port=5666/tcp --permanent
Warning: ALREADY_ENABLED: 5666:tcp
success
[root@huey ~]#
Is this what you were asking for?
[root@nag ~]# traceroute 192.168.1.xxx
traceroute to 192.168.1.42 (192.168.1.xxx), 30 hops max, 60 byte packets
1 server_domain_name(192.168.1.xxx) 1.848 ms !X 1.844 ms !X 1.823 ms !X
[root@nag ~]#
Traceroute from Remote Host to Nag Server
[root@huey ~]# traceroute 192.168.1.xxx
traceroute to 192.168.1.xxx (192.168.1.xxx), 30 hops max, 60 byte packets
1 nag (192.168.1.xxx) 0.345 ms !X 0.340 ms !X 0.348 ms !X
[root@huey ~]#
This is from the remote host.
[root@huey ~]# firewall-cmd --zone=public --add-port=5666/tcp --permanent
Warning: ALREADY_ENABLED: 5666:tcp
success
[root@huey ~]#
Is this what you were asking for?
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: check_nrpe : No route to host
Did you compile from source and also install our agent or use yum repos? xinetd says it's disabled, but you having something listening on 5666.
You could try just enabling /etc/xinetd.d/nrpe but I suspect that's not going to help anything.
Until 5666 says open instead of filtered, you are likely to have problems, so I think your best bet is to speak with someone on your networking team.
You could try just enabling /etc/xinetd.d/nrpe but I suspect that's not going to help anything.
Until 5666 says open instead of filtered, you are likely to have problems, so I think your best bet is to speak with someone on your networking team.
-
- Posts: 357
- Joined: Tue Jun 13, 2017 2:17 pm
Re: check_nrpe : No route to host
I installed the source. I used the following documentation NRPE DOCUMENTATION that gets installed on the server.
What do you suspect I do? As you can see the ports are open on the remote hosts?
What do you suspect I do? As you can see the ports are open on the remote hosts?
-
- Posts: 357
- Joined: Tue Jun 13, 2017 2:17 pm
Re: check_nrpe : No route to host
I tired this
You could try just enabling /etc/xinetd.d/nrpe but I suspect that's not going to help anything.
this did service xinetd restart & systemctl reload xinetd
Then tried again from the nag server and got the same below.
[root@nag ~]# /usr/local/nagios/libexec/check_nrpe -H 192.168.1.42
connect to address 192.168.1.42 port 5666: No route to host
connect to host 192.168.1.42 port 5666: No route to host
any idea or thoughts on this?
You could try just enabling /etc/xinetd.d/nrpe but I suspect that's not going to help anything.
this did service xinetd restart & systemctl reload xinetd
Then tried again from the nag server and got the same below.
[root@nag ~]# /usr/local/nagios/libexec/check_nrpe -H 192.168.1.42
connect to address 192.168.1.42 port 5666: No route to host
connect to host 192.168.1.42 port 5666: No route to host
any idea or thoughts on this?