Issue with Installation of NRPE

[Srinija544]

Moderator Note: Content removed per request

[mbellerue]

The no route to host error suggests that Nagios and the server you want to monitor are not on the same subnet. This can be tricky, especially with the 10.*.*.* IPs. It's muscle memory for most people to tell Linux, "Oh my IP address is 10.42.164.82/24," and suddenly your subnet mask is 255.255.255.0 where all of the other machines are using 255.0.0.0, for example.

So the first thing to do is check the subnet mask of both your Nagios server and the server you're trying to monitor. Run this command on both servers, and let's see the output.

Code: Select all

ip addr

Also make sure you can ping the remote server from your Nagios server.

[Srinija544]

Hi mbellerue,

Thank you for your response.

Please find the subnet mask IP of remote server:
======================================================
ifconfig
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.42.162.136 netmask 255.255.255.0 broadcast 10.42.162.255
inet6 fe80::93fb:dc67:aa7a:5c09 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:b9:dc:42 txqueuelen 1000 (Ethernet)
RX packets 4021423 bytes 647072166 (617.0 MiB)
RX errors 0 dropped 2618 overruns 0 frame 0
TX packets 179229 bytes 21838123 (20.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 111 bytes 21313 (20.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 111 bytes 21313 (20.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

And also find the subnet mask IP of Nagios server:
=====================================================================
ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.129.1.66 netmask 255.255.255.0 broadcast 10.129.1.255
inet6 fe80::a44:a9ab:b5d6:aa72 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:b9:68:be txqueuelen 1000 (Ethernet)
RX packets 10126939031 bytes 1316578453457 (1.1 TiB)
RX errors 0 dropped 40964 overruns 0 frame 0
TX packets 14127898737 bytes 2056899463399 (1.8 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 6693057186 bytes 1076516320413 (1002.5 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6693057186 bytes 1076516320413 (1002.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
===============================================================================
Please check and let me know for further information required to solve this issue.

Regards,
Srinija

[mbellerue]

Okay, so they are on different subnets. Is there a router between that they use to find each other? Can you ping 10.42.162.136 from your Nagios server?

[Srinija544]

Hi mbellerue,

Yes we are able to ping the mentioned IP: 10.42.162.136 from our Nagios Server.

[root@dc1prd484 ~]# ping 10.42.162.136
PING 10.42.162.136 (10.42.162.136) 56(84) bytes of data.
64 bytes from 10.42.162.136: icmp_seq=1 ttl=60 time=11.0 ms
64 bytes from 10.42.162.136: icmp_seq=2 ttl=60 time=11.0 ms
64 bytes from 10.42.162.136: icmp_seq=3 ttl=60 time=12.0 ms
64 bytes from 10.42.162.136: icmp_seq=4 ttl=60 time=11.3 ms
64 bytes from 10.42.162.136: icmp_seq=5 ttl=60 time=11.6 ms
^C
--- 10.42.162.136 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 11.013/11.409/12.016/0.408 ms


Regards,
Srinija

[lmiltchev]

NRPE can run under xinetd or as a "standalone daemon" but not as both... I suspect that this is what is happening here. You have nrpe service running:

/usr/local/nagios/bin/nrpe -c /usr/local/nagios/etc/nrpe.cfg -f

and it seems that you are also trying to run nrpe under xinetd:

Dec 11 15:37:01 cr2prd131.tower.lan xinetd[18435]: bind failed (Address already in use (errno = 98)). service = nrpe

How did you install NRPE on the client? What documentation/guide/tutorial did you follow?

Can you run the following commands on the client (remote box), and show the output in code wraps?

Code: Select all

netstat -nap | grep 5666
iptables -L -v -n
cat /usr/local/nagios/etc/nrpe.cfg
cat /etc/xinetd.d/nrpe
ping 10.129.1.66

Also, run the following commands on the Nagios XI server, and show the output:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H 10.42.162.136
nmap 10.42.162.136 -p 5666

[Srinija544]

Hi lmiltchev,

Please find the below of remote server details:

Code: Select all

[root@*******~]# netstat -nap | grep 5666
tcp        0      0 0.0.0.0:5666            0.0.0.0:*               LISTEN      12674/nrpe
tcp6       0      0 :::5666                 :::*                    LISTEN      12674/nrpe

Code: Select all

[root@*******~]#iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
47101 6481K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
 1350  129K ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0
    2   120 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
  226 12476 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
 139K   28M REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 77428 packets, 9868K bytes)
 pkts bytes target     prot opt in     out     source               destination

Code: Select all

cat /usr/local/nagios/etc/nrpe.cfg

[root@********* ~]# cat /usr/local/nagios/etc/nrpe.cfg
#############################################################################
#
#  Sample NRPE Config File
#
#  Notes:
#
#  This is a sample configuration file for the NRPE daemon.  It needs to be
#  located on the remote host that is running the NRPE daemon, not the host
#  from which the check_nrpe client is being executed.
#
#############################################################################


# LOG FACILITY
# The syslog facility that should be used for logging purposes.

log_facility=daemon



# LOG FILE
# If a log file is specified in this option, nrpe will write to
# that file instead of using syslog.

#log_file=/usr/local/nagios/var/nrpe.log



# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the
# syslog facility.
# Values: 0=debugging off, 1=debugging on

debug=0



# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID
# number.  The file is only written if the NRPE daemon is started by the root
# user and is running in standalone mode.

pid_file=/usr/local/nagios/var/nrpe.pid



# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-privileged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

server_port=5666



# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface
# and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

#server_address=127.0.0.1



# LISTEN QUEUE SIZE
# Listen queue size (backlog) for serving incoming connections.
# You may want to increase this value under high load.

#listen_queue_size=5



# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_user=nagios



# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_group=nagios



# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon. Network addresses with a bit mask
# (i.e. 192.168.1.0/24) are also supported. Hostname wildcards are not currently
# supported.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address.  I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1,10.129.1.66,10.129.1.67,10.137.1.60



# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed.  This option only works
# if the daemon was configured with the --enable-command-args configure script
# option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=1



# BASH COMMAND SUBSTITUTION
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments that contain bash command substitutions of the form
# $(...).  This option only works if the daemon was configured with both
# the --enable-command-args and --enable-bash-command-substitution configure
# script options.
#
# *** ENABLING THIS OPTION IS A HIGH SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow bash command substitutions,
#         1=allow bash command substitutions

allow_bash_command_substitution=0



# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the
# command line from the command definition.
#
# *** THIS EXAMPLE MAY POSE A POTENTIAL SECURITY RISK, SO USE WITH CAUTION! ***
# Usage scenario:
# Execute restricted commmands using sudo.  For this to work, you need to add
# the nagios user to your /etc/sudoers.  An example entry for allowing
# execution of the plugins from might be:
#
# nagios          ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
#
# This lets the nagios user run all commands in that directory (and only them)
# without asking for a password.  If you do this, make sure you don't give
# random users write access to that directory or its contents!

# command_prefix=/usr/bin/sudo


# MAX COMMANDS
# This specifies how many children processes may be spawned at any one
# time, essentially limiting the fork()s that occur.
# Default (0) is set to unlimited
# max_commands=0



# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# allow plugins to finish executing before killing them off.

command_timeout=60



# CONNECTION TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# wait for a connection to be established before exiting. This is sometimes
# seen where a network problem stops the SSL being established even though
# all network sessions are connected. This causes the nrpe daemons to
# accumulate, eating system resources. Do not set this too low.

connection_timeout=300



# WEAK RANDOM SEED OPTION
# This directive allows you to use SSL even if your system does not have
# a /dev/random or /dev/urandom (on purpose or because the necessary patches
# were not applied). The random number generator will be seeded from a file
# which is either a file pointed to by the environment valiable $RANDFILE
# or $HOME/.rnd. If neither exists, the pseudo random number generator will
# be initialized and a warning will be issued.
# Values: 0=only seed from /dev/[u]random, 1=also seed from weak randomness

#allow_weak_random_seed=1



# SSL/TLS OPTIONS
# These directives allow you to specify how to use SSL/TLS.

# SSL VERSION
# This can be any of: SSLv2 (only use SSLv2), SSLv2+ (use any version),
#        SSLv3 (only use SSLv3), SSLv3+ (use SSLv3 or above), TLSv1 (only use
#        TLSv1), TLSv1+ (use TLSv1 or above), TLSv1.1 (only use TLSv1.1),
#        TLSv1.1+ (use TLSv1.1 or above), TLSv1.2 (only use TLSv1.2),
#        TLSv1.2+ (use TLSv1.2 or above)
# If an "or above" version is used, the best will be negotiated. So if both
# ends are able to do TLSv1.2 and use specify SSLv2, you will get TLSv1.2.
# If you are using openssl 1.1.0 or above, the SSLv2 options are not available.

#ssl_version=SSLv2+

# SSL USE ADH
# This is for backward compatibility and is DEPRECATED. Set to 1 to enable
# ADH or 2 to require ADH. 1 is currently the default but will be changed
# in a later version.

#ssl_use_adh=1

# SSL CIPHER LIST
# This lists which ciphers can be used. For backward compatibility, this
# defaults to 'ssl_cipher_list=ALL:!MD5:@STRENGTH' for < OpenSSL 1.1.0,
# and 'ssl_cipher_list=ALL:!MD5:@STRENGTH:@SECLEVEL=0' for OpenSSL 1.1.0 and
# greater.

#ssl_cipher_list=ALL:!MD5:@STRENGTH
#ssl_cipher_list=ALL:!MD5:@STRENGTH:@SECLEVEL=0
#ssl_cipher_list=ALL:!aNULL:!eNULL:!SSLv2:!LOW:!EXP:!RC4:!MD5:@STRENGTH

# SSL Certificate and Private Key Files

#ssl_cacert_file=/etc/ssl/servercerts/ca-cert.pem
#ssl_cert_file=/etc/ssl/servercerts/nagios-cert.pem
#ssl_privatekey_file=/etc/ssl/servercerts/nagios-key.pem

# SSL USE CLIENT CERTS
# This options determines client certificate usage.
# Values: 0 = Don't ask for or require client certificates (default)
#         1 = Ask for client certificates
#         2 = Require client certificates

#ssl_client_certs=0

# SSL LOGGING
# This option determines which SSL messages are send to syslog. OR values
# together to specify multiple options.

# Values: 0x00 (0)  = No additional logging (default)
#         0x01 (1)  = Log startup SSL/TLS parameters
#         0x02 (2)  = Log remote IP address
#         0x04 (4)  = Log SSL/TLS version of connections
#         0x08 (8)  = Log which cipher is being used for the connection
#         0x10 (16) = Log if client has a certificate
#         0x20 (32) = Log details of client's certificate if it has one
#         -1 or 0xff or 0x2f = All of the above

#ssl_logging=0x00



# NASTY METACHARACTERS
# This option allows you to override the list of characters that cannot
# be passed to the NRPE daemon.

# nasty_metachars="|`&><'\\[]{};\r\n"



# COMMAND DEFINITIONS
# Command definitions that this daemon will run.  Definitions
# are in the following format:
#
# command[<command_name>]=<command_line>
#
# When the daemon receives a request to return the results of <command_name>
# it will execute the command specified by the <command_line> argument.
#
# Unlike Nagios, the command line cannot contain macros - it must be
# typed exactly as it should be executed.
#
# Note: Any plugins that are used in the command lines must reside
# on the machine that this daemon is running on!  The examples below
# assume that you have plugins installed in a /usr/local/nagios/libexec
# directory.  Also note that you will have to modify the definitions below
# to match the argument format the plugins expect.  Remember, these are
# examples only!


# The following examples use hardcoded command arguments...
# This is by far the most secure method of using NRPE

#command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
#command[check_load]=/usr/local/nagios/libexec/check_load -r -w .15,.10,.05 -c .30,.25,.20
command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1
#command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
#command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200

command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_load]=/usr/local/nagios/libexec/check_load -w 4,4,4 -c 5,5,5
command[check_swap]=/usr/local/nagios/libexec/check_swap -w 20% -c 10%
command[check_disk_root]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /
command[check_disk_boot]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /boot
command[check_disk_ora0]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /ora0
command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 340 -c 350
#command[check_mem]=/usr/local/nagios/libexec/check_mem.pl  -w 5 -c 6
command[check_mem]=/usr/local/nagios/libexec/check_mem.pl -f -C -w 20 -c 10
command[check_procs_java_percent]=/usr/local/nagios/libexec/check_procs -w 40 -c 50 -a java --metric=CPU



# The following examples allow user-supplied arguments and can
# only be used if the NRPE daemon was compiled with support for
# command arguments *AND* the dont_blame_nrpe directive in this
# config file is set to '1'.  This poses a potential security risk, so
# make sure you read the SECURITY file before doing this.

### MISC SYSTEM METRICS ###
#command[check_users]=/usr/local/nagios/libexec/check_users $ARG1$
#command[check_load]=/usr/local/nagios/libexec/check_load $ARG1$
#command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
#command[check_swap]=/usr/local/nagios/libexec/check_swap $ARG1$
#command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats.sh $ARG1$
#command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$

### GENERIC SERVICES ###
#command[check_init_service]=sudo /usr/local/nagios/libexec/check_init_service $ARG1$
#command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$

### SYSTEM UPDATES ###
#command[check_yum]=/usr/local/nagios/libexec/check_yum
#command[check_apt]=/usr/local/nagios/libexec/check_apt

### PROCESSES ###
#command[check_all_procs]=/usr/local/nagios/libexec/custom_check_procs
#command[check_procs]=/usr/local/nagios/libexec/check_procs $ARG1$

### OPEN FILES ###
#command[check_open_files]=/usr/local/nagios/libexec/check_open_files.pl $ARG1$

### NETWORK CONNECTIONS ###
#command[check_netstat]=/usr/local/nagios/libexec/check_netstat.pl -p $ARG1$ $ARG2$

### ASTERISK ###
#command[check_asterisk]=/usr/local/nagios/libexec/check_asterisk.pl $ARG1$
#command[check_sip]=/usr/local/nagios/libexec/check_sip $ARG1$
#command[check_asterisk_sip_peers]=sudo /usr/local/nagios/libexec/check_asterisk_sip_peers.sh $ARG1$
#command[check_asterisk_version]=/usr/local/nagios/libexec/nagisk.pl -c version
#command[check_asterisk_peers]=/usr/local/nagios/libexec/nagisk.pl -c peers
#command[check_asterisk_channels]=/usr/local/nagios/libexec/nagisk.pl -c channels
#command[check_asterisk_zaptel]=/usr/local/nagios/libexec/nagisk.pl -c zaptel
#command[check_asterisk_span]=/usr/local/nagios/libexec/nagisk.pl -c span -s 1



# INCLUDE CONFIG FILE
# This directive allows you to include definitions from an external config file.

#include=<somefile.cfg>



# INCLUDE CONFIG DIRECTORY
# This directive allows you to include definitions from config files (with a
# .cfg extension) in one or more directories (with recursion).

#include_dir=<somedirectory>
#include_dir=<someotherdirectory>

Code: Select all

cat /etc/xinetd.d/nrpe

[root@*************~]# cat /etc/xinetd.d/nrpe
# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
        flags           = REUSE
        socket_type     = stream
        port            = 5666
        wait            = no
        user            = nagios
        group           = nagios
        server          = /usr/local/nagios/bin/nrpe
        server_args     = -c /usr/local/nagios/etc/nrpe.cfg --inetd
        log_on_failure  += USERID
    disable         = yes
    per_source      = unlimited
        only_from       =127.0.0.1
        only_from       += ::1
        only_from       +=10.137.1.38 10.129.1.66 10.129.1.67 10.137.1.60
}

Code: Select all

[root@************** ~]# ping 10.129.1.66
PING 10.129.1.66 (10.129.1.66) 56(84) bytes of data.
64 bytes from 10.129.1.66: icmp_seq=1 ttl=59 time=11.0 ms
64 bytes from 10.129.1.66: icmp_seq=2 ttl=59 time=10.9 ms
64 bytes from 10.129.1.66: icmp_seq=3 ttl=59 time=11.2 ms
64 bytes from 10.129.1.66: icmp_seq=4 ttl=59 time=11.0 ms
64 bytes from 10.129.1.66: icmp_seq=5 ttl=59 time=11.2 ms
^C
--- 10.129.1.66 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 10.935/11.098/11.239/0.133 ms

Nagios XI Server:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H 10.42.162.136

[root@*************** ~]# /usr/local/nagios/libexec/check_nrpe -H 10.42.162.136
connect to address 10.42.162.136 port 5666: No route to host
connect to host 10.42.162.136 port 5666: No route to host

Code: Select all

nmap 10.42.162.136 -p 5666

[root@*************** ~]# nmap 10.42.162.136 -p 5666

Starting Nmap 6.47 ( http://nmap.org ) at 2019-12-17 12:05 AEDT
Nmap scan report for cr2prd131.tower.lan (10.42.162.136)
Host is up (0.011s latency).
PORT     STATE    SERVICE
5666/tcp filtered nrpe

Nmap done: 1 IP address (1 host up) scanned in 0.15 seconds

Please find the attached document that i followed and help.

Regards,
Srinija.

[lmiltchev]

I see two main issues here:

1. NRPE runs as a "standalone" daemon on your system, but you also are trying to run it under xinetd. Remove the /etc/xinetd.d/nrpe file:

Code: Select all

rm -f /etc/xinetd.d/nrpe

and restart both, the xinetd and nrpe:

Code: Select all

service xinetd restart
service nrpe restart

This should fix the problem with both, NRPE and xinetd listening on the same port.

2. The second issue is that TCP port 5666 is not open (by looking at your firewall rules). Open the port (or temporarily stop the firewall for troubleshooting purposes), and test your check from the command line on the Nagios XI server:

- make sure the port is open:

Code: Select all

nmap 10.42.162.136 -p 5666

-test check_nrpe:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H 10.42.162.136

[Srinija544]

Hi,

I ran the commands as mentioned below, the error output was changed but still the alerts exist in CRITICAL state.

Please find the screenshot and help me in solving.

Regards,
Srinija.

[lmiltchev]

Open a ssh (putty) session to your Nagios XI server, run the following commands, and show the output:

Code: Select all

ip addr
nmap 10.42.162.136 -p 5666
/usr/local/nagios/libexec/check_nrpe -H 10.42.162.136

Does the status change if you click on the service from the web UI, and force an immediate check?