Error in AWS EC2 Uubuntu Monitoring

[ivp2015]

[root@ivpnagiosprod varen]# /usr/local/nagios/libexec/check_nrpe -H 54.76.125.226 -u
CHECK_NRPE: Error - Could not complete SSL handshake.
[root@ivpnagiosprod varen]# /usr/local/nagios/libexec/check_nrpe -H 54.76.125.226 -n
CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages.


Port 5666, 5667, 5668 are opened and listing,

Thanks

[mcapra]

Can you tell us how you installed NRPE on this machine? Are these machines on the same subnet?

Can you share the output of the following commands executed from the remote Ubuntu machine you are attempting to monitor:

Code: Select all

cat /etc/xinetd.d/nrpe
cat /usr/local/nagios/etc/nrpe.cfg
ps aux | grep nrpe
ps aux | grep xinetd

Can you also share the outputs of the following command executed from your Nagios XI machine (you may need to yum install nmap):

Code: Select all

nmap -v 54.76.125.226 -p 5666
curl ipinfo.io/ip
ip addr

Feel free to PM any potentially sensitive information.

[ivp2015]

Hi Team,

I have followed the instruction to install NRPE agent from Nagios document,

When we going to add new host in Nagios document and setup show in our portal,

And the ubuntu machine is on cloud also we have monitor other windows host they are working fine but on ubuntu we got error,

Please find bellow the out of given commands

_____________________________________________________________________

Code: Select all

cat /etc/xinetd.d/nrpe


root@ip-172-31-28-18:~# cat /etc/xinetd.d/nrpe
# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
        flags           = REUSE
        socket_type     = stream
        port            = 5666
        wait            = no
        user            = nagios
        group           = nagios
        server          = /usr/local/nagios/bin/nrpe
        server_args     = -c /usr/local/nagios/etc/nrpe.cfg --inetd
        log_on_failure  += USERID
        disable         = no
        only_from       = 127.0.0.1, 115.114.12.4, 182.74.168.35

______________________________________________________________

Code: Select all

cat /usr/local/nagios/etc/nrpe.cfg


root@ip-172-31-28-18:~# cat /usr/local/nagios/etc/nrpe.cfg
#############################################################################
# Sample NRPE Config File
# Written by: Ethan Galstad ([email protected])
#
# Last Modified: 11-23-2007
#
# NOTES:
# This is a sample configuration file for the NRPE daemon.  It needs to be
# located on the remote host that is running the NRPE daemon, not the host
# from which the check_nrpe client is being executed.
#############################################################################


# LOG FACILITY
# The syslog facility that should be used for logging purposes.

log_facility=daemon



# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID
# number.  The file is only written if the NRPE daemon is started by the root
# user and is running in standalone mode.

pid_file=/var/run/nrpe.pid



# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-priviledged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

server_port=5666



# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface
# and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

#server_address=127.0.0.1



# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_user=nagios



# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_group=nagios



# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon. Network addresses with a bit mask
# (i.e. 192.168.1.0/24) are also supported. Hostname wildcards are not currently                                                                                                                     
# supported.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address.  I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1



# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed.  This option only works
# if the daemon was configured with the --enable-command-args configure script
# option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=0



# BASH COMMAND SUBTITUTION
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments that contain bash command substitutions of the form
# $(...).  This option only works if the daemon was configured with both
# the --enable-command-args and --enable-bash-command-substitution configure
# script options.
#
# *** ENABLING THIS OPTION IS A HIGH SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow bash command substitutions,
#         1=allow bash command substitutions

allow_bash_command_substitution=0



# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the
# command line from the command definition.
#
# *** THIS EXAMPLE MAY POSE A POTENTIAL SECURITY RISK, SO USE WITH CAUTION! ***
# Usage scenario:
# Execute restricted commmands using sudo.  For this to work, you need to add
# the nagios user to your /etc/sudoers.  An example entry for alllowing
# execution of the plugins from might be:
#
# nagios          ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
#
# This lets the nagios user run all commands in that directory (and only them)
# without asking for a password.  If you do this, make sure you don't give
# random users write access to that directory or its contents!

# command_prefix=/usr/bin/sudo



# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the
# syslog facility.
# Values: 0=debugging off, 1=debugging on

debug=0



# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# allow plugins to finish executing before killing them off.

command_timeout=60



# CONNECTION TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# wait for a connection to be established before exiting. This is sometimes
# seen where a network problem stops the SSL being established even though
# all network sessions are connected. This causes the nrpe daemons to
# accumulate, eating system resources. Do not set this too low.

connection_timeout=300



# WEEK RANDOM SEED OPTION
# This directive allows you to use SSL even if your system does not have
# a /dev/random or /dev/urandom (on purpose or because the necessary patches
# were not applied). The random number generator will be seeded from a file
# which is either a file pointed to by the environment valiable $RANDFILE
# or $HOME/.rnd. If neither exists, the pseudo random number generator will
# be initialized and a warning will be issued.
# Values: 0=only seed from /dev/[u]random, 1=also seed from weak randomness

#allow_weak_random_seed=1



# INCLUDE CONFIG FILE
# This directive allows you to include definitions from an external config file.

#include=<somefile.cfg>



# INCLUDE CONFIG DIRECTORY
# This directive allows you to include definitions from config files (with a
# .cfg extension) in one or more directories (with recursion).

#include_dir=<somedirectory>
#include_dir=<someotherdirectory>



# COMMAND DEFINITIONS
# Command definitions that this daemon will run.  Definitions
# are in the following format:
#
# command[<command_name>]=<command_line>
#
# When the daemon receives a request to return the results of <command_name>
# it will execute the command specified by the <command_line> argument.
#
# Unlike Nagios, the command line cannot contain macros - it must be
# typed exactly as it should be executed.
#
# Note: Any plugins that are used in the command lines must reside
# on the machine that this daemon is running on!  The examples below
# assume that you have plugins installed in a /usr/local/nagios/libexec
# directory.  Also note that you will have to modify the definitions below
# to match the argument format the plugins expect.  Remember, these are
# examples only!


# The following examples use hardcoded command arguments...

command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/h                                                                                                                     da1
command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s                                                                                                                      Z
command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200


# The following examples allow user-supplied arguments and can
# only be used if the NRPE daemon was compiled with support for
# command arguments *AND* the dont_blame_nrpe directive in this
# config file is set to '1'.  This poses a potential security risk, so
# make sure you read the SECURITY file before doing this.

#command[check_users]=/usr/local/nagios/libexec/check_users -w $ARG1$ -c $ARG2$
#command[check_load]=/usr/local/nagios/libexec/check_load -w $ARG1$ -c $ARG2$
#command[check_disk]=/usr/local/nagios/libexec/check_disk -w $ARG1$ -c $ARG2$ -p                                                                                                                      $ARG3$
#command[check_procs]=/usr/local/nagios/libexec/check_procs -w $ARG1$ -c $ARG2$ 

____________________________________________________________________________________

Code: Select all

ps aux | grep nrpe


root@ip-172-31-28-18:~# ps aux | grep nrpe
root     14897  0.0  0.0  10460   928 pts/0    S+   11:33   0:00 grep --color=auto nrpe
nagios   23238  0.0  0.0  23332  1136 ?        Ss   Feb25   0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d                                                                                                                 -s $ARG3$

 

_________________________________________________________________________________________________________

Code: Select all

ps aux | grep xinetd


root@ip-172-31-28-18:~# ps aux | grep xinetd
root     15099  0.0  0.0  10460   932 pts/0    S+   11:34   0:00 grep --color=auto xinetd
root     23458  0.0  0.0  14988   912 ?        Ss   Feb25   0:00 /usr/sbin/xinetd -dontfork -pidfile /var/run/xinetd.pid -stayalive -inetd_compat -inetd_ipv6

___________________
From Nagios Server

Code: Select all

nmap -v 54.76.125.226 -p 5666


[root@ivpnagiosprod varen]# nmap -v 54.76.125.226 -p 5666

Starting Nmap 5.51 ( http://nmap.org ) at 2017-03-01 17:01 IST
Initiating Ping Scan at 17:01
Scanning 54.76.125.226 [4 ports]
Completed Ping Scan at 17:01, 0.17s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 17:01
Completed Parallel DNS resolution of 1 host. at 17:01, 6.78s elapsed
Initiating SYN Stealth Scan at 17:01
Scanning ec2-54-76-125-226.eu-west-1.compute.amazonaws.com (54.76.125.226) [1 po                                                                                                                     rt]
Discovered open port 5666/tcp on 54.76.125.226
Completed SYN Stealth Scan at 17:01, 0.17s elapsed (1 total ports)
Nmap scan report for ec2-54-76-125-226.eu-west-1.compute.amazonaws.com (54.76.12                                                                                                                     5.226)
Host is up (0.16s latency).
PORT     STATE SERVICE
5666/tcp open  nrpe

Read data files from: /usr/share/nmap
Nmap done: 1 IP address (1 host up) scanned in 7.29 seconds
           Raw packets sent: 5 (196B) | Rcvd: 3 (124B)

_________________________________________________________________________________________________________________
curl ipinfo.io/ip

Code: Select all

[root@ivpnagiosprod varen]# curl ipinfo.io/ip
115.114.12.4

_________________________________________________________________________________________________________

Code: Select all

ip addr


[root@ivpnagiosprod varen]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 1000
    link/ether 00:50:56:b3:7a:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.90/24 brd 192.168.0.255 scope global eth0
    inet6 fe80::250:56ff:feb3:7aa3/64 scope link
       valid_lft forever preferred_lft forever

[mcapra]

Since NRPE is running as a stand-alone daemon, the xinetd configuration isn't being used for anything at the moment. That configuration would only apply if NRPE was running under xinetd.

With NRPE running as a stand-alone daemon, you will need to alter your nrpe.cfg to include the Nagios XI machine's IP address. Specifically, the allowed_hosts directive. Be sure to restart NRPE after making the changes.

[ivp2015]

Hi Team,

We have done changes in nrpe.cfg file, put our public IP's in allowed_hosts but still its not working,

Code: Select all

root@ip-172-31-28-18:~# cat /etc/nagios/nrpe.cfg
#############################################################################
# Sample NRPE Config File
# Written by: Ethan Galstad ([email protected])
#
# Last Modified: 11-23-2007
#
# NOTES:
# This is a sample configuration file for the NRPE daemon.  It needs to be
# located on the remote host that is running the NRPE daemon, not the host
# from which the check_nrpe client is being executed.
#############################################################################


# LOG FACILITY
# The syslog facility that should be used for logging purposes.

log_facility=daemon



# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID
# number.  The file is only written if the NRPE daemon is started by the root
# user and is running in standalone mode.

pid_file=/var/run/nagios/nrpe.pid



# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-priviledged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

server_port=5666



# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface
# and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

#server_address=127.0.0.1



# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_user=nagios



# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_group=nagios



# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon. Network addresses with a bit mask
# (i.e. 192.168.1.0/24) are also supported. Hostname wildcards are not currently
# supported.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address.  I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1, 182.74.168.35, 115.114.12.4, 54.76.125.226


# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed.  This option only works
# if the daemon was configured with the --enable-command-args configure script
# option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=0



# BASH COMMAND SUBTITUTION
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments that contain bash command substitutions of the form
# $(...).  This option only works if the daemon was configured with both
# the --enable-command-args and --enable-bash-command-substitution configure
# script options.
#
# *** ENABLING THIS OPTION IS A HIGH SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow bash command substitutions,
#         1=allow bash command substitutions

allow_bash_command_substitution=0



# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the
# command line from the command definition.
#
# *** THIS EXAMPLE MAY POSE A POTENTIAL SECURITY RISK, SO USE WITH CAUTION! ***
# Usage scenario:
# Execute restricted commmands using sudo.  For this to work, you need to add
# the nagios user to your /etc/sudoers.  An example entry for alllowing
# execution of the plugins from might be:
#
# nagios          ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
#
# This lets the nagios user run all commands in that directory (and only them)
# without asking for a password.  If you do this, make sure you don't give
# random users write access to that directory or its contents!

# command_prefix=/usr/bin/sudo



# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the
# syslog facility.
# Values: 0=debugging off, 1=debugging on

debug=0



# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# allow plugins to finish executing before killing them off.

command_timeout=60



# CONNECTION TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# wait for a connection to be established before exiting. This is sometimes
# seen where a network problem stops the SSL being established even though
# all network sessions are connected. This causes the nrpe daemons to
# accumulate, eating system resources. Do not set this too low.

connection_timeout=300



# WEEK RANDOM SEED OPTION
# This directive allows you to use SSL even if your system does not have
# a /dev/random or /dev/urandom (on purpose or because the necessary patches
# were not applied). The random number generator will be seeded from a file
# which is either a file pointed to by the environment valiable $RANDFILE
# or $HOME/.rnd. If neither exists, the pseudo random number generator will
# be initialized and a warning will be issued.
# Values: 0=only seed from /dev/[u]random, 1=also seed from weak randomness

#allow_weak_random_seed=1



# INCLUDE CONFIG FILE
# This directive allows you to include definitions from an external config file.

#include=<somefile.cfg>



# INCLUDE CONFIG DIRECTORY
# This directive allows you to include definitions from config files (with a
# .cfg extension) in one or more directories (with recursion).

#include_dir=<somedirectory>
#include_dir=<someotherdirectory>



# COMMAND DEFINITIONS
# Command definitions that this daemon will run.  Definitions
# are in the following format:
#
# command[<command_name>]=<command_line>
#
# When the daemon receives a request to return the results of <command_name>
# it will execute the command specified by the <command_line> argument.
#
# Unlike Nagios, the command line cannot contain macros - it must be
# typed exactly as it should be executed.
#
# Note: Any plugins that are used in the command lines must reside
# on the machine that this daemon is running on!  The examples below
# assume that you have plugins installed in a /usr/local/nagios/libexec
# directory.  Also note that you will have to modify the definitions below
# to match the argument format the plugins expect.  Remember, these are
# examples only!


# The following examples use hardcoded command arguments...

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200
command[keystone]=/usr/lib64/nagios/plugins/check_procs -c 1: -w 3: -C keystone-all

# The following examples allow user-supplied arguments and can
# only be used if the NRPE daemon was compiled with support for
# command arguments *AND* the dont_blame_nrpe directive in this
# config file is set to '1'.  This poses a potential security risk, so
# make sure you read the SECURITY file before doing this.

#command[check_users]=/usr/lib/nagios/plugins/check_users -w $ARG1$ -c $ARG2$
#command[check_load]=/usr/lib/nagios/plugins/check_load -w $ARG1$ -c $ARG2$
#command[check_disk]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
#command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$

#
# local configuration:
#       if you'd prefer, you can instead place directives here
include=/etc/nagios/nrpe_local.cfg

#
# you can place your config snipplets into nrpe.d/
# only snipplets ending in .cfg will get included
include_dir=/etc/nagios/nrpe.d/

root@ip-172-31-28-18:~#

[mcapra]

From the remote machine running NRPE (54.76.125.226), can you run the following commands:

Code: Select all

sed -i 's/debug=0/debug=1/g' /usr/local/nagios/etc/nrpe.cfg
service nrpe restart

That should enable more verbose debugging in NRPE. Then, from the Nagios XI machine, can you run:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H 54.76.125.226 -n

Then, again from the remote machine running NRPE (54.76.125.226), share the output of:

Code: Select all

tail -n 100 /var/log/syslog | grep nrpe

[ivp2015]

When run command from remote machine show error

root@ip-172-31-28-18:~# sed -i 's/debug=0/debug=1/g' /usr/local/nagios/etc/nrpe.cfg
root@ip-172-31-28-18:~# service nrpe restart
nrpe: unrecognized service
root@ip-172-31-28-18:~#

From Nagios Server

[root@ivpnagiosprod varen]# /usr/local/nagios/libexec/check_nrpe -H 54.76.125.226 -n
CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages.


from Remove machine

root@ip-172-31-28-18:~# tail -n 100 /var/log/syslog | grep nrpe
Mar 7 10:07:46 ip-172-31-28-18 nrpe[1338]: Host 115.114.12.4 is not allowed to talk to us!
Mar 7 10:07:50 ip-172-31-28-18 nrpe[1340]: Host 115.114.12.4 is not allowed to talk to us!

[mcapra]

How are you starting/stopping NRPE? It doesn't look like the init.d script exists.

At any rate, I think your nrpe configuration file isn't getting picked up correctly. You'll probably need to restart the nrpe process some how to get the changes in. For whatever reason, the daemon doesn't think your Nagios XI host is allowed despite it existing in the configuration file.

[ivp2015]

We use service xinetd restart

please suggest what's going wrong,

Thanks

[mcapra]

But as we established in a previous post that nrpe is running as a stand-alone daemon; It's not running under xinetd so restarting xinetd shouldn't be doing much of anything:

Code: Select all

ps aux | grep nrpe


root@ip-172-31-28-18:~# ps aux | grep nrpe
root     14897  0.0  0.0  10460   928 pts/0    S+   11:33   0:00 grep --color=auto nrpe
nagios   23238  0.0  0.0  23332  1136 ?        Ss   Feb25   0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d 

Is this information incorrect? I've been working with the understanding that NRPE is running stand-alone. If NRPE is running under xinetd and as a stand-alone daemon at the same time, that's going to cause problems.

Can you share the output of:

Code: Select all

ls -al /etc/init.d

If there isn't an init script in there for nrpe, i'm wondering how/why there is a stand-alone NRPE daemon running. I would suggest killing that existing NRPE process, running a service xinetd restart, then executing this from your Nagios XI machine:

/usr/local/nagios/libexec/check_nrpe -H 54.76.125.226 -n

The contents of the xinetd file (/etc/xinetd.d/nrpe) might also be useful to see. It could be that NRPE is actually running under xinetd and the changes for allowed hosts needs to be made there.