Error in AWS EC2 Uubuntu Monitoring

[ivp2015]

Please find out the output of given command,

1) netstat -anp |grep 5666

root@ip-172-31-28-18:~# netstat -anp |grep 5666
tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN 5341/nrpe
tcp6 0 0 :::5666 :::* LISTEN 5341/nrpe
root@ip-172-31-28-18:~#

2) ps -ef --cols=300 | grep nrpe

root@ip-172-31-28-18:~# ps -ef --cols=300 | grep nrpe
nagios 5341 1 0 Mar13 ? 00:00:01 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
root 21090 21004 0 15:17 pts/3 00:00:00 grep --color=auto nrpe
root@ip-172-31-28-18:~#


3) find / -name nrpe.cfg

root@ip-172-31-28-18:~# find / -name nrpe.cfg
/var/tmp/nagios/nrpe-2.14/sample-config/nrpe.cfg
/home/varentest/upload/nrpe.cfg
/usr/local/nagios/etc/nrpe.cfg
/tmp/linux-nrpe-agent/subcomponents/nrpe/nrpe-2.15/sample-config/nrpe.cfg
/tmp/linux-nrpe-agent/subcomponents/nrpe/mods/cfg/nrpe.cfg
/tmp/nrpe-3.0.1/sample-config/nrpe.cfg
/etc/nagios/nrpe.cfg
root@ip-172-31-28-18:~#

4) find / -name check_disk

root@ip-172-31-28-18:~# find / -name check_disk
/usr/local/nagios/libexec/check_disk
/usr/lib/nagios/plugins/check_disk
/tmp/linux-nrpe-agent/subcomponents/nagiosplugins/nagios-plugins-2.0.3/plugins/check_disk
/tmp/nagios-plugins-release-2.2.0/plugins/check_disk
root@ip-172-31-28-18:~#

5) find / -name check_mem

root@ip-172-31-28-18:~# find / -name check_mem
root@ip-172-31-28-18:~#

not found

6) find / -name check_procs

root@ip-172-31-28-18:~# find / -name check_procs
/usr/local/nagios/libexec/check_procs
/usr/lib/nagios/plugins/check_procs
/tmp/linux-nrpe-agent/subcomponents/nagiosplugins/nagios-plugins-2.0.3/plugins/check_procs
/tmp/nagios-plugins-release-2.2.0/plugins/check_procs
root@ip-172-31-28-18:~#


Thanks

[tgriep]

Thanks for posting the output. From this command

Code: Select all

find / -name nrpe.cfg

It looks like there could be 3 to 4 different versions of the NRPE agent installed on that system and they have different settings and paths so I will need to get more information.

Can you post the following file from that remote server so we can view the settings?

Code: Select all

/etc/nagios/nrpe.cfg

Then run this command as root and post the output so we know which version of the NRPE Agent is running on the server.

Code: Select all

/usr/sbin/nrpe

[ivp2015]

Thanks for the support please find out the output of given info


root@ip-172-31-28-18:~# cat /etc/nagios/nrpe.cfg

Code: Select all

#############################################################################
# Sample NRPE Config File
# Written by: Ethan Galstad ([email protected])
#
# Last Modified: 11-23-2007
#
# NOTES:
# This is a sample configuration file for the NRPE daemon.  It needs to be
# located on the remote host that is running the NRPE daemon, not the host
# from which the check_nrpe client is being executed.
#############################################################################


# LOG FACILITY
# The syslog facility that should be used for logging purposes.

log_facility=daemon



# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID
# number.  The file is only written if the NRPE daemon is started by the root
# user and is running in standalone mode.

pid_file=/var/run/nagios/nrpe.pid



# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-priviledged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

server_port=5666



# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface
# and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

#server_address=127.0.0.1



# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_user=nagios



# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_group=nagios



# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon. Network addresses with a bit mask
# (i.e. 192.168.1.0/24) are also supported. Hostname wildcards are not currently
# supported.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address.  I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

#allowed_hosts=127.0.0.1, 182.74.168.35, 115.114.12.4
allowed_hosts=127.0.0.1, 115.114.12.4,182.74.168.35


# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed.  This option only works
# if the daemon was configured with the --enable-command-args configure script
# option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=1



# BASH COMMAND SUBTITUTION
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments that contain bash command substitutions of the form
# $(...).  This option only works if the daemon was configured with both
# the --enable-command-args and --enable-bash-command-substitution configure
# script options.
#
# *** ENABLING THIS OPTION IS A HIGH SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow bash command substitutions,
#         1=allow bash command substitutions

allow_bash_command_substitution=0



# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the
# command line from the command definition.
#
# *** THIS EXAMPLE MAY POSE A POTENTIAL SECURITY RISK, SO USE WITH CAUTION! ***
# Usage scenario:
# Execute restricted commmands using sudo.  For this to work, you need to add
# the nagios user to your /etc/sudoers.  An example entry for alllowing
# execution of the plugins from might be:
#
# nagios          ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
#
# This lets the nagios user run all commands in that directory (and only them)
# without asking for a password.  If you do this, make sure you don't give
# random users write access to that directory or its contents!

# command_prefix=/usr/bin/sudo



# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the
# syslog facility.
# Values: 0=debugging off, 1=debugging on

debug=0



# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# allow plugins to finish executing before killing them off.

command_timeout=60



# CONNECTION TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# wait for a connection to be established before exiting. This is sometimes
# seen where a network problem stops the SSL being established even though
# all network sessions are connected. This causes the nrpe daemons to
# accumulate, eating system resources. Do not set this too low.

connection_timeout=300



# WEEK RANDOM SEED OPTION
# This directive allows you to use SSL even if your system does not have
# a /dev/random or /dev/urandom (on purpose or because the necessary patches
# were not applied). The random number generator will be seeded from a file
# which is either a file pointed to by the environment valiable $RANDFILE
# or $HOME/.rnd. If neither exists, the pseudo random number generator will
# be initialized and a warning will be issued.
# Values: 0=only seed from /dev/[u]random, 1=also seed from weak randomness

#allow_weak_random_seed=1



# INCLUDE CONFIG FILE
# This directive allows you to include definitions from an external config file.

#include=<somefile.cfg>



# INCLUDE CONFIG DIRECTORY
# This directive allows you to include definitions from config files (with a
# .cfg extension) in one or more directories (with recursion).

#include_dir=<somedirectory>
#include_dir=<someotherdirectory>



# COMMAND DEFINITIONS
# Command definitions that this daemon will run.  Definitions
# are in the following format:
#
# command[<command_name>]=<command_line>
#
# When the daemon receives a request to return the results of <command_name>
# it will execute the command specified by the <command_line> argument.
#
# Unlike Nagios, the command line cannot contain macros - it must be
# typed exactly as it should be executed.
#
# Note: Any plugins that are used in the command lines must reside
# on the machine that this daemon is running on!  The examples below
# assume that you have plugins installed in a /usr/local/nagios/libexec
# directory.  Also note that you will have to modify the definitions below
# to match the argument format the plugins expect.  Remember, these are
# examples only!


# The following examples use hardcoded command arguments...

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200
command[keystone]=/usr/lib64/nagios/plugins/check_procs -c 1: -w 3: -C keystone-all

# The following examples allow user-supplied arguments and can
# only be used if the NRPE daemon was compiled with support for
# command arguments *AND* the dont_blame_nrpe directive in this
# config file is set to '1'.  This poses a potential security risk, so
# make sure you read the SECURITY file before doing this.

#command[check_users]=/usr/lib/nagios/plugins/check_users -w $ARG1$ -c $ARG2$
#command[check_load]=/usr/lib/nagios/plugins/check_load -w $ARG1$ -c $ARG2$
#command[check_disk]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
#command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$

#
# local configuration:
#       if you'd prefer, you can instead place directives here
include=/etc/nagios/nrpe_local.cfg

#
# you can place your config snipplets into nrpe.d/
# only snipplets ending in .cfg will get included
include_dir=/etc/nagios/nrpe.d/


command[check_disk]=/usr/lib/nagios/plugins/check_disk $ARG1$ $ARG2$ $ARG3$

root@ip-172-31-28-18:~#


2)

root@ip-172-31-28-18:~# sudo /usr/sbin/nrpe

Code: Select all

NRPE - Nagios Remote Plugin Executor
Copyright (c) 1999-2008 Ethan Galstad ([email protected])
Version: 2.15
Last Modified: 09-06-2013
License: GPL v2 with exemptions (-l for more info)
SSL/TLS Available: Anonymous DH Mode, OpenSSL 0.9.6 or higher required
TCP Wrappers Available

***************************************************************
** POSSIBLE SECURITY RISK - COMMAND ARGUMENTS ARE SUPPORTED! **
**      Read the NRPE SECURITY file for more information     **
***************************************************************

Usage: nrpe [-n] -c <config_file> [-4|-6] <mode>

Options:
 -n            = Do not use SSL
 <config_file> = Name of config file to use
 -4            = use ipv4 only
 -6            = use ipv6 only
 <mode>        = One of the following operating modes:
   -i          =    Run as a service under inetd or xinetd
   -d          =    Run as a standalone daemon
   -d -s       =    Run as a subsystem under AIX

Notes:
This program is designed to process requests from the check_nrpe
plugin on the host(s) running Nagios.  It can run as a service
under inetd or xinetd (read the docs for info on this), or as a
standalone daemon. Once a request is received from an authorized
host, NRPE will execute the command/plugin (as defined in the
config file) and return the plugin output and return code to the
check_nrpe plugin.

root@ip-172-31-28-18:~#

[tgriep]

The check_mem plugin is not installed on your system so you will have to create it.

The custom_check_mem script that is attached to this post needs to be uploaded to this folder.

Code: Select all

/usr/lib/nagios/plugins/

You will have to rename it by running the following as root.

Code: Select all

mv /usr/lib/nagios/plugins/custom_check_mem.txt /usr/lib/nagios/plugins/custom_check_mem

Then the permissions need to be set on the plugin by running

Code: Select all

chmod 777 /usr/lib/nagios/plugins/custom_check_mem

Then add the following commands to the nrpe.cfg file to add the missing command definition.

Code: Select all

command[check_mem]=/usr/lib/nagios/plugins/custom_check_mem -n $ARG1$
command[check_procs]=/usr/lib/nagios/plugins/check_procs $ARG1$

Save the file and restart the nrpe daemon so it will reload the changes. This has to be done when any changes to the NRPE configs files are done.
Run this as root on the server

Code: Select all

service nrpe restart

Try that and see if it fixes the issue.

[ivp2015]

Hey,

I have tried this setup its still not worked,

please find out attachment

[tgriep]

The Critical for the Total Processes check just needs to have the Threshold settings increased to fix that.

The disk service check is misconfigured as it is calling the custom_check_mem command and is should be using the check_disk command so if you edit that service check and change it, that should fix that error.

If still have any errors after doing this change, you will need to post the nrpe.cfg file from the remote system and how the check is defined on the Nagios system so we can check the settings.

[ivp2015]

Hi,

Its still not working please find out the attachment nrpe.cfg config file,k

Thanks

[tgriep]

The configuration error is on the Nagios server, go to the Core Config Manager > Services menu and find the service called disk for that host
Change the $ARG1$ from custom_check_mem to check_disk and save it out and Apply the Config.

If that still doesn't work, could you post or PM me your Nagios XI System Profile so we can review it?
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and either post it of PM it to me.

Can you post this file from the remote server so we can view it?

Code: Select all

/etc/nagios/nrpe_local.cfg

[tgriep]

I took a look at the profile and it looks like the command was edited, are you still having an error and could you post it so we can view it?

[ivp2015]

Hi,
You want me to upload the profile again

Thanks