Page 1 of 2
Monitor a Centos 7 host
Posted: Mon Oct 19, 2015 1:11 pm
by JohnFLi
I am trying to monitor a new centos 7 machine.
I used the wizard in XI to setup the items to check.
most of the checks are failing, all with a similar message.
I have also tried from the cli on the Nagios server as well:
Code: Select all
./check_nrpe -H g1vtwebgitlab01 -c check_disk -a '-w 20% -c 10% -p /'
NRPE: Command 'check_disk' not defined
I have added the Nagios machine to the NRPE.cfg file. the Centos firewall has been disabled.
yes, the nagios client has been installed on the machine, and I can see the commands in the /usr/local/nagios/libexec directory.
ANy other items to check?
Re: Monitor a Centos 7 host
Posted: Mon Oct 19, 2015 1:56 pm
by tgriep
Could you post the following files from your remote system called g1vtwebgitlab01 ?
Code: Select all
/usr/local/nagios/etc/nrpe/common.cfg
/usr/local/nagios/etc/nrpe.cfg
Re: Monitor a Centos 7 host
Posted: Mon Oct 19, 2015 2:51 pm
by JohnFLi
Code: Select all
### GENERIC SERVICES ###
command[check_init_service]=sudo /usr/local/nagios/libexec/check_init_service $ARG1$
command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$
### MISC SYSTEM METRICS ###
#command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_users]=/usr/local/nagios/libexec/check_users $ARG1$
command[check_load]=/usr/local/nagios/libexec/check_load $ARG1$
command[check_swap]=/usr/local/nagios/libexec/check_swap $ARG1$
command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats.sh $ARG1$
command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$
### SYSTEM UPDATES ###
command[check_yum]=/usr/local/nagios/libexec/check_yum
command[check_apt]=/usr/local/nagios/libexec/check_apt
### DISK ###
command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
command[check_ide_smart]=/usr/local/nagios/libexec/check_ide_smart $ARG1$
### PROCESSES ###
command[check_all_procs]=/usr/local/nagios/libexec/custom_check_procs
command[check_procs]=/usr/local/nagios/libexec/check_procs $ARG1$
### OPEN FILES ###
command[check_open_files]=/usr/local/nagios/libexec/check_open_files.pl $ARG1$
### NETWORK CONNECTIONS ###
command[check_netstat]=/usr/local/nagios/libexec/check_netstat.pl -p $ARG1$ $ARG2$
Code: Select all
#############################################################################
# Sample NRPE Config File
# Written by: Ethan Galstad ([email protected])
#
# Last Modified: 11-23-2007
#
# NOTES:
# This is a sample configuration file for the NRPE daemon. It needs to be
# located on the remote host that is running the NRPE daemon, not the host
# from which the check_nrpe client is being executed.
#############################################################################
# LOG FACILITY
# The syslog facility that should be used for logging purposes.
log_facility=daemon
# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID
# number. The file is only written if the NRPE daemon is started by the root
# user and is running in standalone mode.
pid_file=/var/run/nrpe.pid
# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-priviledged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
server_port=5666
# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface
# and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
#server_address=127.0.0.1
# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
nrpe_user=nagios
# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
nrpe_group=nagios
# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address. I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
allowed_hosts=127.0.0.1,G1VPNAG03
# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed. This option only works
# if the daemon was configured with the --enable-command-args configure script
# option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments
dont_blame_nrpe=1
# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the
# command line from the command definition.
#
# *** THIS EXAMPLE MAY POSE A POTENTIAL SECURITY RISK, SO USE WITH CAUTION! ***
# Usage scenario:
# Execute restricted commmands using sudo. For this to work, you need to add
# the nagios user to your /etc/sudoers. An example entry for alllowing
# execution of the plugins from might be:
#
# nagios ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
#
# This lets the nagios user run all commands in that directory (and only them)
# without asking for a password. If you do this, make sure you don't give
# random users write access to that directory or its contents!
# command_prefix=/usr/bin/sudo
# command_prefix=/usr/bin/sudo
# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the
# syslog facility.
# Values: 0=debugging off, 1=debugging on
debug=0
# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# allow plugins to finish executing before killing them off.
command_timeout=60
# CONNECTION TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# wait for a connection to be established before exiting. This is sometimes
# seen where a network problem stops the SSL being established even though
# all network sessions are connected. This causes the nrpe daemons to
# accumulate, eating system resources. Do not set this too low.
connection_timeout=300
# WEEK RANDOM SEED OPTION
# This directive allows you to use SSL even if your system does not have
# a /dev/random or /dev/urandom (on purpose or because the necessary patches
# were not applied). The random number generator will be seeded from a file
# which is either a file pointed to by the environment valiable $RANDFILE
# or $HOME/.rnd. If neither exists, the pseudo random number generator will
# be initialized and a warning will be issued.
# Values: 0=only seed from /dev/[u]random, 1=also seed from weak randomness
#allow_weak_random_seed=1
# INCLUDE CONFIG FILE
# This directive allows you to include definitions from an external config file.
#include=<somefile.cfg>
# INCLUDE CONFIG DIRECTORY
# This directive allows you to include definitions from config files (with a
# .cfg extension) in one or more directories (with recursion).
#include_dir=<somedirectory>
include_dir=/usr/local/nagios/etc/nrpe
# COMMAND DEFINITIONS
# Command definitions that this daemon will run. Definitions
# are in the following format:
#
# command[<command_name>]=<command_line>
#
# When the daemon receives a request to return the results of <command_name>
# it will execute the command specified by the <command_line> argument.
#
# Unlike Nagios, the command line cannot contain macros - it must be
# typed exactly as it should be executed.
#
# Note: Any plugins that are used in the command lines must reside
# on the machine that this daemon is running on! The examples below
# assume that you have plugins installed in a /usr/local/nagios/libexec
# directory. Also note that you will have to modify the definitions below
# to match the argument format the plugins expect. Remember, these are
# examples only!
# The following examples use hardcoded command arguments...
#command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
#command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
#command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1
#command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
#command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200
# The following examples allow user-supplied arguments and can
# only be used if the NRPE daemon was compiled with support for
# command arguments *AND* the dont_blame_nrpe directive in this
# config file is set to '1'. This poses a potential security risk, so
# make sure you read the SECURITY file before doing this.
#command[check_users]=/usr/local/nagios/libexec/check_users -w $ARG1$ -c $ARG2$
#command[check_load]=/usr/local/nagios/libexec/check_load -w $ARG1$ -c $ARG2$
#command[check_disk]=/usr/local/nagios/libexec/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
#command[check_procs]=/usr/local/nagios/libexec/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
the contents of /usr/local/nagios/libexec
Code: Select all
[root@G1VTWEBGITLAB01 libexec]# ls
check_apt check_init_service check_sensors
check_asterisk.pl check_ircd check_services
check_asterisk_sip_peers.sh check_jabber check_simap
check_breeze check_load check_sip
check_by_ssh check_log check_smtp
check_clamd check_mailq check_snmp
check_cluster check_mrtg check_spop
check_cpu_stats.sh check_mrtgtraf check_ssh
check_dhcp check_nagios check_ssmtp
check_dig check_netstat.pl check_swap
check_disk check_nntp check_tcp
check_disk_smb check_nntps check_time
check_dns check_nrpe check_udp
check_dummy check_nt check_ups
check_file_age check_ntp check_uptime
check_flexlm check_ntp_peer check_users
check_fping check_ntp_time check_wave
check_ftp check_nwstat check_yum
check_game check_open_files.pl custom_check_mem
check_hpjd check_oracle custom_check_procs
check_http check_overcr nagisk.pl
check_icmp check_ping negate
check_ide_smart check_pop send_nsca
check_ifoperstatus check_procs urlize
check_ifstatus check_real utils.pm
check_imap check_rpc utils.sh
Code: Select all
[root@G1VTWEBGITLAB01 libexec]# ./check_nrpe -H 127.0.0.1
NRPE v2.15
[root@G1VTWEBGITLAB01 libexec]#
Re: Monitor a Centos 7 host
Posted: Mon Oct 19, 2015 3:13 pm
by jolson
Is NRPE capable of looking up your Nagios server?
What does your xinetd nrpe configuration look like?
Is SELinux messing with anything?
Code: Select all
sestatus
tail /var/log/audit/audit.log
If you have NRPE running under xinetd, try giving it a restart and see if any SELinux stuff is generated:
Code: Select all
systemctl restart xinetd
tail /var/log/audit/audit.log
Otherwise, your configurations look proper to me. The red flag here is:
NRPE: Command 'check_disk' not defined
Meaning that for one reason or another the common.cfg file could not be accessed by the nagios user.
Re: Monitor a Centos 7 host
Posted: Mon Oct 19, 2015 3:57 pm
by JohnFLi
Code: Select all
[root@G1VTWEBGITLAB01 libexec]# tail /var/log/audit/audit.log
type=CRYPTO_KEY_USER msg=audit(1445290675.125:421): pid=2146 uid=0 auid=0 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=destroy kind=session fp=? direction=from-server spid=2146 suid=0 rport=52314 laddr=10.100.5.33 lport=22 exe="/usr/sbin/sshd" hostname=? addr=10.100.34.16 terminal=? res=success'
type=USER_ACCT msg=audit(1445290801.251:422): pid=4654 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:accounting acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_ACQ msg=audit(1445290801.251:423): pid=4654 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=LOGIN msg=audit(1445290801.251:424): pid=4654 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=39 res=1
type=USER_START msg=audit(1445290801.260:425): pid=4654 uid=0 auid=0 ses=39 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_open acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_REFR msg=audit(1445290801.262:426): pid=4654 uid=0 auid=0 ses=39 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_DISP msg=audit(1445290801.271:427): pid=4654 uid=0 auid=0 ses=39 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=USER_END msg=audit(1445290801.272:428): pid=4654 uid=0 auid=0 ses=39 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_close acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=SERVICE_STOP msg=audit(1445290992.300:429): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="xinetd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_START msg=audit(1445290992.307:430): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="xinetd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Code: Select all
[root@G1VTWEBGITLAB01 libexec]# cat /etc/xinetd.d/nrpe
# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
flags = REUSE
socket_type = stream
port = 5666
wait = no
user = nagios
group = nagios
server = /usr/local/nagios/bin/nrpe
server_args = -c /usr/local/nagios/etc/nrpe.cfg --inetd
log_on_failure += USERID
disable = no
only_from = 127.0.0.1 G1VPNAG03
}
yes, ping works just fine.
I forgot about SELinux, and set it to disabled. didn't help
Re: Monitor a Centos 7 host
Posted: Mon Oct 19, 2015 8:20 pm
by Box293
What I haven't see here is a check_nrpe test from the Nagios XI server. Can you please execute the following on your Nagios XI server:
Then from the g1vtwebgitlab01 server itself can you execute this:
Code: Select all
./check_nrpe -H 127.0.0.1 -c check_disk -a '-w 20% -c 10% -p /'
And this:
Code: Select all
ps aux | grep xinetd
ps aux | grep nrpe
Re: Monitor a Centos 7 host
Posted: Tue Oct 20, 2015 10:01 am
by JohnFLi
a couple of those, I meant to post already
Code: Select all
[root@G1VPNAG03 libexec]# ./check_nrpe -H g1vtwebgitlab01
NRPE v2.15
running the 'check_disk' as you requested, gave me the not defined message.
but:
Code: Select all
[root@G1VTWEBGITLAB01 libexec]# ./check_nrpe -H 127.0.0.1 -c check_disk -a '-w 20% -c 10% -p /'
NRPE: Command 'check_disk' not defined
[root@G1VTWEBGITLAB01 libexec]# ./check_nrpe -H 127.0.0.1 -c check_load
OK - load average: 0.00, 0.01, 0.05|load1=0.000;15.000;30.000;0; load5=0.010;10.000;25.000;0; load15=0.050;5.000;20.000;0;
Code: Select all
[root@G1VTWEBGITLAB01 libexec]# ps aux | grep xinetd
root 6409 0.0 0.0 33496 1256 ? Ss Oct19 0:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid
root 29997 0.0 0.0 112640 968 pts/1 S+ 08:00 0:00 grep --color=auto xinetd
Code: Select all
[root@G1VTWEBGITLAB01 libexec]# ps aux | grep nrpe
nrpe 6310 0.0 0.0 46312 1460 ? Ss Oct19 0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
root 30008 0.0 0.0 112640 972 pts/1 S+ 08:00 0:00 grep --color=auto nrpe
Re: Monitor a Centos 7 host
Posted: Tue Oct 20, 2015 11:22 am
by tgriep
This line from your previous post
Code: Select all
nrpe 6310 0.0 0.0 46312 1460 ? Ss Oct19 0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
Looks like nrpe is running as a stand alone daemon and not under xinetd, is that how you want it to run?
If you do want it to run under xinetd, kill off that process and restart xinetd and see if that helps out.
Re: Monitor a Centos 7 host
Posted: Tue Oct 20, 2015 11:34 am
by JohnFLi
it can run under xinetd, im fine with that.....how would I make it so?
would that fix the issue?
Re: Monitor a Centos 7 host
Posted: Tue Oct 20, 2015 2:54 pm
by jolson
It *could* help. To run NRPE under xinetd, follow these instructions:
Kill the current NRPE process on your listening server.
ps -ef | grep nrpe
kill XXXX (where XXXX is the PID of the NRPE process)
Install xinetd on your server:
yum install xinetd
Add the following xinetd configuration file to /etc/xinetd.d/:
Code: Select all
# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
flags = REUSE
socket_type = stream
port = 5666
wait = no
user = nagios
group = nagios
server = /usr/local/nagios/bin/nrpe
server_args = -c /usr/local/nagios/etc/nrpe.cfg --inetd
log_on_failure += USERID
disable = no
only_from = 127.0.0.1
}
The only setting you'll need to change is 'only_from'. Set it to the IP address of your Nagios Server. After you have done so, give xinetd a restart:
Is there any change in behavior after doing so?