Page 3 of 4

Re: check_procs not working with nrpe

Posted: Fri Aug 15, 2014 11:31 am
by slansing
The process checks you added from the wizard are trying to use check_services? Not check_procs? Do you have a check_services defined in the remote host's nrpe.cfg? If so, please share the command line. It sounds like you are confusing services with processes, previously you mentioned these were processes, you are now saying they are services, what are they actually?

Re: check_procs not working with nrpe

Posted: Fri Aug 15, 2014 2:25 pm
by klee
Hi Slasing,

Unfortunately, I am not in the office today (since I don't have access to these systems right now, I'm not 100% sure of my comments below)

Apparently, these 5 items (bioserver, workserver, server cmr, server mgr, server timer) we're trying to monitor are services (since check_services is able to see them).

We rather check for processes as opposed to just services, since a service can still be running even after its processes die (and that wouldn't help us very much).

I ran the full install for NRPE package that came with Linux Monitoring Wizard.

After which, it seemed to have replaced the /usr/local/nagios/etc/nrpe.cfg file on the remote host.

The previous check_procs definition Abrist suggested was overwritten and the entire file reverted back to default (preceded by comments).

Therefore, I am not sure where check_services is being called on the remote host.

When I checked on the monitoring host's service definition (via Core Configuration Manager), NRPE is calling check_services. Is this behavior incorrect?

Perhaps I entered them into the wrong field when configuring with the Linux Monitoring Wizard?

Again,I will send you the specifics as soon as I'm back in the office.

Thanks for your help and have a great weekend.

-Klee

Re: check_procs not working with nrpe

Posted: Fri Aug 15, 2014 3:10 pm
by sreinhardt
Let's verify your statements are correct when you have a chance before we take action on this further. Just don't want to send you off in the wrong direction unless we know it is an avenue we have to take.

Re: check_procs not working with nrpe

Posted: Mon Aug 18, 2014 1:45 am
by klee
I've confirmed the following:

1) These 5 items (bioserver, workserver, server cmr, server mgr, server timer) are services; they are listening to connections for the AVID iNews application. So idealistically, we'd like to monitor these services and their processes.

2) I double-checked the /usr/local/nagios/etc/nrpe.cfg file and verified that there's no reference to check_services. I was unable to locate alternate nrpe.cfg files anywhere else in the file-system.

3) The Linux Server Monitoring Wizard has options for monitoring services and processes. So I tested again to verify, except this time, I configured it to check for services as well as processes. This is what I found out:

- Apparently service checks are defined as "check_init_service"

- Whereas process checks are defined as "check_services"

4) Below are the definitions for check_init_service & check_services as defined by the Linux Server Monitoring Wizard.

Code: Select all

Service Management
Check Command: check_nrpe
Command view: $USER1$/check_nrpe -H $HOSTADDRESS$ -t 30 -c $ARG1$ $ARG2$
$ARG1$: check_init_service
$ARG2$: -a 'bioserver'

Service Management
Check Command: check_nrpe
Command view: $USER1$/check_nrpe -H $HOSTADDRESS$ -t 30 -c $ARG1$ $ARG2$
$ARG1$: check_services
$ARG2$: -a 'bioserver'
Questions:

5) check_init_service --help: "this plugin checks the status of services normally started by the init process"
check_init_service is currently failing for all 5 services we're trying to monitor: "NRPE: Unable to read output"
These 5 services (bioserver, workserver, server cmr, server mgr, server timer) are not started by init process. Is there a way to monitor such services?

6) Is it possible that the Linux Server Monitoring Wizard does not utilize nrpe.cfg on the remote server?

7) Can you please verify for sure if check_services is actually checking for services or processes?

8) Is there a way to get check_services to recognize 2 word strings (i.e. 'server timer')?

Thanks Much,

-klee

Re: check_procs not working with nrpe

Posted: Mon Aug 18, 2014 4:37 pm
by tmcdonald
When you run the wizard, it creates a download package that you use to install NRPE on the remote machine. It includes all the configs and plugins you need. It should definitely include check_services and check_init_services now. Can you post the nrpe.cfg file from the remote server for confirmation?

Re: check_procs not working with nrpe

Posted: Tue Aug 19, 2014 9:18 am
by klee
Hi tmcdonald,

Yes, when I ran the Linux Monitoring Wizard on the monitoring host, it provided me with the linux-nrpe-agent.tar.gz.

I uploaded this linux-nrpe-agent file to the remote host and ran the fullinstall script as per the instructions.

Below is the /usr/local/nagios/etc/nrpe.cfg file from the remote server.

Thanks,

-klee

Code: Select all

#############################################################################
# Sample NRPE Config File
# Written by: Ethan Galstad ([email protected])
#
# Last Modified: 11-23-2007
#
# NOTES:
# This is a sample configuration file for the NRPE daemon.  It needs to be
# located on the remote host that is running the NRPE daemon, not the host
# from which the check_nrpe client is being executed.
#############################################################################


# LOG FACILITY
# The syslog facility that should be used for logging purposes.

log_facility=daemon



# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID
# number.  The file is only written if the NRPE daemon is started by the root
# user and is running in standalone mode.

pid_file=/var/run/nrpe.pid



# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-priviledged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

server_port=5666



# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface
# and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

#server_address=127.0.0.1



# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_user=nagios



# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_group=nagios



# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address.  I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1



# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed.  This option only works
# if the daemon was configured with the --enable-command-args configure script
# option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=1



# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the
# command line from the command definition.
#
# *** THIS EXAMPLE MAY POSE A POTENTIAL SECURITY RISK, SO USE WITH CAUTION! ***
# Usage scenario:
# Execute restricted commmands using sudo.  For this to work, you need to add
# the nagios user to your /etc/sudoers.  An example entry for alllowing
# execution of the plugins from might be:
#
# nagios          ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
#
# This lets the nagios user run all commands in that directory (and only them)
# without asking for a password.  If you do this, make sure you don't give
# random users write access to that directory or its contents!

# command_prefix=/usr/bin/sudo



# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the
# syslog facility.
# Values: 0=debugging off, 1=debugging on

debug=0



# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# allow plugins to finish executing before killing them off.

command_timeout=60



# CONNECTION TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# wait for a connection to be established before exiting. This is sometimes
# seen where a network problem stops the SSL being established even though
# all network sessions are connected. This causes the nrpe daemons to
# accumulate, eating system resources. Do not set this too low.

connection_timeout=300



# WEEK RANDOM SEED OPTION
# This directive allows you to use SSL even if your system does not have
# a /dev/random or /dev/urandom (on purpose or because the necessary patches
# were not applied). The random number generator will be seeded from a file
# which is either a file pointed to by the environment valiable $RANDFILE
# or $HOME/.rnd. If neither exists, the pseudo random number generator will
# be initialized and a warning will be issued.
# Values: 0=only seed from /dev/[u]random, 1=also seed from weak randomness

#allow_weak_random_seed=1



# INCLUDE CONFIG FILE
# This directive allows you to include definitions from an external config file.

#include=<somefile.cfg>



# INCLUDE CONFIG DIRECTORY
# This directive allows you to include definitions from config files (with a
# .cfg extension) in one or more directories (with recursion).

#include_dir=<somedirectory>
include_dir=/usr/local/nagios/etc/nrpe



# COMMAND DEFINITIONS
# Command definitions that this daemon will run.  Definitions
# are in the following format:
#
# command[<command_name>]=<command_line>
#
# When the daemon receives a request to return the results of <command_name>
# it will execute the command specified by the <command_line> argument.
#
# Unlike Nagios, the command line cannot contain macros - it must be
# typed exactly as it should be executed.
#
# Note: Any plugins that are used in the command lines must reside
# on the machine that this daemon is running on!  The examples below
# assume that you have plugins installed in a /usr/local/nagios/libexec
# directory.  Also note that you will have to modify the definitions below
# to match the argument format the plugins expect.  Remember, these are
# examples only!


# The following examples use hardcoded command arguments...

#command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
#command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
#command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1
#command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
#command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200


# The following examples allow user-supplied arguments and can
# only be used if the NRPE daemon was compiled with support for
# command arguments *AND* the dont_blame_nrpe directive in this
# config file is set to '1'.  This poses a potential security risk, so
# make sure you read the SECURITY file before doing this.

#command[check_users]=/usr/local/nagios/libexec/check_users -w $ARG1$ -c $ARG2$
#command[check_load]=/usr/local/nagios/libexec/check_load -w $ARG1$ -c $ARG2$
#command[check_disk]=/usr/local/nagios/libexec/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
#command[check_procs]=/usr/local/nagios/libexec/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$

Re: check_procs not working with nrpe

Posted: Tue Aug 19, 2014 4:02 pm
by tmcdonald
Let's also look at the /usr/local/nagios/etc/nrpe directory permissions:

Code: Select all

ls -ld /usr/local/nagios/etc/nrpe
ls -l /usr/local/nagios/etc/nrpe
and please attach the files within that directory.

Re: check_procs not working with nrpe

Posted: Tue Aug 19, 2014 6:03 pm
by klee

Code: Select all

ls -ld /usr/local/nagios/etc/nrpe
drwxr-xr-x. 2 nagios nagios 4096 Aug 14 18:00 /usr/local/nagios/etc/nrpe

ls -l /usr/local/nagios/etc/nrpe
total 8
-rw-r--r--. 1 nagios nagios  687 Aug 14 18:00 asterisk.cfg
-rw-r--r--. 1 nagios nagios 1250 Aug 14 18:00 common.cfg

Code: Select all

### ASTERISK ###

command[check_asterisk]=/usr/local/nagios/libexec/check_asterisk.pl $ARG1$
command[check_sip]=/usr/local/nagios/libexec/check_sip $ARG1$
command[check_asterisk_sip_peers]=sudo /usr/local/nagios/libexec/check_asterisk_sip_peers.sh $ARG1$

# Nagisk (Asterisk) checks
command[check_asterisk_version]=/usr/local/nagios/libexec/nagisk.pl -c version
command[check_asterisk_peers]=/usr/local/nagios/libexec/nagisk.pl -c peers
command[check_asterisk_channels]=/usr/local/nagios/libexec/nagisk.pl -c channels
command[check_asterisk_zaptel]=/usr/local/nagios/libexec/nagisk.pl -c zaptel

Code: Select all

### GENERIC SERVICES ###
command[check_init_service]=sudo /usr/local/nagios/libexec/check_init_service $ARG1$
command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$

### MISC SYSTEM METRICS ###
#command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_users]=/usr/local/nagios/libexec/check_users $ARG1$
command[check_load]=/usr/local/nagios/libexec/check_load $ARG1$
command[check_swap]=/usr/local/nagios/libexec/check_swap $ARG1$
command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats.sh $ARG1$
command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$

### YUM UPDATES ###
command[check_yum]=/usr/local/nagios/libexec/check_yum

### DISK ###
command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
command[check_ide_smart]=/usr/local/nagios/libexec/check_ide_smart $ARG1$

### PROCESSES ###
command[check_all_procs]=/usr/local/nagios/libexec/custom_check_procs
command[check_procs]=/usr/local/nagios/libexec/check_procs $ARG1$

### OPEN FILES ###
command[check_open_files]=/usr/local/nagios/libexec/check_open_files.pl $ARG1$

### NETWORK CONNECTIONS ###
command[check_netstat]=/usr/local/nagios/libexec/check_netstat.pl -p $ARG1$ $ARG2$

Re: check_procs not working with nrpe

Posted: Wed Aug 20, 2014 4:38 pm
by slansing
Would you be okay with moving this to a support ticket? I'll grab it when it comes in and set up a remote with you to figure out what is going on, this is the second time I've seen this recently and I'd like to take a look at your NRPE install to see if it is consistent with the other user's while helping you get this sorted. If you'd like to move it, send an email in to [email protected] with the title of this thread and a link to it please, thanks!

Re: check_procs not working with nrpe

Posted: Wed Aug 20, 2014 5:02 pm
by klee
Done, thank you Slansing.

I look forward to hearing from you soon.

Regards

-klee