Strange NRPE behaviour.

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
as300182
Posts: 36
Joined: Tue May 17, 2016 8:09 am

Strange NRPE behaviour.

Post by as300182 »

I have a server running the nrpe daemon version: 2.15. And when I call this remotely from my Nagios core server all plugins respond except one. That is the check_apachestatus plugin.

If I run the plugin locally from the command line it works just fine and returns what I am expecting. If I attempt to run it from nrpe I get an error. Thinking this might be something strange between the Nagios server and the agent I ran the plugin locally via nrpe and got the same error. How is it that this plugin works fine when called directly from the command line, but fails when run via nrpe? After much messing about I discovered that if I re-enable tlsv1 on the server it works. What is nrpe attempting to do differently inside the nrpe shell, and why?

Anyone got any thoughts on this?
dwasswa

Re: Strange NRPE behaviour.

Post by dwasswa »

Hi @as300182,

This is just a security related scenario. NRPE is compiled with SSL(Secure Socket Layer)/TLS(Transport layer Security) as a security measure.The plugin

Code: Select all

 check_apachestatus
parses the status page of an apache server so security protocols have to be implemented as you connect to apache.

SSL (Secure Socket Layer) is a cryptographic protocol that is used to provide security for the communications taking place over the internet. SSL uses asymmetric cryptography to preserve privacy and message authentication codes for ensuring the reliability for all the network connections above the transport layer.


What is the difference between SSL and HTTPS(apache)?

Main difference between SSL and HTTPS is that SSL is a cryptographic protocol, while HTTPS is protocol created combining HTTP and SSL. But, sometimes, HTTPS is not identified as a protocol per se, but a mechanism that merely uses HTTP over encrypted SSL connections. In other words, HTTPS uses SSL to create a secure HTTP connection.
as300182
Posts: 36
Joined: Tue May 17, 2016 8:09 am

Re: Strange NRPE behaviour.

Post by as300182 »

As interesting as that is, it doesn't help me any.

I still don't understand why the plugin works directly from the command line but not when called from the same server via check_nrpe.

From the command line

Code: Select all

/usr/local/nagios/libexec/check_apachestatus.pl -H test-server1.com -p 443
works a treat and I get a response. The same is defined in the nrpe.cfg file, but when I run

Code: Select all

/etc/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_apache
I get

Code: Select all

CRITICAL Unable to reach status page (no HTTP response code) Host: https://test-server1.com
I've upgraded nrpe to the latest version 3.2.1 which is built with the '--enable-ssl' switch (using openssl 1.0.2h), but that's made no difference.

I still fail to understand why the nrpe shell is getting a different result to when you run the check_apachestatus from the command line.
npolovenko
Support Tech
Posts: 3457
Joined: Mon May 15, 2017 5:00 pm

Re: Strange NRPE behaviour.

Post by npolovenko »

Hi, @as300182 . Could you upload you nrpe.cfg file here? Also, where did you download this plugin? There's a slight chance that it could be a permission issue. Go to the /usr/local/nagios/libexec and run ls-l , and make sure that check_apachestatus.pl has the same permissions as other plugins, make sure that it runs under the same user as well.
Now, you mentioned that if you enable tlsv1 than the check starts working. Did you enable it on the Nagios Server or remote NRPE server? Can you also run

Code: Select all

nmap YourIP -p 443
from the remote NRPE server and let us know the result?
Thanks
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
as300182
Posts: 36
Joined: Tue May 17, 2016 8:09 am

Re: Strange NRPE behaviour.

Post by as300182 »

Code: Select all

#############################################################################
# Sample NRPE Config File
# Written by: Ethan Galstad (nagios@nagios.org)
#
# Last Modified: 11-23-2007
#
# NOTES:
# This is a sample configuration file for the NRPE daemon.  It needs to be
# located on the remote host that is running the NRPE daemon, not the host
# from which the check_nrpe client is being executed.
#############################################################################

# LOG FACILITY
# The syslog facility that should be used for logging purposes.

log_facility=daemon

# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID
# number.  The file is only written if the NRPE daemon is started by the root
# user and is running in standalone mode.

pid_file=/var/run/nrpe/nrpe.pid

# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-priviledged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

server_port=5666

# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface
# and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

#server_address=127.0.0.1

# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_user=nrpe

# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_group=nrpe

# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon. Network addresses with a bit mask
# (i.e. 192.168.1.0/24) are also supported. Hostname wildcards are not currently
# supported.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address.  I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1,10.1.109.118,10.1.109.231

# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed.  This option only works
# if the daemon was configured with the --enable-command-args configure script
# option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=0

# BASH COMMAND SUBTITUTION
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments that contain bash command substitutions of the form
# $(...).  This option only works if the daemon was configured with both
# the --enable-command-args and --enable-bash-command-substitution configure
# script options.
#
# *** ENABLING THIS OPTION IS A HIGH SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow bash command substitutions,
#         1=allow bash command substitutions

allow_bash_command_substitution=0

# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the
# command line from the command definition.
#
# *** THIS EXAMPLE MAY POSE A POTENTIAL SECURITY RISK, SO USE WITH CAUTION! ***
# Usage scenario:
# Execute restricted commmands using sudo.  For this to work, you need to add
# the nagios user to your /etc/sudoers.  An example entry for alllowing
# execution of the plugins from might be:
#
# nagios          ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
#
# This lets the nagios user run all commands in that directory (and only them)
# without asking for a password.  If you do this, make sure you don't give
# random users write access to that directory or its contents!

# command_prefix=/usr/bin/sudo

# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the
# syslog facility.
# Values: 0=debugging off, 1=debugging on

debug=1

# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# allow plugins to finish executing before killing them off.

command_timeout=60

# CONNECTION TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# wait for a connection to be established before exiting. This is sometimes
# seen where a network problem stops the SSL being established even though
# all network sessions are connected. This causes the nrpe daemons to
# accumulate, eating system resources. Do not set this too low.

connection_timeout=300

# WEEK RANDOM SEED OPTION
# This directive allows you to use SSL even if your system does not have
# a /dev/random or /dev/urandom (on purpose or because the necessary patches
# were not applied). The random number generator will be seeded from a file
# which is either a file pointed to by the environment valiable $RANDFILE
# or $HOME/.rnd. If neither exists, the pseudo random number generator will
# be initialized and a warning will be issued.
# Values: 0=only seed from /dev/[u]random, 1=also seed from weak randomness

#allow_weak_random_seed=1

# INCLUDE CONFIG FILE
# This directive allows you to include definitions from an external config file.

#include=<somefile.cfg>

# COMMAND DEFINITIONS
# Command definitions that this daemon will run.  Definitions
# are in the following format:
#
# command[<command_name>]=<command_line>
#
# When the daemon receives a request to return the results of <command_name>
# it will execute the command specified by the <command_line> argument.
#
# Unlike Nagios, the command line cannot contain macros - it must be
# typed exactly as it should be executed.
#
# Note: Any plugins that are used in the command lines must reside
# on the machine that this daemon is running on!  The examples below
# assume that you have plugins installed in a /usr/local/nagios/libexec
# directory.  Also note that you will have to modify the definitions below
# to match the argument format the plugins expect.  Remember, these are
# examples only!

# The following examples use hardcoded command arguments...

# command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
# command[check_load]=/usr/local/nagios/libexec/check_load -r -w 0.9,0.8,0.7 -c 1.0,0.9,0.8
# command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1
# command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
# command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200

command[check_apache]=/usr/local/nagios/libexec/check_apachestatus.pl -H  test-server1.com -p 443

# The following examples allow user-supplied arguments and can
# only be used if the NRPE daemon was compiled with support for
# command arguments *AND* the dont_blame_nrpe directive in this
# config file is set to '1'.  This poses a potential security risk, so
# make sure you read the SECURITY file before doing this.

#command[check_users]=/usr/local/nagios/libexec/check_users -w $ARG1$ -c $ARG2$
#command[check_load]=/usr/local/nagios/libexec/check_load -w $ARG1$ -c $ARG2$
#command[check_disk]=/usr/local/nagios/libexec/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
#command[check_procs]=/usr/local/nagios/libexec/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$

# INCLUDE CONFIG DIRECTORY
# This directive allows you to include definitions from config files (with a
# .cfg extension) in one or more directories (with recursion).

include_dir=/etc/nrpe.d/
The NRPE plugin was downloaded from here: https://github.com/NagiosEnterprises/nr ... 0.0.tar.gz

The check_apachestatus.pl has the same permissions as other plugins, and it runs under the same user as well. As I said before, it runs just fine from the command line, it's only a problem when trying to run inside the NRPE shell.

tlsv1 is disabled on the NRPE server only. I don't understand why this is an issue for NRPE unless it's trying to use that protocol to run check_apachestatus.pl for some reason?!

nmap does not appear to be installed so I am unable to run that.
npolovenko
Support Tech
Posts: 3457
Joined: Mon May 15, 2017 5:00 pm

Re: Strange NRPE behaviour.

Post by npolovenko »

@as300182, Please upload your check_apachestatus.pl plugin here as well. Also, we need to see whether your NRPE runs as a daemon or under xinet.d. Please run netstat -anp|grep 5666 and ps -ef|grep nrpe, and post the output here.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
as300182
Posts: 36
Joined: Tue May 17, 2016 8:09 am

Re: Strange NRPE behaviour.

Post by as300182 »

npolovenko wrote:@as300182, Please upload your check_apachestatus.pl plugin here as well. Also, we need to see whether your NRPE runs as a daemon or under xinet.d. Please run netstat -anp|grep 5666 and ps -ef|grep nrpe, and post the output here.
I've attached the plugin. I believe NRPE is running as a daemon.

This is the result from ps -ef|grep nrpe

Code: Select all

nagios   15507     1  0 Jun14 ?        00:05:09 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
and this is the result from netstat -anp|grep 5666

Code: Select all

tcp        0      0 0.0.0.0:5666                0.0.0.0:*                   LISTEN      15507/nrpe
tcp        0      0 10.1.104.170:5666           10.1.109.231:47737          TIME_WAIT   -
tcp        0      0 10.1.104.170:5666           10.1.109.118:44120          TIME_WAIT   -
Attachments
check_apachestatus.pl
(8.38 KiB) Downloaded 553 times
bolson

Re: Strange NRPE behaviour.

Post by bolson »

From the CLI of the host you are monitoring, run these commands and post the output:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1
/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1 -n
/usr/local/nagios/libexec/check_nrpe -H localhost
/usr/local/nagios/libexec/check_nrpe -H localhost -n
And from your Nagios server

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H remote_ip
/usr/local/nagios/libexec/check_nrpe -H remote_ip -n
Replace remote_ip with the IP address of the host you are monitoring.
as300182
Posts: 36
Joined: Tue May 17, 2016 8:09 am

Re: Strange NRPE behaviour.

Post by as300182 »

Code: Select all

[root@qa-connect-gold ~]# /usr/local/nagios/libexec/check_nrpe -H 127.0.0.1
NRPE v3.2.1
[root@qa-connect-gold ~]# /usr/local/nagios/libexec/check_nrpe -H 127.0.0.1 -n
CHECK_NRPE: Receive header underflow - only -1 bytes received (4 expected).
[root@qa-connect-gold ~]# /usr/local/nagios/libexec/check_nrpe -H localhost
NRPE v3.2.1
[root@qa-connect-gold ~]# /usr/local/nagios/libexec/check_nrpe -H localhost -n
CHECK_NRPE: Receive header underflow - only -1 bytes received (4 expected).

Code: Select all

[root@wds-monitoring ~]# /usr/local/nagios/libexec/check_nrpe -H 10.1.1.75
NRPE v3.2.1
[root@wds-monitoring ~]# /usr/local/nagios/libexec/check_nrpe -H 10.1.1.75 -n
CHECK_NRPE: Error receiving data from daemon.
I hope this makes some sort of sense?

Please note that I upgraded the NRPE installation in the hope that a later version may produce the desired results, but it didn't make a difference.
bolson

Re: Strange NRPE behaviour.

Post by bolson »

The script is actually working... The error you're receiving is being returned by Apache, not the script. As the error says, the script is able to connect to the web server but unable to open the /server-status page. The fact that the script returns valid data when run from the command line but not when run through nrpe points to the command definition in nrpe.cfg. However, the definition looks fine to me. One thing you might try while I continue to research this... In the command definition in nrpe.cfg, change this:

Code: Select all

command[check_apache]=/usr/local/nagios/libexec/check_apachestatus.pl -H  test-server1.com -p 443
to this

Code: Select all

command[check_apache]=/usr/local/nagios/libexec/check_apachestatus.pl -H  127.0.0.1 -p 443
Also, immediately after running the command via nrpe, run this on the Apache host and post the result:

Code: Select all

tail -50 /var/log/httpd/error_log
Locked