nrpe throwing not allowed to talk to us error

[embegee]

I have a strange issue with nrpe.

nrpe is running in daemon mode and not under inetd.

I have a Nagios 3.2.1 server monitoring an environment of predominantly RHEL 5 and 6 servers. The clients and server work perfectly however at 23:55 nightly .
NB nrpe is running in daemon mode and not under inetd

I am seeing a message file entry across all clients similar:

nrpe[68471]: Host 192.168.33.33 is not allowed to talk to us

The address specified in the error is the address of the client itself NOT the Nagios server.

All nrpe clients are configured to allow connections from a single Nagios server only. I do not understand why the client would try to talk to itself on 5666 ?

nrpe is version 2.14 client and server side.

The exact same messages occurs in /var/log/messages at the same time every night at 23:55 across 20 + servers

If anyone has seen this kind of behavior elsewhere I would be interested to hear.

Thanks in advance !

[abrist]

Have you specified your nagios server ip in the remote host's nrpe.cfg in the "allowed_hosts" directive?

[embegee]

Yes, the address is correct in allowed hosts

[embegee]

Code: Select all

11:27:58 # more nrpe.cfg
#############################################################################
# Sample NRPE Config File
# Written by: Ethan Galstad ([email protected])
#
# Last Modified: 11-23-2007
#
# NOTES:
# This is a sample configuration file for the NRPE daemon.  It needs to be
# located on the remote host that is running the NRPE daemon, not the host
# from which the check_nrpe client is being executed.
#############################################################################


# LOG FACILITY
# The syslog facility that should be used for logging purposes.

log_facility=daemon



# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID
# number.  The file is only written if the NRPE daemon is started by the root
# user and is running in standalone mode.

pid_file=/var/run/nrpe.pid



# PORT NUMBER
# Port number we should wait for connections on.
# NOTE: This must be a non-priviledged port (i.e. > 1024).
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

server_port=5666



# SERVER ADDRESS
# Address that nrpe should bind to in case there are more than one interface
# and you do not want nrpe to bind on all interfaces.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

server_address=192.168.33.33



# NRPE USER
# This determines the effective user that the NRPE daemon should run as.
# You can either supply a username or a UID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_user=nagios



# NRPE GROUP
# This determines the effective group that the NRPE daemon should run as.
# You can either supply a group name or a GID.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

nrpe_group=nagios

# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon. Network addresses with a bit mask
# (i.e. 192.168.1.0/24) are also supported. Hostname wildcards are not currently
# supported.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address.  I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
allowed_hosts=192.168.32.4

[abrist]

Is the remote host ip: 192.168.33.33 ?
And the nagios server: 192.168.32.4 ?
Or should these be swapped in your config?

nrpe[68471]: Host 192.168.33.33 is not allowed to talk to us

The above error usually implies the nagios server is 192.168.33.33, which does not match your config.

[embegee]

Which is exactly my point.

As initially said this configuration works perfectly for the other 23hrs 55min every day.

I am thinking some scheduled task (elsewhere on our environment) is acting like a DOS attack but would that cause that kind of error from the nrpe daemon?

[abrist]

If 192.168.33.33 is the remote host, then you must have a check that is trying to run check_nrpe locally from the remote host. You could add 192.168.33.33 or 127.0.0.1 to the allowed_hosts directive, restart nrpe, and then wait until the error would normally appear and then see what is getting run by the check.

[embegee]

Thanks for coming back so quickly. But I believe I have the answer.

It seems that this error is being caused by a Tripwire Enterprise Policy that runs at 23:55 every night.

Not only is nrpe affected but also EMC navisphere and Oracle cssd. I just watched one of my colleagues interactively with a tail running on the messages file of the client the error popped straight up !

I have worked with nrpe for 10 years and not seen anything like this before.

Thanks again !

[abrist]

No problem. Thanks for hunting down this edge case. Should we lock up the thread, or do you want to post in verbose mode for future forum goers?
Cheers.

[slansing]

Aha! Sometimes it's just a culprit in the network .