Issue with check_log over nrpe for Linux remote host

[Hikerguy86]

good idea,

its running as the nrpe user!

Code: Select all

nrpe     28399     1  0 10:03 ?        00:00:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

this gave me an idea, this was installed from the EPEL repo, not from source. Maybe a little background would explain: I am rolling out nagios monitoring to the entire company, this box is just a test box the UNIX team gave me to work out an autodeployment script and configuration. There are rules here preventing compile tools being install on production, so I can't rollout source tarballs across the fleet and have each server install it. I also would prefer not to compile and then roll out the binaries because of the fragmentation across versions and the (probable) need to roll out associated libraries as well. So, the last option is the EPEL repo, there must have been a setting in the RPM for RHEL 7 that is affecting this.

less /usr/lib/systemd/system/nrpe.service:

Code: Select all

[Unit]
Description=NRPE
After=network.target
Requires=network.target

[Service]
Type=forking
User=nrpe
Group=nrpe
EnvironmentFile=/etc/sysconfig/nrpe
ExecStart=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d $NRPE_SSL_OPT

[Install]
WantedBy=multi-user.target

BOOM!

changed the service to use nagios but it is still writing to the /var/secure/log the command i'm running so it's still only detecting itself...

from nagios:

check_AuthFailures
This service is flapping between states
CRITICAL 11-10-2016 11:01:48 0d 0h 58m 18s 4/4 (1) < Nov 10 11:01:30 remotehost sudo: nagios : TTY=unknown : PWD=/ : USER=root : COMMAND=/usr/lib64/nagios/plugins/check_log -F /var/log/secure -O /home/nagios/authFailures -q authentication failure

tried running these commands on the command line again as the nagios user. Got a normal result when I ran

Code: Select all

 /usr/lib64/nagios/plugins/check_log -F /var/log/secure -O /home/nagios/authFailures -q "authentication failure"

Log check ok - 0 pattern matches found

ran with sudo and got the error:

Code: Select all

sudo -n /usr/lib64/nagios/plugins/check_log -F /var/log/secure -O /home/nagios/authFailures -q "authentication failure"
(1) < Nov 10 11:05:35 remotehost sudo:  nagios : TTY=pts/1 ; PWD=/root ; USER=root ; COMMAND=/usr/lib64/nagios/plugins/check_log -F /var/log/secure -O /home/nagios/authFailures -q authentication failure

removed "sudo -n" from the nrpe.cfg file and got the correct result in nagios:

Check_AuthFailures
This service is flapping between states
OK 11-10-2016 11:06:49 0d 0h 2m 12s 1/4 Log check ok - 0 pattern matches found


Opened SSH terminal to the remote host and tried to log into root with a bad password

Check_AuthFailures
This service is flapping between states
CRITICAL 11-10-2016 11:12:25 0d 0h 0m 30s 1/4 (2) < Nov 10 11:11:43 remotehost sshd[32684]: PAM 2 more authentication failures: logname= uid=0 euid=0 tty=ssh ruser= rhost=my_desktop.domain.ad user=root


HUZZAH!

this is why I much prefer tarballs.

If you install the rpms for nrpe for RHEL 7.2 from the EPEL database, the service may use the nrpe user REGARDLESS of what is in the nrpe.cfg file for the systemd service!! (for any lurkers with this issue) I do NOT know if this is the case for other service managers or inet/xinetd

Now I need to go back and check the 4.x, 5.x, and 6.x test boxes. I'll post back with results, the other versions use init as well, 7.x is the only place we are using systemd so maybe that's the issue?

[tgriep]

If the other system are running the agent using the xinetd daemon, the username it runs under is in the /etc/xinetd.d/nrpe file, that could be the difference.
Glad it is working for you.
Yea, that is why I like the source files to install, you know what is happening and how it is configured.

[Hikerguy86]

The other servers are running with init scripts and exhibiting the same behavior, Log check error: Log file /var/log/secure is not readable!

example on a 5.11 Redhat server:

the nrpe.cfg file specified the nagios user and it is running as the nagios user:

Code: Select all

nagios   30647     1  0 16:02 ?        00:00:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

/etc/sudoers:

Code: Select all

Defaults:nagios !requiretty
nagios ALL=(ALL) NOPASSWD: ALL

this has to be simple, what am I missing?

[tgriep]

It's a permission issue. The NRPE Agent runs as the nagios user but only a root account can read the /var/log/secure and that is where the issue is coming from.

Try adding the following to the /etc/sudoers file adjusting the path for where the plugin is installed.

Code: Select all

nagios ALL=NOPASSWD: /usr/lib64/nagios/plugins/check_log

[Hikerguy86]

Apologies, missed your response

changed the sudoers file to:

Code: Select all

...
%monitoring ALL=(ALL) NOPASSWD: ALL
...
nagios ALL=(root) /usr/lib64/nagios/plugins/check_log

nagios groups

Code: Select all

groups nagios
nagios : nagios monitoring

still works on the server that uses systemd, still doesn't work with the same error on systems that use init.

Appreciate all the help by the way, I don't understand why RHEL/CentOS 5/6 are having so many issues compared to RHEL 7.2

[tgriep]

On my Centos 6 system, I had to add sudo to the command in the nrpe.cfg file like the example below

Code: Select all

command[diff_log]= sudo /usr/local/nagios/libexec/check_log -F $ARG1$ -O $ARG2$ -q $ARG3$

I have these 2 entries in the /etc/sudoers file

Code: Select all

Defaults:nagios !requiretty
nagios ALL=NOPASSWD: /usr/local/nagios/libexec/check_log

Make sure the NRPE Agent is set to run as the nagios user on the server and it should work for you.

[Hikerguy86]

that does get the information from the log, but the issue as I posted before is that it only detects is own runs because every time it runs the check, the system logs the run to /var/log/secure

Code: Select all

(1) < Nov 16 11:12:09 vmlshbobj1d sudo: nagios : TTY=unknown : PWD=/ : USER=root : COMMAND=/usr/lib64/nagios/plugins/check_log -F /var/log/secure -O /home/nagios/authFailures -q authentication failure 

added "| grep -v /usr/lib64/nagios/plugins/check_log" to the end of the command.

Hope this works right haha.

THanks for all the help

[tgriep]

Sorry about that, I thought you were still getting the permission problem.
The grep -v should work for you.
Test it out and let us know if you still need any help.