nrpe intermittent Error: Request contained command arguments

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
stanrate
Posts: 9
Joined: Thu Feb 15, 2018 7:49 pm

nrpe intermittent Error: Request contained command arguments

Post by stanrate »

We are running nagios-4.3.4-7.el6.x86_64 and nrpe-3.2.0-6.el6.x86_64 (under xinetd).
nrpe.cfg contains dont_blame_nrpe=0
The Nagios server frequently checks clients to make sure they are responsive. The same check is done on all clients via nrpe.
On some clients, following messages are intermittently logged:
Feb 1 04:13:25 xxxx nrpe[20334]: Error: Request contained command arguments, but argument option is not enabled!
Feb 1 04:13:25 xxxx nrpe[20334]: Client request from was invalid, bailing out...
I ran tcpdump to see if encrypted TCP packet lengths increased when the messages were logged but they didn't. This suggests Nagios server sends the same 15 packets all the time both when the messages are not logged as well when the messages are logged. When the messages are logged, after the TCP session has ended, a second TCP session starts and 21 packets are exchanged.

I don't understand why these messages are logged intermittently on some hosts.

When I started tcpdump on one of the hosts, I noticed two checks were being done every minute. I had to wait almost an hour before the two messages were logged.

Has anyone else seen this intermittent behavior and does anyone know what might be causing it?

Thanks
kyang

Re: nrpe intermittent Error: Request contained command argum

Post by kyang »

Hello @stanrate!

Are you passing arguments through the check_nrpe command? Which is passing -a?
Example:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H 192.168.x -c check_ps.sh -a '-p master'
If you have dont_blame_nrpe=0 then you are not allowing this command to be passed.

Setting it to 1 or hardcoding it into your nrpe.cfg are the two options.

Could you provide us the command you are using or any of the nrpe.cfg
stanrate
Posts: 9
Joined: Thu Feb 15, 2018 7:49 pm

Re: nrpe intermittent Error: Request contained command argum

Post by stanrate »

Here is a copy of the Nagios server host.cfg file:

define host{
use linux-server ; Name of host template to use
host_name xxxx
alias xxxxxxxxxxxxxxxxx
address xxxx
check_command check_nrpe!check_success
active_checks_enabled 1
notifications_enabled 1
}

Here is a copy of the "check_nrpe" command definition file:

define command{
command_name check_nrpe
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$


Here is a copy of the nrpe.cfg statement from the client RHEL6 host :

command[check_success]=/bin/echo "OK output to stdout by /bin/echo"

The Nagios server runs "check_nrpe" commands frequently. Most of the time, no messages are logged. Intermittently, the following messages are logged on the client host that runs nrpe via xinetd:

Feb 19 13:49:21 xxxx nrpe[7861]: Error: Request contained command arguments, but argument option is not enabled!
Feb 19 13:49:21 xxxx nrpe[7861]: Client request from was invalid, bailing out...

Thanks,
John
stanrate
Posts: 9
Joined: Thu Feb 15, 2018 7:49 pm

Re: nrpe intermittent Error: Request contained command argum

Post by stanrate »

To debug this intermittent problem, I disabled the Nagios command that ran the nrpe check and instead ran it via the command line on the Nagios server host:

Code: Select all

# while date;do /usr/lib64/nagios/plugins/check_nrpe -H xxxx -c check_success;sleep 1;done
Mon Feb 19 14:51:57 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:51:58 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:51:59 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:00 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:01 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:02 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:03 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:04 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:05 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:06 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:07 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:08 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:10 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:11 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:12 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:13 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:14 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:15 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:16 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:17 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:18 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:19 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:20 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:21 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:22 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:23 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:24 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:26 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:27 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:28 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:29 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:30 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:31 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:32 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:33 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:34 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:35 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:36 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:37 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:38 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:40 AEDT 2018
OK output to stdout by /bin/echo
Mon Feb 19 14:52:41 AEDT 2018
OK output to stdout by /bin/echo

I checked the RHEL6 client host running nrpe under xinetd and found that one of the above check_nrpe commands resulted in the following messages being logged in /var/log/messages:

Feb 19 14:52:37 xxxx nrpe[22719]: Error: Request contained command arguments, but argument option is not enabled!
Feb 19 14:52:37 xxxx nrpe[22719]: Client request from was invalid, bailing out...

I don't understand why most of the check_nrpe commands didn't cause these messages to be logged and why one of them did. What would explain this intermittent behavior?

Thanks
Last edited by tmcdonald on Mon Feb 19, 2018 10:44 am, edited 1 time in total.
Reason: Please use [code][/code] tags around long output
bolson

Re: nrpe intermittent Error: Request contained command argum

Post by bolson »

One or more of your check commands contains arguments. The setting dont_blame_nrpe = 0 instructs nrpe to not accept command line arguments and instead use the arguments hard coded in your command definitions. If, after looking through your various check commands you find that this is not the case, let us know and we'll investigate further. Thank you!
stanrate
Posts: 9
Joined: Thu Feb 15, 2018 7:49 pm

Re: nrpe intermittent Error: Request contained command argum

Post by stanrate »

Please note that none of our nrpe check commands contain arguments. If fact, there is only one check command (shown in an earlier post). It does not contain arguments. This is an intermittent problem. 99% of the time when the check command is run, the error messages are not logged. The messages are logged less than 1% of the time. The problem occurs randomly. The Nagios server checks the client a couple of times every minute. Sometimes the error messages are not logged for one or two hours. Sometimes they are logged several times an hour.

I suspect this problem might be caused by a bug in nrpe. I suspect that when nrpe checks whether an argument has been provided, it checks in the wrong place. When a V3 packet has been received, I think it would be logical to check for a "!" in the V3 buffer rather than in the V2 buffer. I suggest changing the following line of code in nrpe.c

from

if (strchr(v2pkt->buffer, '!')) {

to

if (packet_ver == NRPE_PACKET_VERSION_2 && strchr(v2pkt->buffer, '!' || packet_ver == NRPE_PACKET_VERSION_3 && strchr(v3pkt->buffer, '!')) {

Thanks and I hope you have a great day!
Best regards,
stanrate
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: nrpe intermittent Error: Request contained command argum

Post by tmcdonald »

Do you have any theories as to why this would appear randomly though? I would need to speak to the NRPE developers (or take a peak at the source myself) to get a better idea, but if you have any thoughts on the matter it could speed things up. The snippet you provided seems to be pretty deterministic.
Former Nagios employee
stanrate
Posts: 9
Joined: Thu Feb 15, 2018 7:49 pm

Re: nrpe intermittent Error: Request contained command argum

Post by stanrate »

Please get the Developers to investigate what I suspect to be a bug and to provide a fix asap.
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: nrpe intermittent Error: Request contained command argum

Post by tmcdonald »

The fastest way for you to get this in front of our developers is to open an issue on our GitHub page: https://github.com/NagiosEnterprises/nrpe

I am not able to prioritize their workflow to get an ASAP fix for a suspected bug without being able to replicate it internally. I have looked at the code you referenced and cannot see how that would be a random issue - it seems it would appear every time if it was in fact in that section of code. That is why I asked you for any theories you might have, so I can try to force that situation in my test environment, or otherwise look at the code with a specific trigger in mind.
Former Nagios employee
stanrate
Posts: 9
Joined: Thu Feb 15, 2018 7:49 pm

Re: nrpe intermittent Error: Request contained command argum

Post by stanrate »

It is very simple to run a test to illustrate that nrpe is non-deterministic and does not properly check whether an argument has been passed.

I found that nrpe.c does not check the correct string for a "!" to find out whether arguments have been passed.

There are two consequences as a result of this bug:
1. If you have dont_blame_nrpe=0 and run check_nrpe specifying "-a xxxx" i.e. an argument, most of the time nrpe will not log the following two messages (which should be logged):

Feb 23 08:43:48 xxxx nrpe[25715]: Error: Request contained command arguments, but argument option is not enabled!
Feb 23 08:43:48 xxxx nrpe[25715]: Client request from was invalid, bailing out...

2. If you have dont_blame_nrpe=0 and run check_nrpe without specifying "-a xxxx" i.e. no argument, sometimes nrpe will log the above two messages even though they should not be logged.

If you run a command similar to the following command for a few minutes, it will invoke nrpe once a second. Most of the time, the above two messages will not be logged even though they should be logged. Just replace xxxx with your nrpe hostname.

while date;do /usr/lib64/nagios/plugins/check_nrpe -H xxxx -c check_success -a my_arg;sleep 1;done

If you remove "-a my_arg" and run the command for a few minutes, occasionally the two messages will be logged even though they should not be logged.

I'll be away on leave next week and back the following week.

Thanks
Locked