nrpe.cfg reconfig?

[xpac]

Yep that works fine. In fact, check_nrpe commands work fine for everything other than this plugin.

That said, I "think" I might have found the problem.

To allow the nagios user to run the necessary Cassandra command, I had to add it to the group that has permissions to run the Cassandra command (in addition to its primary nagios group of course).

However, when I look at the /etc/xinetd.d/nrpe file, I noticed that it has specified
user = nagios
group = nagios.

Could the problem be that "group" nagios does not have permission to run the Cassandra command? And if so, can I just add an additional group to this line in file, or does NRPE require that only one group is specified?

[xpac]

Bummer, didn't work either. I even tried changing the user and group in the NRPE conf file to the Cassandra user, still gets the silly output

[Box293]

Do you need to add this Cassandra command to sudoers?

What happens when you run the command on the remote server as the user nagios?

Code: Select all

su nagios
/usr/local/nagios/libexec/check_cassandra_cluster.sh -H localhost -P 7199 -w 1 -c 0

[xpac]

I'm able to run the Cassandra command fine as user nagios, both locally on the machine and via ssh.

The only problem seems to be trying to run it via NRPE for some reason

[lmiltchev]

./check_nrpe -H mytestcassandranode-c check_cassandra_cluster -a '-H mytestcassandranode -P 7199 -w 1 -c 0'

Is "mytestcassandranode-c" a typo? You need to have a space there:

Code: Select all

./check_nrpe -H mytestcassandranode -c check_cassandra_cluster -a '-H mytestcassandranode -P 7199 -w 1 -c 0'

Can you show us how the "check_cassandra_cluster" command is defined on the client (remote machine)? Also, do you have "dont_blame_nrpe=1" in the "nrpe.cfg" file?

[xpac]

Is "mytestcassandranode-c" a typo? You need to have a space there

Yep it was just a typo, would've been awesome if it was the problem

Can you show us how the "check_cassandra_cluster" command is defined on the client (remote machine)? Also, do you have "dont_blame_nrpe=1" in the "nrpe.cfg" file?

don't_blame_nrpe=1 is definitely defined, this is a totally clean virtual test environment so I started from scratch configuring the client properly to allow command arguments.

However, I've now switched to a plugin written by someone who has the arguments actually hardcoded in plugin/script, but still have the same problem. With the new script that doesn't require arguments here is the command definition from the client nrpe file:

command[check_cassandra]=/usr/local/nagios/libexec/check_cassandra

I've also tried it by putting the exact host in the definition:

command[check_cassandra]=/usr/local/nagios/libexec/check_cassandra - H IPaddressofCassandrahost

[Box293]

don't_blame_nrpe=1 is definitely defined, this is a totally clean virtual test environment so I started from scratch configuring the client properly to allow command arguments.

I would double check that with your clean system that you compiled using --enable-command-args

command[check_cassandra]=/usr/local/nagios/libexec/check_cassandra - H IPaddressofCassandrahost

Defining the arguments in the command definition is the best way to diagnose the problem. If it doesn't work like this then there is something else going on.

Now I know this is probably another typo but I'm going to point it out:

Code: Select all

command[check_cassandra]=/usr/local/nagios/libexec/check_cassandra - H IPaddressofCassandrahost

There's a space between - and H

Code: Select all

command[check_cassandra]=/usr/local/nagios/libexec/check_cassandra -H IPaddressofCassandrahost

Does the nagios user have a home directory on the remote host?


OK to share some thoughts about other things to look at.

I'm seen with NRPE running plugins in the past that the plugin itself might be using an internal command which works fine as the nagios user who is logged on, but when it's not in an interactive session then it fails.

An example of this is the "top" command.

Code: Select all

top -n 1

It runs once and terminates. But if used in a script in a non-interactive session, it needs to be:

Code: Select all

top -n 1 -b

One of the troubleshooting steps I take in this situation is to comment out all the lines in the script except for the first two and then test it with a minimal script. If that works then keep uncommenting lines until the problem appears.

[xpac]

I would double check that with your clean system that you compiled using --enable-command-args

Yep this has been verified, I can run other plugins w/args just fine remotely via NRPE, such as "check_users"

Now I know this is probably another typo but I'm going to point it out:

Yep another typo

Does the nagios user have a home directory on the remote host?

It does, and I made sure the nagios user also can execute the necessary Cassandra commands both locally on that machine and remotely via ssh so we're good there.

I think I'll have to take a look at the plugin/s themselves next as you suggested. The problem definitely lies with NRPE and/or the plugin itself - as I mentioned earlier when I run the command via NRPE it IMMEDIATELY returns the incorrect value, the Cassandra command at the very minimum requires 4-5 seconds to run locally and produce the output (more if there is a problem or the Cassandra cluster is under a heavy workload), so there is no way the command is being properly executed.

[ssax]

I haven't seen anyone ask if you have selinux enabled:

Code: Select all

getenforce

or

Code: Select all

sestatus

[xpac]

I haven't seen anyone ask if you have selinux enabled:

Code: Select all

getenforce

or

Code: Select all

sestatus

In this test instance I have selinux set to permissive and iptables is stopped and set to off.