plugins, variables and nrpe

[rdubya]

Hi, I'm revisiting my bandwidth monitoring plugin. I'm using iperf3 in a client server configuration that's working well in it's first iteration, but this next phase is giving me fits.

My problem is that while the script works from source to destination from the source host;

Code: Select all

/usr/local/nagios/libexec/check_iperf.sh
OK- Throughput: 957|xferrate=957MB;;;1;1000

...and from my nagios server, requesting the same command remotely (source host to queries the destination host);

Code: Select all

ssh <client> /usr/local/nagios/libexec/check_iperf.sh
OK- Throughput: 954|xferrate=954MB;;;1;1000

...running it through checknrpe strips out the return from iperf;

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H <client> -c check_iperf
CRITICAL- Throughput: to|xferrate=toMB;;;1;1000

If it matters, here is my nrpe.cfg command.

Code: Select all

command[check_iperf]=/usr/local/nagios/libexec/check_iperf.sh

At first I thought it might be the variable not coming through because I have assigned the output of iperf3 to a variable, but I've also used variables within the script for the target host name and the scale for the performance metric.

I'll add that this is working fine and graphing data from the Nagios server out to selected and configured hosts, but I want to expand this to connections between hosts that I want to monitor latency.

Any help would be appreciated.

[scottwilkerson]

Make sure the contents of /usr/local/nagios/libexec/check_iperf.sh contain full paths to all binaries as when running under nrpe it will not have the same PATH as your local user.

Also, I would test the command as the nagios user on the remote machine like this

Code: Select all

sudo su nagios -c '/usr/local/nagios/libexec/check_iperf.sh'

[rdubya]

Thanks Scott.

I've made sure the full path to iperf3 (/usr/bin/iperf3) has been called out in the script and I'm still not getting results. Looking a little deeper I was able to find something that helped. By redirecting output I get a little better view into what's going on;

Code: Select all

command[check_iperf]=/usr/local/nagios/libexec/check_iperf.sh 2>&1

By using check_nrpe with the output visible, I can see that ssh is the roadblock.

Code: Select all

[nagios@engmon ~]$  /usr/local/nagios/libexec/check_nrpe -H svpxqacn7lbs01  -c check_iperf
/usr/local/nagios/libexec/check_iperf.sh: [b]line 12: /bin/ssh: Permission denied
CRITICAL- Throughput: to|xferrate=toMB;;;1;1000

On the source for the bandwidth check, the path for ssh under the nagios user is;

Code: Select all

[nagios@svpxqacn7lbs01 ~]$ which ssh
/bin/ssh

I can ssh to the bandwidth test source host;

Code: Select all

[nagios@engmon ~]$ ssh svpxqacn7lbs01
Last login: Mon Jun 25 14:41:47 2018
[nagios@svpxqacn7lbs01 ~]$

And I can ssh to the bandwidth test destination host from the source;

Code: Select all

[nagios@svpxqacn7lbs01 ~]$ ssh svpxxngcn7ljm01
Last login: Mon Jun 25 15:12:37 2018 from 10.100.23.20
bash-4.2$

Lastly, running the command from the nagios server to the source (which probes the destination) works using the same script and parameters.

Code: Select all

[nagios@engmon ~]$ ssh svpxqacn7lbs01 /usr/local/nagios/libexec/check_iperf.sh
OK- Throughput: 955|xferrate=955MB;;;1;1000

I've played a little bit with the path for the SSH command but I'm still a little stuck. From the system standpoint it works well, I don't mind telling you I'm stuck. Your initial thought was helpful, is there anything else that comes to mind?

Thanks, R

[scottwilkerson]

The only thing I can see is that when you test it from the console you have a tty

I have no idea what the script looks like but this could be a problem
Can you share the plugin?

[rdubya]

Why not? If nothing else it's as heinous a hack as anyone here will ever see.
It requires iperf3 on source and target and passwordless ssh for the nagios user from the source to the target. I'm willing to to learn a better way. Eventually I'd use command parameters, but I'm still in "make it work" territory.

Code: Select all

#!/bin/bash

[scottwilkerson]

Line 10 I see this

Code: Select all

/bin/ssh $target iperf3 -s -D -1

I'm guessing you need a full path to iperf3 on target machine? But honestly I don't really understand what that ssh command is doing because in your example showing you can run a command via ssh you used the following which is a different command with a full path

Code: Select all

 ssh svpxqacn7lbs01 /usr/local/nagios/libexec/check_iperf.sh

Also, I don't see anything being done with the data from that line so I would first try commenting it out....

[rdubya]

Yeah, my comment was out of place for that. The iperf command requires something to connect to (the destination).
This is what those lines should look like;

Code: Select all

#Start iperf3 as a daemon on $target for one connection only
/bin/ssh $target /usr/bin/iperf3 -s -D -1

And agreed. I'm not sure why nrpe is stumbling on the ssh command either since there's not a problem otherwise.

[scottwilkerson]

Does the nagios user have the rights to start something as a daemon on that system? That could be why you are getting the Permission denied

[rdubya]

I could find no logic into why NRPE was roadblocking me and I'd spent enough time on it. I ended up splitting the process into two scripts; one on the server to call the script on the remote source host and the actual command on the source host that does the work. It's not that much more overhead and it solved a much needed requirement for me. I'm going to pull down my existing script to remove any possible confusion, if I ever sort out the original problem I may publish a plugin. Thanks for the help.

[scottwilkerson]

thanks again for keeping the thread tidy and sharing!