check_by_ssh works fine on cli, but not in UI

[tbmay]

Hello.

We're using Nagios XI 2014R1.1 on CentOS 6.5.

I have a couple of remote boxes I need to monitor over an ssh tunnel. Had no problems making that happen with VMWare hosts; however, using the check_by_ssh plugin is giving some trouble in the UI. It is working fine on the command line.

CLI

[root@nagios libexec]# su - nagios -c '/usr/local/nagios/libexec/check_by_ssh -H localhost -i /home/nagios/.ssh/id_rsa -l root -p 32222 -C "/usr/local/bin/check_disk -w 20% -c 10% -p /home" -E'
DISK OK - free space: /home 318081 MB (79% inode=99%);| /home=83669MB;338600;380925;0;423251


Test from UI - note....I added .ssh to /var/www just to see what happened. No key is in there. It's like apache is trying to set up

Testing check from command line...

COMMAND: /usr/local/nagios/libexec/check_by_ssh -H localhost -i /home/nagios/.ssh/id_rsa -l root -p 32222 -C "check_disk -w 20% -c 10% -p /home"
OUTPUT: Remote command execution failed: Failed to add the host to the list of known hosts (/var/www/.ssh/known_hosts).

I went ahead and opened the permissions up on /var/www/.ssh/ and let apache make the known_hosts file and ran the test again.

Testing check from command line...

COMMAND: /usr/local/nagios/libexec/check_by_ssh -H localhost -i /home/nagios/.ssh/id_rsa -l root -p 32222 -C "check_disk -w 20% -c 10% -p /home"
OUTPUT: Remote command execution failed: Permission denied, please try again.

One more time from the CLI.....

[root@nagios libexec]# su - nagios -c '/usr/local/nagios/libexec/check_by_ssh -H localhost -i /home/nagios/.ssh/id_rsa -l root -p 32222 -C "/usr/local/bin/check_disk -w 20% -c 10% -p /home"'
DISK OK - free space: /home 318081 MB (79% inode=99%);| /home=83669MB;338600;380925;0;423251
[root@nagios libexec]#


Since keys are involved here, I'd rather not run this command as apache.

I set StrictHostKeyChecking no in /etc/ssh/ssh_config even though it was fine on the CLI, and it moved me from an earlier error to the one I'm getting now.



What am I missing here.

Thanks a million.

PS....I tried again adding the full path to the UI but it still didn't work.

Testing check from command line...

COMMAND: /usr/local/nagios/libexec/check_by_ssh -H localhost -i /home/nagios/.ssh/id_rsa -l root -p 32222 -C "/usr/local/bin/check_disk -w 20% -c 10% -p /home"
OUTPUT: Remote command execution failed: Permission denied, please try again.

[tmcdonald]

The UI and the CLI will run as different users with different permissions, so some checks simply won't work when using the "Test Check Command" button. The UI also has some escaping done due to being a PHP page, so there can be issues with backslashes. Testing from the CLI as the Nagios user is the best way to test.

[tbmay]

Thanks.

I did and it worked. Notice I su'ed here...but I also logged in a nagios with successful results.

[root@nagios libexec]# su - nagios -c '/usr/local/nagios/libexec/check_by_ssh -H localhost -i /home/nagios/.ssh/id_rsa -l root -p 32222 -C "/usr/local/bin/check_disk -w 20% -c 10% -p /home"'
DISK OK - free space: /home 318081 MB (79% inode=99%);| /home=83669MB;338600;380925;0;423251


At first blush it appears apache is trying to run the command.

[tmcdonald]

Correct, when run in the UI the apache user will run the check. If the "Test Check Command" fails, the next step is usually to run it from the CLI and if that works, apply config and in the service's detail page you can click the "Schedule forced immediate check" (verbage may change based on what version you are on).

[tbmay]

Thanks. Since I'm dealing with an ssh keypair I'm not crazy about apache running the command. I can tell from watching it bonk out it want's the private key in /var/www/.ssh. BTW..it doesn't work forcing a check once it's configured either. Any thoughts for the most practial way to run this check without putting a private key in /var/www/.ssh?

[abrist]

Apache will not run the command once it is configured and scheduled in nagios. Only when testing from the UI, and no, you cannot change the user the "test check command" runs as.

[tbmay]

Understand. I never got it to work in the UI. I have a screenshot up the string showing what I got. I searched the forum this past weekend and saw others had a similar problem with connect_by_ssh...works fine on cli but not UI. The particular string I read didn't seem to end with any resolution so I'm going to assume this is bug. I don't have time, at the moment, to search for it.

For now I am checking these remote machines outside of Nagios with a bash script crontab'ed once a day. This is OK for these particular boxes because we only need a limited amount of data from them, but it would be nice for this work in NagiosXI lest we need it in the future.

I know I can write plugin that reads the output of a local file created from the output of the command on the CLI...and handle it with proper Nagios escape codes....in a crontab'ed script for instance. However, I wouldn't think that would be necessary since we actually have a check_by_ssh option.

I've included all the details I can think of, if you have any ideas, I would appreciate it. If you need more information, just let me know.

[abrist]

hmmm. I just tested this. The test command, as mentioned before, indeed runs as apache. But the check itself (scheduled and run by nagios) does run as the user 'nagios'. If you are receiving "Permission Denied", it most likely not due to the user key, but permissions on the plugins or key instead. Here are the steps I followed:
On the XI server:

Code: Select all

su nagios
cd /home/nagios
ssh-keygen -t dsa

Now copy the contents of home/nagios/.ss/id_rsa.pub to the remote system's authorized keyfile located at:

Code: Select all

/home/nagios/.ssh/authorized_keys

Now test the connection from the XI server:

Code: Select all

su nagios
ssh <ip of remote>

If it works, you are all set. Configure the checks in XI through the CCM or the SSH proxy wizard. Don't bother testing them in the UI as the test will run as apache. Just configure the checks and apply config. Within a few minutes you should see the check results coming through.

[tbmay]

Thanks. I have been ssh'ing in to the remote machine as root and don't even have a nagios user on that box. (I know that's bad security but this isn't really mission critical and time is at a premium this time of year.) The local user is nagios and, as we've discussed, is perfectly able to ssh in to the box as root with the existing keys on the CLI. I will create a nagios user on the remote box and give this a try; however, I'm not sure why it would make any difference since local nagios is ssh'ing in to remote root with no issues on the command line.

Thanks for your help.

[Box293]

When check_by_ssh runs, it tries to connect to the remote machine as the user who you are currently logged in as.

So if you are logged in as root, it tries to connect to the remote machine as the user root.

When the check is scheduled and run by nagios, it tries to connect to the remote machine as the user nagios.

This will be the reason why you are getting the permission denied errors (because you don't have a nagios user on that box).

IF you wanted the check that is scheduled and run by nagios to connect to the remote system as root, you will need to add this to your service definition / command definition:

Code: Select all

-l root

That's a lower case L.

HOWEVER I strongly advise against it, permissions, security and all that important stuff.