check_by_ssh errors in web UI

[nag4csec42]

I am seeing an issue with the check_by_ssh check. I have no issues with our ubuntu and centos linux / freebsd servers. I am seeing an issue with three new debian based servers. I have tried changing the nagios user default shell from /bin/sh to /bin/bash, and that has not helped.

Here are the issues:

Untitled.png

If I add the -E, I receive the 'return code of 255 out of bounds' error. If I remove the -E from the command, I receive the 'remote command execution failed, host key verification failed'.

The command works from the command line from the nagios server as the nagios user and it also works locally on the server I am attempting to monitor. I can also run the command successfully from the run check command test in the web UI, but do not get the same results in the web UI.

Untitled 2.png

Thanks in advance!

[lmiltchev]

We haven't been able to recreate the issue in house. I ran some check_by_ssh checks against a Debian 9 machine, and they seemed to work fine (both, from the CLI, and the web UI).

Code: Select all

root@debian:~# cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Code: Select all

[nagios@main-nagios-xi .ssh]$ /usr/local/nagios/libexec/check_by_ssh -H 192.168.3.107 -E -C "/usr/local/nagios/libexec/check_disk -w 10% -c 5% /"
DISK OK - free space: / 12404 MB (73.23% inode=87%);| /=4533MB;16080;16973;0;17867

example01.PNG

example02.PNG

It's possible that the IP on the Debian box changed. Can you compare the IP that you use while testing from the CLI to the one in your configs?

You are running all these checks locally, other words, you are not using ModGearman workers, are you?

[nag4csec42]

I am running the checks from a remote ssh pub key login from the nagios server. When I test the same check from the nagios server, I receive this:

Code: Select all

[nagios@nagios1 ~]$ /usr/local/nagios/libexec/check_by_ssh -H 1xx.xx.xx.61 -t 60 -C "/usr/local/nagios/libexec/check_disk -w 10% -c 5% /" -E
DISK OK - free space: / 712184 MB (70% inode=99%);| /=300103MB;959835;1013159;0;1066484
[nagios@nagios1 ~]$ /usr/local/nagios/libexec/check_by_ssh -H 1xx.xx.xx.61 -t 60 -C "/usr/local/nagios/libexec/check_disk -w 10% -c 5% /"
DISK OK - free space: / 712180 MB (70% inode=99%);| /=300107MB;959835;1013159;0;1066484

If I run the check from the web UI, I get this:

Untitled 2.png

output from cat /etc/*release:
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

[npolovenko]

@nag4csec42, Why do you have "-v" at the end of your command on the screenshot? "v" usually stands for verbose, this could explain the unusual output.

[nag4csec42]

The -v is to create verbose output for troubleshooting this issue.

I receive the same issue without the -v. Run check via web UI:

Untitled5.png

Here is the output in Nagios XI (check at the top):

Untitled6.png

Could it have something to do with I am running Debian 8 Jessie and you are testing on Debian 9?

[npolovenko]

@nag4csec42, Please send me a system profile from your nagios server:
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file, upload it to a cloud storage of your choice and share a link with me in a personal message.

Also, from the remote server that you're trying to monitor, please send me the following file:

Code: Select all

/var/log/secure

and

Code: Select all

/var/log/messages

I'd like to get the actual IP of the remote server so that I could verify that your host configuration is correct. If you could PM it to me that would be great.

*After you send the files please post something in this thread to bring it back up in the support queue.

[nag4csec42]

These servers do not have a /var/log/secure file. The logs within /var/log/messages are only rsyslog service related messages.

I sent a PM to you as well.

[npolovenko]

@nag4csec42, I was looking into some kind of log file that would at least show if there was a failed ssh access attempt from the Nagios server. You may have to enable the access logging in /etc/rsyslog.conf
PS: Let me know if you decided to open a ticket with us so I can close this thread and take it from there.