SSH keys based sheduled backup "Could not authenticate"

[oskargaboda]

Hi all,

I'm trying to configure a SSH backup between two XI servers, the "Test Connection" and "Test SCP transfer" both fail with a "Could not authenticate. " error. These are configured with the Publick Key method, manually executing a ssh/scp using the private key created by XI succeeds.

Config is as follows
SSH Server : FQDN of target
User : nagios
Port: 22
SSH Auth Type: Public Key
Remote Directory : /apps/backups/monprdappla011/

Server OS: RHE 8.5 ( both source/destination are the same image/build )
CIS hardened

Key details/permissions on source

-rw-r----- 1 apache nagios 2635 Nov 25 11:01 /usr/local/nagiosxi/var/keys/ssh.xi.1637798479
-rw-r----- 1 apache nagios 554 Nov 25 11:01 /usr/local/nagiosxi/var/keys/ssh.xi.1637798479.pub


The remote server is reporting the following in /var/log/secure
Dec 15 15:05:03 sshd[3400658]: AuthorizedKeysCommand /opt/ssh/ssh_auth_keys nagios failed, status 2
Dec 15 15:05:03 monprdappla019 sshd[3400658]: Accepted key RSA SHA256:d/lUV1ONzhBXTh5LrMf1CGgfZr5zYbRLpJs9ItKxzAg found at /home/nagios/.ssh/authorized_keys:3
Dec 15 15:05:03 monprdappla019 sshd[3400658]: Postponed publickey for nagios from 10.118.3.171 port 40246 ssh2 [preauth]
Dec 15 15:05:03 monprdappla019 sshd[3400658]: Received disconnect from 10.118.3.171 port 40246:11: PECL/ssh2 (http://pecl.php.net/packages/ssh2) [preauth]
Dec 15 15:05:03 monprdappla019 sshd[3400658]: Disconnected from authenticating user nagios 10.118.3.171 port 40246 [preauth]
Dec 15 15:05:03 monprdappla019 systemd: pam_unix(systemd-user:session): session closed for user root
Dec 15 15:05:03 monprdappla019 sshd[3400669]: Connection from 10.118.3.171 port 40248 on 10.118.3.176 port 22
Dec 15 15:05:04 monprdappla019 sshd[3400669]: AuthorizedKeysCommand /opt/ssh/ssh_auth_keys nagios failed, status 2
Dec 15 15:05:04 monprdappla019 sshd[3400669]: Accepted key RSA SHA256:d/lUV1ONzhBXTh5LrMf1CGgfZr5zYbRLpJs9ItKxzAg found at /home/nagios/.ssh/authorized_keys:3
Dec 15 15:05:04 monprdappla019 sshd[3400669]: Postponed publickey for nagios from 10.118.3.171 port 40248 ssh2 [preauth]
Dec 15 15:05:04 monprdappla019 sshd[3400669]: Received disconnect from 10.118.3.171 port 40248:11: PECL/ssh2 (http://pecl.php.net/packages/ssh2) [preauth]
Dec 15 15:05:04 monprdappla019 sshd[3400669]: Disconnected from authenticating user nagios 10.118.3.171 port 40248 [preauth]




As the sourcecode is protected , we're unable to determine what XI is performing.

Thanks in advance

[kfanselow]

Hi oskargaboda,

Just wanted to clarify - when you run ssh on the command line it authenticates successfully ?

So there are a couple of things to look at, could you run the following commands and provide the output:

As root run the following on both systems:

Code: Select all

sestatus

On your XI server as the nagios user could you run the following:

Code: Select all

ssh -i /usr/local/nagiosxi/var/keys/ssh.xi.1637798479  nagios@YOURFQDN  uname -a 

Even though we're not seeing it in the logs and it sounds like you may be authenticating successfully could you verify the perms on the authorized_keys file ?

Code: Select all

ls -laF /home/nagios/.ssh/authorized_keys

Also in your ssh_config on the client side try changing

GSSAPIAuthentication yes
to
GSSAPIAuthentication no

and see if there is any change in behavior.

Thanks and Best Regards,
Keith

[oskargaboda]

Hi Keith, thanks for your response. Yes connects successfully. I've run the commands and made the changes to the sshd_config however find no change.

[root@monprdappla011 ~]# sestatus
SELinux status: disabled
[root@monprdappla011 ~]#

[root@monprdappla019 ~]# sestatus
SELinux status: disabled
[root@monprdappla019 ~]#

[nagios@monprdappla011 ~]$ ssh -i /usr/local/nagiosxi/var/keys/ssh.xi.1637798479 nagios@monprdappla019 uname -a
#### Unauthorised access prohibited .All accesses are logged. ####

Enter passphrase for key '/usr/local/nagiosxi/var/keys/ssh.xi.1637798479':
Linux monprdappla019 4.18.0-348.2.1.el8_5.x86_64 #1 SMP Mon Nov 8 13:30:15 EST 2021 x86_64 x86_64 x86_64 GNU/Linux
[nagios@monprdappla011 ~]$

[nagios@monprdappla019 .ssh]$ ls -laF /home/nagios/.ssh/authorized_keys
-rw------- 1 nagios users 1399 Dec 16 17:34 /home/nagios/.ssh/authorized_keys
[nagios@monprdappla019 .ssh]$


Dec 16 17:37:14 monprdappla019 sshd[3865627]: Connection from 10.118.3.171 port 57688 on 10.118.3.176 port 22
Dec 16 17:37:14 monprdappla019 sshd[3865627]: AuthorizedKeysCommand /opt/ssh/ssh_auth_keys nagios failed, status 2
Dec 16 17:37:14 monprdappla019 sshd[3865627]: Accepted key RSA SHA256:d/lUV1ONzhBXTh5LrMf1CGgfZr5zYbRLpJs9ItKxzAg found at /home/nagios/.ssh/authorized_keys:3
Dec 16 17:37:14 monprdappla019 sshd[3865627]: Postponed publickey for nagios from 10.118.3.171 port 57688 ssh2 [preauth]
Dec 16 17:37:14 monprdappla019 sshd[3865627]: Received disconnect from 10.118.3.171 port 57688:11: PECL/ssh2 (http://pecl.php.net/packages/ssh2) [preauth]
Dec 16 17:37:14 monprdappla019 sshd[3865627]: Disconnected from authenticating user nagios 10.118.3.171 port 57688 [preauth]


Regards
Oskar

[kfanselow]

Hi Oskar,

This is an interesting problem - yeah it appears that the keys are setup properly.

Try turning off the the issue banner on the server you are logging into and give it a go - those sometimes interfere with the expected response.

----> "#### Unauthorised access prohibited .All accesses are logged. ####"

If that doesn't have any effect please generate a system profile and send it to me via a private message, along with the sshd_config file from monprdappla019 (I think we have the current ssh_config from monprdappla011) .

To send us your system profile:
- Login to the Nagios XI GUI using a web browser.
- Click the "Admin" (Top) -> "System Profile" Menu (Left)
- Click the "Download Profile" button


Thanks and Best Regards,
Keith

[oskargaboda]

Thanks Keith,


Modifiyed to disable the banner and still get the same error from the UI.

[nagios@monprdappla011 ~]$ ssh -i /usr/local/nagiosxi/var/keys/ssh.xi.1637798479 nagios@monprdappla019 uname -a
Enter passphrase for key '/usr/local/nagiosxi/var/keys/ssh.xi.1637798479':
Linux monprdappla019 4.18.0-348.2.1.el8_5.x86_64 #1 SMP Mon Nov 8 13:30:15 EST 2021 x86_64 x86_64 x86_64 GNU/Linux
[nagios@monprdappla011 ~]$

I will send you the system profile dump.

Regards
Oskar

[kfanselow]

Hi Oskar,

We'll have to take a look into your profile and sshd_config to confirm but, depending upon the changes made by the CIS hardening process it's possible the ciphers being used aren't compatible with the php libraries in use by nagios. Our attempts to replicate your problem haven't been successful as of yet.

Thanks and Best Regards,
Keith

[oskargaboda]

Hi Keith,

Sorry for the late reply as I've been on leave.

Are you able to share details on what ciphers are supported/used by the php libraries in nagios? This will help with my inquiries I will need to make to the linux admin that is assisting me with my investigation.

Regards
Oskar

[kfanselow]

Hi Oskar,

Received your profile - we'll take a look.


Thanks and Best Regards,
Keith

[kfanselow]

Hi Oskar,

In direct answer to your question, based upon the information in the profile I believe the following are available:

Code: Select all

Key Exchange: curve25519-sha256, [email protected], ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group-exchange-sha1, diffie-hellman-group14-sha256, diffie-hellman-group14-sha1, diffie-hellman-group15-sha512, diffie-hellman-group16-sha512, diffie-hellman_group17-sha512, diffie-hellman-group18-sha512, diffie-hellman-group1-sha1

Keys: ssh-ed25519, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp521, rsa-sha2-256, rsa-sha2-512, ssh-rsa, ssh-dss

Encryption:  [email protected], [email protected], arcfour256, arcfour128, aes128-ctr, aes192-ctr, aes256-ctr, [email protected], twofish128-ctr, twofish192-ctr, twofish256-ctr, aes128-cbc, aes192-cbc, aes256-cbc, twofish128-cbc, twofish192-cbc, twofish256-cbc, twofish-cbc, blowfish-ctr, blowfish-cbc, 3des-ctr, 3des-cbc

MAC: [email protected], [email protected], [email protected], [email protected], [email protected], hmac-sha2-256, hmac-sha2-512, [email protected], [email protected], hmac-sha1-96, hmac-sha1, hmac-md5-96, hmac-md5

With that being said if the problem persists I would recommend opening up a ticket so we can take a closer look at your system configuration to see if we can replicate the problem.

Thanks and Best Regards,
Keith