Check_by_ssh issue

[MarMottE]

it's an ubuntu server 14.04 LTS on vmware

[rkennedy]

What sort of storage is the client machine using?

Let's try taking Nagios out of the equation completely. Without a SSH session opened, what is the output of the following commands executed from the XI machine?

Code: Select all

su nagios
ssh [email protected] "uptime"

[MarMottE]

What sort of storage is the client machine using?

Let's try taking Nagios out of the equation completely. Without a SSH session opened, what is the output of the following commands executed from the XI machine?

Code: Select all

su nagios
ssh [email protected] "uptime"

Local Storage

root@nagiosserver:/home/nagios# su - nagios
nagios@nagiosserver:~$ ssh nagios@xxxx "uptime"
07:46:52 up 6 days, 23:28, 0 users, load average: 0.13, 0.11, 0.19

[tgriep]

The only thing left to do is to run the sshd server in debug mode on the remote system and see if we can get any errors.
You will have to do this from the console of the system if it is a physical server, you will have to be there, if it is running under VMWare, you will have to have a console open.
Login to that systems console and stop the sshd daemon from running in the background.

Code: Select all

service sshd stop

Then run the sshd daemon in the foreground with debugging enabled.

Code: Select all

/usr/sbin/sshd -D -d

Make the failure happen and post the output from the console.
Else, check the logs on the remote server and see if there are any errors in it.

[MarMottE]

Please find bellow the debug of the ssh

root@remoteserver:# /usr/sbin/sshd -D -d
debug1: sshd version OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type ECDSA
debug1: private host key: #2 type 3 ECDSA
debug1: private host key: #3 type 4 ED25519
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-D'
debug1: rexec_argv[2]='-d'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from nagiosserver port 34580 on remoteserver port 22
debug1: Client protocol version 2.0; client software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 pat OpenSSH_6.6.1* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
debug1: permanently_set_uid: 104/65534 [preauth]
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: client->server aes128-ctr [email protected] none [preauth]
debug1: kex: server->client aes128-ctr [email protected] none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Connection closed by nagiosserver [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: Killing privsep child 11997
debug1: audit_event: unhandled event 12


The commande /usr/sbin/sshd -D -d start the ssh deamon definitively or just for the debug ?

[tgriep]

Running the sshd daemon this way was only for this time to see if we can get any errors that could help but it all looked good.
When you ran this test, did the check_by_ssh fail or did it pass?
If it passed, we need to run it again until it fails so we can get some error messages.
A few suggestions,
1. Redo the ssh public and private keys for the server and see if that helps.
2. Run the command from a command prompt and add the -v option to the check_by_ssh command to get the verbose output while it is failing. That should give you a clue on what the error is.

[MarMottE]

When I have run this test, the check was up when i have established the session on the nagios server

root@beuxp0014162:/home/migeonj# /usr/sbin/sshd -D -d
debug1: sshd version OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type ECDSA
debug1: private host key: #2 type 3 ECDSA
debug1: private host key: #3 type 4 ED25519
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-D'
debug1: rexec_argv[2]='-d'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from xxxx port 39534 on xxxx port 22
debug1: Client protocol version 2.0; client software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 pat OpenSSH_6.6.1* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
debug1: permanently_set_uid: 104/65534 [preauth]
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: client->server aes128-ctr [email protected] none [preauth]
debug1: kex: server->client aes128-ctr [email protected] none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user nagios service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "nagios"
debug1: PAM: setting PAM_RHOST to "xxxx"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user nagios service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: test whether pkalg/pkblob are acceptable [preauth]
debug1: temporarily_use_uid: 1002/1002 (e=0/0)
debug1: trying public key file /home/nagios/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /home/nagios/.ssh/authorized_keys, line 1 DSA b9:ce:73:f8:11:e0:fd:b9:1d:57:70:e3:fe:3b:cc:fd
debug1: restore_uid: 0/0
Postponed publickey for nagios from xxxx port 39534 ssh2 [preauth]
debug1: userauth-request for user nagios service ssh-connection method publickey [preauth]
debug1: attempt 2 failures 0 [preauth]
debug1: temporarily_use_uid: 1002/1002 (e=0/0)
debug1: trying public key file /home/nagios/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /home/nagios/.ssh/authorized_keys, line 1 DSA b9:ce:73:f8:11:e0:fd:b9:1d:57:70:e3:fe:3b:cc:fd
debug1: restore_uid: 0/0
debug1: ssh_dss_verify: signature correct
debug1: do_pam_account: called
Accepted publickey for nagios from xxxx port 39534 ssh2: DSA b9:ce:73:f8:11:e0:fd:b9:1d:57:70:e3:fe:3b:cc:fd
debug1: monitor_child_preauth: nagios has been authenticated by privileged process
debug1: monitor_read_log: child log fd closed
debug1: PAM: establishing credentials
User child is on pid 19984
debug1: SELinux support disabled
debug1: PAM: establishing credentials
debug1: permanently_set_uid: 1002/1002
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 2097152 max 32768
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype [email protected] want_reply 0
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request exec reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req exec
Starting session: command for nagios from xxxx port 39534
debug1: Received SIGCHLD.
debug1: session_by_pid: pid 19985
debug1: session_exit_message: session 0 channel 0 pid 19985
debug1: session_exit_message: release channel 0
debug1: session_by_channel: session 0 channel 0
debug1: session_close_by_channel: channel 0 child 0
debug1: session_close: session 0 pid 0
debug1: channel 0: free: server-session, nchannels 1
Received disconnect from xxxx: 11: disconnected by user
debug1: do_cleanup
debug1: do_cleanup
debug1: PAM: cleanup
debug1: PAM: closing session
debug1: PAM: deleting credentials
debug1: audit_event: unhandled event 12


When I execute the command and i'm closing my ssh connexion on the nagios server


debug1: sshd version OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type ECDSA
debug1: private host key: #2 type 3 ECDSA
debug1: private host key: #3 type 4 ED25519
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-D'
debug1: rexec_argv[2]='-d'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from xxxx port 39930 on xxxxx port 22
debug1: Client protocol version 2.0; client software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 pat OpenSSH_6.6.1* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
debug1: permanently_set_uid: 104/65534 [preauth]
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: client->server aes128-ctr [email protected] none [preauth]
debug1: kex: server->client aes128-ctr [email protected] none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Connection closed by xxxxx [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: Killing privsep child 20344
debug1: audit_event: unhandled event 12

[tgriep]

There has to be something wrong with sync on the keys between the 2 servers.
Try deleting the settings on both the remote server and the XI server and re-read this document to setup the public keys, etc...
https://assets.nagios.com/downloads/nag ... ng_SSH.pdf
Also, you may want to try and put the sshd_config back to the default settings on the remote system and see if that helps.

[MarMottE]

Hello,

Just to clarify, If I open a console session the check goes up, and If I logout from my console session, the check goes down.

Yes it's only the check_by_ssh but I don't think it's really a problem with the SSH client or something like that

[rkennedy]

Hello,

Just to clarify, If I open a console session the check goes up, and If I logout from my console session, the check goes down.

Yes it's only the check_by_ssh but I don't think it's really a problem with the SSH client or something like that

The behavior your seeing, is something that we have not ever had an issue with previously. This leads me to believe that it is indeed something with your SSH configuration file.

Code: Select all

Also, you may want to try and put the sshd_config back to the default settings on the remote system and see if that helps.

Has your sshd_config been modified or is it still stock? If changes have been made, it's hard to say what all exactly it will affect.