check_by_ssh on Solaris 11.2 fails

[steveherring]

We have a script that was working fine on Solaris 11.1 however we updated the zone to Solaris 11.2 and it now returns:
Remote command execution failed: *******************************

we wrote a simple test script and ran it:
:: more test_script.sh
#!/bin/bash

echo "!!Script Executed Successfully!!"
exit 0
This is how we ran it and what we got:
nagios@nagios:~$ check_by_ssh -H ldap2.cs.tamu.edu -l gfi_monitor -i /etc/opt/csw/nagios/gfi_monitor_id_dsa -C "/admin/gfi_monitor/nms/Nagios_Scripts/test_script.sh" -o StrictHostKeyChecking=no -v
Command: /opt/csw/bin/ssh
Argument 1: -l
Argument 2: gfi_monitor
Argument 3: -i
Argument 4: /etc/opt/csw/nagios/gfi_monitor_id_dsa
Argument 5: -o
Argument 6: StrictHostKeyChecking=no
Argument 7: ldap2.cs.tamu.edu
Argument 8: /admin/gfi_monitor/nms/Nagios_Scripts/test_script.sh
Remote command execution failed: *******************************
nagios@nagios:~$ echo $?
3

This is the tail end of a truss for it:
4268: write(4, " 9DCB29F N7F IA38DD7 GEE".., 96) = 96
4268: pollsys(0xFEFFDBD0, 1, 0x00000000, 0x00000000) = 1
4268: read(4, "C4DED0B0CC17 oD2 4\r +C2".., 8192) = 80
4268: pollsys(0xFEFFDBD0, 2, 0x00000000, 0x00000000) = 1
4268: read(4, " UC2A4BFDB LC2A3 b 4 HB8".., 8192) = 272
4268: close(5) = 0
4268: pollsys(0xFEFFDBD0, 2, 0x00000000, 0x00000000) = 1
4268: write(6, " ! ! S c r i p t E x e".., 33) = 33
4267: read(4, " ! ! S c r i p t E x e".., 4096) = 33
4268: close(6) = 0
4268: close(7) = 0
4268: pollsys(0xFEFFDBD0, 1, 0x00000000, 0x00000000) = 1
4268: write(4, " j *AE m93 dFDBC j0E19AB".., 32) = 32
4268: sigaction(SIGWINCH, 0x00000000, 0xFEFFDD20) = 0
4268: sigaction(SIGWINCH, 0xFEFFDCC0, 0x00000000) = 0
4268: write(4, "A8 LA7D4 2 D $BEDAEE879F".., 64) = 64
4268: ioctl(0, TCGETA, 0xFEFFDD40) = 0
4268: ioctl(1, TCGETA, 0xFEFFDD40) Err#22 EINVAL
4268: fcntl(1, F_GETFL) = 130
4268: fcntl(1, F_SETFL, FWRITE) = 0
4268: ioctl(2, TCGETA, 0xFEFFDD40) Err#22 EINVAL
4268: fcntl(2, F_GETFL) = 130
4268: fcntl(2, F_SETFL, FWRITE) = 0
4268: shutdown(4, SHUT_RDWR, SOV_DEFAULT) = 0
4268: close(4) = 0
4268: _exit(0)
4267: read(4, 0xFEFFDAB0, 4096) = 0
4267: read(6, " * * * * * * * * * * * *".., 4096) = 1017
4267: read(6, 0xFEFFDAB0, 4096) = 0
4267: close(4) = 0
4267: Received signal #18, SIGCLD, in waitid() [default]
4267: siginfo: SIGCLD CLD_EXITED pid=4268 status=0x0000
4267: waitid(P_PID, 4268, 0xFEFFEA10, WEXITED|WTRAPPED) = 0
4267: getuid() = 100 [100]
4267: getuid() = 100 [100]
4267: getgid() = 101 [101]
4267: getgid() = 101 [101]
4267: write(1, " R e m o t e c o m m a".., 65) = 65
4267: _exit(3)
The bolded parts seem to imply that the script ran and returned a 0, but for some reason Nagios is failing it after that

[eloyd]

My big guess? SSH Kost Keys changed and now your SSH isn't working as the nagios user from Nagios box to Solaris box.

Try su - nagios and ssh'ing to the remote box. I'll bet you get a hostkey verification failure.

[steveherring]

Sorry I forgot to mention that it works fine when we are actually SSH into the box:

nagios@nagios:~$ ssh -i ~/gfi_monitor_id_dsa -l gfi_monitor ldap2.cs.tamu.edu /admin/gfi_monitor/nms/Nagios_Scripts/test_script.sh

***removed MOTD***

!!Script Executed Successfully!!
nagios@nagios:~$ echo $?
0

[eloyd]

Wow, that is a complex command:

check_by_ssh -H ldap2.cs.tamu.edu -l gfi_monitor -i /etc/opt/csw/nagios/gfi_monitor_id_dsa -C "/admin/gfi_monitor/nms/Nagios_Scripts/test_script.sh" -o StrictHostKeyChecking=no -v

What about:

Code: Select all

su - nagios
check_by_ssh -H ldap2.cs.tamu.edu -l gfi_monitor -i /etc/opt/csw/nagios/gfi_monitor_id_dsa -o "StrickHostKeyChecking=no" -C "hostname"

Does that work?

And my next suggestion is going to be put all those options in ~nagios/.ssh/config so that you can skip them on the command line and simplify things a bit.

[abrist]

Let us know if this all worked for you OP.