Page 1 of 1
Nagios XI stderr question when running shell scripts...
Posted: Mon Feb 01, 2016 9:43 am
by JakeHatMacys
So we've been running some shell scripts and log in using SSH and we've run into some issues where we can't log into some servers and get "access denied". NagiosXI ends up reporting back by default in the gui that the service check timed out after 60.01 seconds.
My question is I was wondering if you could point me in the right direction of recognizing when we get "access denied" in a shell script. Is that stored in stderr where I could use $ARG1$ or $ARG2$ to recognize it on the service check in core? or is it something I could code into the shell script itself to bring back.
I'm pretty basic when it comes to shell scripting and still attempting to teach myself so any help would be appreciated. We'd like to focus specifically on "access denied" for now anyhow.
Thanks.
Re: Nagios XI stderr question when running shell scripts...
Posted: Mon Feb 01, 2016 12:23 pm
by lmiltchev
Can you show us the actual check run from the command line along with the output of it, and the shell script itself? Please hide/obfuscate sensitive information.
Re: Nagios XI stderr question when running shell scripts...
Posted: Tue Feb 02, 2016 9:34 am
by JakeHatMacys
lmiltchev wrote:Can you show us the actual check run from the command line along with the output of it, and the shell script itself? Please hide/obfuscate sensitive information.
Gist of it is we're logging in with SSH, taking some metrics and then looking at file systems over thresholds of 95% in this version:
Code: Select all
#***********************************
#
#!/bin/bash
# Nagios return codes
STATE_OK=0
STATE_WARNING=1
STATE_CRITICAL=2
STATE_UNKNOWN=3
HOST=$1
USER="*******"
PASS="xxxxxxx"
COMMAND="iostat -c 5 2 | tr -s ' ' ';' | sed '/^$/d' | tail -1"
CPU_REPORT=`/usr/local/nagios/sshpass-1.05/sshpass -p$PASS ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 $USER@$HOST "$COMMAND"`
CPU_USER=`echo $CPU_REPORT | cut -d ";" -f 2`
CPU_SYSTEM=`echo $CPU_REPORT | cut -d ";" -f 4`
CPU_IOWAIT=`echo $CPU_REPORT | cut -d ";" -f 5`
CPU_IDLE=`echo $CPU_REPORT | cut -d ";" -f 7`
COMMAND2="df -v "
SPACE_REPORT=`/usr/local/nagios/sshpass-1.05/sshpass -p$PASS ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 $USER@$HOST "$COMMAND2"`
# Set defult to OK
msg_text="$HOST has a NO File systems that are full "
result=$STATE_OK
#
SPACE70=`echo $SPACE_REPORT | grep 95%`
if [ "$SPACE70" != "" ]; then
msg_text="CRITICAL $HOST has a File system at 95% full $SPACE70 "
result=$STATE_CRITICAL
fi
#
SPACE70=`echo $SPACE_REPORT | grep 96%`
if [ "$SPACE70" != "" ]; then
msg_text="CRITICAL $HOST has a File system at 96% full $SPACE70 "
result=$STATE_CRITICAL
fi
#
SPACE70=`echo $SPACE_REPORT | grep 97%`
if [ "$SPACE70" != "" ]; then
msg_text="CRITICAL $HOST has a File system at 97% full $SPACE70 "
result=$STATE_CRITICAL
fi
#
SPACE70=`echo $SPACE_REPORT | grep 98%`
if [ "$SPACE70" != "" ]; then
msg_text="CRITICAL $HOST has a File system at 98% full $SPACE70 "
result=$STATE_CRITICAL
fi
#
SPACE70=`echo $SPACE_REPORT | grep 99%`
if [ "$SPACE70" != "" ]; then
msg_text="CRITICAL $HOST has a File system at 99% full $SPACE70 "
result=$STATE_CRITICAL
fi
#
SPACE70=`echo $SPACE_REPORT | grep 100%`
if [ "$SPACE70" != "" ]; then
msg_text="CRITICAL $HOST has a File system at 100% full $SPACE70 "
result=$STATE_CRITICAL
fi
#
COMMAND="free -t -m | grep Total "
CHECK_MEM=`/usr/local/nagios/sshpass-1.05/sshpass -p$PASS ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 $USER@$HOST "$COMMAND"`
TOTAL_MEMORY=`echo $CHECK_MEM | cut -d " " -f 2`
USED_MEMORY=`echo $CHECK_MEM | cut -d " " -f 3`
FREE_MEMORY=`echo $CHECK_MEM | cut -d " " -f 4`
#
echo "$msg_text user=${CPU_USER}% system=${CPU_SYSTEM}% iowait=${CPU_IOWAIT}% idle=${CPU_IDLE}% Total_Memory=${TOTAL_MEMORY}MB Used_Memory=${USED_MEMORY}MB Free_Memory=${FREE_MEMORY}MB | user=${CPU_USER}% system=${CPU_SYSTEM}% iowait=${CPU_IOWAIT}% idle=${CPU_IDLE}% Total_Memory=${TOTAL_MEMORY}MB Used_Memory=${USED_MEMORY}MB Free_Memory=${FREE_MEMORY}MB "
exit $result
./
Problem is when we can't login to the box I don't know how to trigger an error to nagios and capture that access was denied:
Results of server I'm testing when logging in manually:
******* *******
** NOTE - This system is authenticating against Active Directory **
** Use your WINDOWS LAN password **
** BEWARE: 3 Wrong Guesses will LOCK you out **
******* *******
Using keyboard-interactive authentication.
Password:
Access denied
Using keyboard-interactive authentication.
Password:
When I run it from a command line on the server or GUI:
Capture.JPG
It basically just runs, takes no metrics or file system data and is green in Nagios. Thoughts on how I can get Nagios to recognize "Access Denied"???
Re: Nagios XI stderr question when running shell scripts...
Posted: Tue Feb 02, 2016 3:24 pm
by tmcdonald
Is there any particular reason you don't use the built-in check_by_ssh method?
https://assets.nagios.com/downloads/nag ... ng_SSH.pdf
Re: Nagios XI stderr question when running shell scripts...
Posted: Wed Feb 03, 2016 2:51 pm
by JakeHatMacys
If I remember correctly the security team shot that down pretty quickly, no access to the servers would be permitted without account credentials and the Unix team didn't want to manage installing anything on our remote machines. We had a lot of problems across all the different flavors of Unix we run in the past keeping agents up to date and working with other tools. Since then the shop standard has been to avoid anything like it at all costs.
Re: Nagios XI stderr question when running shell scripts...
Posted: Wed Feb 03, 2016 5:02 pm
by lmiltchev
Modifying a custom script is out of the scope of Nagios support. It's more in the realm of custom development.
Having said that, you could try using something like this:
Code: Select all
#!/bin/bash
mycommands="command1; command2; exit"
sshpass -p mypassword ssh -o StrictHostKeyChecking=no [email protected] $mycommands
if [ $? == 0 ]; then
echo "Files created"
exit 0
else
#Show a message & exit
echo "Error"
exit 2
fi
Disclaimer: my bash skills are quite limited. The above example worked for me but I haven't really try to do anything complicated.
You can also look at the "expect" scripts, which can be used in bash.
http://www.admin-magazine.com/Articles/ ... ct-Scripts
Hope this helps.