Nagios Command

raamardhani7 · Post by **raamardhani7** » Thu Apr 14, 2016 11:09 am

I have a file as shown below:
OK, mccdev, MCCD13X, ONLINE INCREMENTAL completed on 04/13/16 at 20.15.30 in 00.15.08 h.m.s.
OK, mccqa, MCCQ13X, ONLINE INCREMENTAL completed on 04/13/16 at 20.17.56 in 00.01.07 h.m.s.
OK, mccref, MCCR13X, ONLINE INCREMENTAL completed on 04/13/16 at 20.20.47 in 00.01.34 h.m.s.
OK, mcctrain, MCCT13X, ONLINE INCREMENTAL completed on 04/13/16 at 20.23.28 in 00.01.06 h.m.s.
OK, plmdev, PLMD14X, ONLINE INCREMENTAL completed on 04/13/16 at 20.27.59 in 00.02.52 h.m.s.
OK, plmqa, PLMQ14X, ONLINE INCREMENTAL completed on 04/13/16 at 20.32.30 in 00.03.06 h.m.s.
OK, plmref, PLMR14X, ONLINE INCREMENTAL completed on 04/13/16 at 20.35.26 in 00.01.22 h.m.s.
OK, plmsand, PLMS14X, ONLINE INCREMENTAL completed on 04/13/16 at 20.42.07 in 00.05.10 h.m.s.
OK, plmstag, STG14X, ONLINE INCREMENTAL completed on 04/13/16 at 20.47.34 in 00.04.06 h.m.s.
OK, plmtrain, PLMT14X, ONLINE INCREMENTAL completed on 04/13/16 at 21.02.04 in 00.12.39 h.m.s.

And, I'd like to generate Nagios alert for each above single line in my script.
Pls advise what command should be added into my script to generate alert for each line in a file.

Here is the command that I'm using on the command prompt:
/usr/local/nagios/libexec/check_nrpe -H $DB_HOSTNAME -t 30 -c check_backup.sh -a '$DB_NAME' -n

Best Regards

rkennedy · Post by **rkennedy** » Thu Apr 14, 2016 12:21 pm

Can you please post your script for us to look at?

raamardhani7 · Post by **raamardhani7** » Thu Apr 14, 2016 12:35 pm

Here is the script:

Code: Select all

#!/usr/bin/bash
###############################################################################
#         Title: DB2 Database backup Check
#        Author: Chris Park
#          Date: Apr. 13rd, 2016
# Example usage: ./check_Db2_backup.sh
#
# This script checks the Db2 database bavkup status.
#
# 1. Retrieves CRITICAL alarm when below status occurs:
#    * Backup instance is down
#    * Backup failed due to TSM-50 found
#    * Backup failed due to TMS4ACS mount failed
#    * Backup terminated
#
# 2. Retrieves WARNING alarm when below status occurs:
#    * Query failed
#    * Verify failed
#    * Verify failed due to DSM_Server was not set in the DSMI_Config
#
# 3. Retrieves OK alarm when below status occurs:
#    * Full/Incremental backup completed
#
###############################################################################
# Nagios return codes
FLAG=1
STATE_OK=0
STATE_WARNING=1
STATE_CRITICAL=2
STATE_UNKNOWN=3

PROGNAME=$(basename $0)

# Nagios backup condition
CRITICAL_1="Instance is DOWN"
CRITICAL_2="Backup failed"
CRITICAL_3="backup terminated"
WARNING_1="Query Failed"
WARNING_2="Verify Failed"

SVR_NAME=`hostname`
TMP_OUTPUT="/tmp/db2_backup_check.txt"
OUTPUTFILE="/tmp/db2_backup_check.$SVR_NAME"
NUM="/tmp/db2_backup_check.temp"

if [ -f $OUTPUTFILE ]
   then
       rm -f $OUTPUTFILE
fi

DATE_PRT=`TZ=EST+24 date +%b.%d.%Y`
YESTERDAY=`TZ=EST+24 date +%Y%m%d`
WORKDIR="/tmp/db2backup/backup/${YESTERDAY}*"

NODIR=`find $WORKDIR -type d | wc -l`
if [ $NODIR -gt 1 ]
   then
       CONDITION=`find $WORKDIR -type d`

       for i in "$CONDITION"
       do
          if [ -d "$i" ];
          then
              continue
          fi
       done
       echo "$i" > $NUM

       LINE1=$(cat $NUM | awk -F/ '{print $NF}'|head -n 1)
       LINE2=$(cat $NUM | awk -F/ '{print $NF}'|sed -n 2p)

       if [ $LINE1 -gt $LINE2 ]
          then
              WORKDIR="/tmp/db2backup/backup/${LINE1}"
              FLAG=0
       fi

       if [ $FLAG -eq 1 ]
          then
              WORKDIR="/tmp/db2backup/backup/${LINE2}"
       fi
fi

if [ ! -d $WORKDIR ]
   then
       TEMP_STATUS="CRITICAL"
       RETURN_STATUS=$STATE_CRITICAL
       echo "$TEMP_STATUS, $SVR_NAME, backup did NOT run last night on $DATE_PRT. Need to check..." > $OUTPUTFILE
       cat $OUTPUTFILE
       exit
fi

ls $WORKDIR/backup_results.txt > /dev/null 2>&1
if [[ $? -ne 0 ]]
   then
       TEMP_STATUS="CRITICAL"
       RETURN_STATUS=$STATE_CRITICAL
       echo "$TEMP_STATUS, $SVR_NAME, it looks terminated during the backup process... log file does NOT exist on `date +'%B %m. %Y'`."  >> $OUTPUTFILE
       cat $OUTPUTFILE
       exit
fi

egrep -e "^SUCCESS" $WORKDIR/backup_results.txt > $TMP_OUTPUT
egrep -e "$CRITICAL_1|$CRITICAL_2|$CRITICAL_3" $WORKDIR/backup_results.txt | sed -e 's/FAILURE :/CRITICAL :/g' -e 's/RC=[0-9]//g' -e 's/"//g'>> $TMP_OUTPUT
egrep -e "$WARNING_1|$WARNING_2" $WORKDIR/backup_results.txt | sed -e 's/FAILURE :/WARNING :/g' -e 's/Exit Code [0-9]//g' >> $TMP_OUTPUT

cat "$TMP_OUTPUT" | while read -r line
do
    STATUS=`echo $line | awk '{print $1}'`
    DB_INSTANCE=`echo $line | awk -F: '{print $2}' | sed -e 's/^.//' -e 's/.$//'`
    DB_HOSTNAME=`echo $line | awk -F: '{print $3}' | sed -e 's/^.//' -e 's/.$//'`
    DB_NAME=`echo $line | awk -F: '{print $4}' | sed -e 's/^.//' -e 's/.$//'`
    DESCRIPTION=`echo $line | awk -F: '{print $NF}' | sed 's/^.//'`

    case "$STATUS" in
         "SUCCESS")
                TEMP_STATUS="OK"
                RETURN_STATUS=$STATE_OK
                echo "$TEMP_STATUS, ${DB_INSTANCE}, ${DB_NAME}, $DESCRIPTION" >> $OUTPUTFILE

############ Do I really need another script? ###############################
/usr/local/nagios/libexec/check_nrpe -H $DB_HOSTNAME -t 30 -c check_Db2_backup_instance.sh -n
#############################################################################
                ;;

         "CRITICAL")
                TEMP_STATUS="CRITICAL"
                RETURN_STATUS=$STATE_CRITICAL
                VALUE=$(echo $DESCRIPTION|grep -c "Instance is DOWN")
                if [ $VALUE -eq 1 ]
                   then
                       echo "$TEMP_STATUS, ${DB_INSTANCE}, ${DB_NAME}, $DESCRIPTION" >> $OUTPUTFILE
                fi

                VALUE=$(echo $DESCRIPTION|grep -c "Backup failed")
                if [ $VALUE -eq 1 ]
                   then
                       echo "$TEMP_STATUS, ${DB_INSTANCE}, ${DB_NAME}, $DESCRIPTION" >> $OUTPUTFILE
                fi

                VALUE=$(echo $DESCRIPTION|grep -c "backup terminated")
                if [ $VALUE -eq 1 ]
                   then
                       echo "$TEMP_STATUS, ${DB_INSTANCE}, ${DB_NAME}, $DESCRIPTION" >> $OUTPUTFILE
                fi

                ;;

         "WARNING")
                TEMP_STATUS="WARNING"
                RETURN_STATUS=$STATE_WARNING
                VALUE=$(echo $DESCRIPTION|grep -c "Query Failed")
                if [ $VALUE -eq 1 ]
                   then
                       echo "$TEMP_STATUS, ${DB_INSTANCE}, ${DB_NAME}, $DESCRIPTION" >> $OUTPUTFILE
                fi

                VALUE=$(echo $DESCRIPTION|grep -c "Verify Failed")
                if [ $VALUE -eq 1 ]
                   then
                       echo "$TEMP_STATUS, ${DB_INSTANCE}, ${DB_NAME}, $DESCRIPTION" >> $OUTPUTFILE
                fi
                ;;

               *)
                TEMP_STATUS="UNKNOWN"
                RETURN_STATUS=$STATE_UNKNOWN
                echo "$TEMP_STATUS, No status found" >> $OUTPUTFILE
                ;;
    esac
done

cat $OUTPUTFILE

if [ -f $TMP_OUTPUT ]
   then
       rm -f $TMP_OUTPUT; rm -f $NUM
fi

#echo $FINAL_STATUS
#exit $RETURN_STATUS

Here is the output for the above script:

Code: Select all

OK,  mccdev,  MCCD13X,  ONLINE INCREMENTAL completed on 04/13/16 at 20.15.30 in 00.15.08 h.m.s.
OK,  mccqa,  MCCQ13X,  ONLINE INCREMENTAL completed on 04/13/16 at 20.17.56 in 00.01.07 h.m.s.
OK,  mccref,  MCCR13X,  ONLINE INCREMENTAL completed on 04/13/16 at 20.20.47 in 00.01.34 h.m.s.
OK,  mcctrain,  MCCT13X,  ONLINE INCREMENTAL completed on 04/13/16 at 20.23.28 in 00.01.06 h.m.s.
OK,  plmdev,  PLMD14X,  ONLINE INCREMENTAL completed on 04/13/16 at 20.27.59 in 00.02.52 h.m.s.
OK,  plmqa,  PLMQ14X,  ONLINE INCREMENTAL completed on 04/13/16 at 20.32.30 in 00.03.06 h.m.s.
OK,  plmref,  PLMR14X,  ONLINE INCREMENTAL completed on 04/13/16 at 20.35.26 in 00.01.22 h.m.s.
OK,  plmsand,  PLMS14X,  ONLINE INCREMENTAL completed on 04/13/16 at 20.42.07 in 00.05.10 h.m.s.
OK,  plmstag,  STG14X,  ONLINE INCREMENTAL completed on 04/13/16 at 20.47.34 in 00.04.06 h.m.s.
OK,  plmtrain,  PLMT14X,  ONLINE INCREMENTAL completed on 04/13/16 at 21.02.04 in 00.12.39 h.m.s.

And, below script is sub script, which calls from above script. However, I really don't think that I need this.

Code: Select all

#!/usr/bin/bash
###############################################
# Example usage: ./check_Db2_backup_instance.sh
###############################################
STATE_OK=0
STATE_WARNING=1
STATE_CRITICAL=2
STATE_UNKNOWN=3

SVR_NAME=`hostname`
OUTPUTFILE="/tmp/db2_backup_check.$SVR_NAME"

cat "$OUTPUTFILE" | while read -r line
do
    FINAL_STATUS=`echo $line|awk -F, '{print $1}'`
    FINAL_INSTANCE=`echo $line |awk -F, '{print $2}'`
    FINAL_DB=`echo $line|awk -F, '{print $3}'`
    FINAL_DESC=`echo $line|awk -F, '{print $NF}'`

    case "$FINAL_STATUS" in
         "OK")
                RETURN_STATUS=$STATE_OK
                echo "$FINAL_STATUS, ${FINAL_INSTANCE}, ${FINAL_DB}, $FINAL_DESC"
                ;;

         "CRITICAL")
                RETURN_STATUS=$STATE_CRITICAL
                echo "$FINAL_STATUS, ${FINAL_INSTANCE}, ${FINAL_DB}, $FINAL_DESC"
                ;;

         "WARNING")
                RETURN_STATUS=$STATE_WARNING
                echo "$FINAL_STATUS, ${FINAL_INSTANCE}, ${FINAL_DB}, $FINAL_DESC"
                ;;

         "UNKNOWN")
                RETURN_STATUS=$STATE_UNKNOWN
                echo "$FINAL_STATUS, ${FINAL_INSTANCE}, ${FINAL_DB}, $FINAL_DESC"
                ;;
    esac
done

tmcdonald · Post by **tmcdonald** » Thu Apr 14, 2016 2:43 pm

It won't work that way. The plugin you are writing is going to be run against that file, and Core/XI will look at the results and determine if an alert needs to be sent. The way around this is to run this as a passive check on a cron, and write directly to the nagios.cmd file to generate an alert for each match:

http://old.nagios.org/developerinfo/ext ... ndlist.php

probably with SEND_CUSTOM_SVC_NOTIFICATION:

http://old.nagios.org/developerinfo/ext ... and_id=135

But this is somewhat circumventing the normal check logic.

Post by **eloyd** » Thu Apr 14, 2016 2:51 pm

Also, the assumption is that this is on your local Nagios server, which I believe is the case. If you need to do this on a remote box, you also need to use some tool (NRDP would be my preferred method) of sending the results back to your Nagios server.

ssax · Post by **ssax** » Thu Apr 14, 2016 4:41 pm

raamardhani7, take a look through the docs that tmcdonald posted and let us know if you have any questions.

Thank you

raamardhani7 · Post by **raamardhani7** » Fri Apr 15, 2016 8:14 am

Thank you for the prompt response, everyone.

Current data is sitting on the remote server, So, can I also run SEND_CUSTOM_SVC_NOTIFICATION command from the remote server?

Best Regards.

Post by **eloyd** » Fri Apr 15, 2016 8:16 am

As I said,

If you need to do this on a remote box, you also need to use some tool (NRDP would be my preferred method) of sending the results back to your Nagios server.

raamardhani7 · Post by **raamardhani7** » Fri Apr 15, 2016 8:25 am

Thanks, eloyd
Best Regards

Post by **eloyd** » Fri Apr 15, 2016 8:27 am

NRDP is nice because it's built in to XI and if you read the documentation, you'll find how to use a really easy PHP script that lets you send results back to Nagios XI from a remote server.

Nagios Support Forum

Nagios Command

Nagios Command

Re: Nagios Command

Re: Nagios Command

Re: Nagios Command

Re: Nagios Command

Re: Nagios Command

Re: Nagios Command

Re: Nagios Command

Re: Nagios Command

Re: Nagios Command