No Output Returned From Plugin

[nagios4krish]

I am running core server 4.3.4 on RHEL5.11 and use bespoke plugins for monitoring our database environments.

One particular plugin keeps on reporting "No Output Returned From Plugin" on the dashboard but the second runs clears that out. Manual run of the script runs OK. Below is the service definition we use.

define service {
use generic-service
host_name
servicegroups App and DB
service_description test: DB GC CR Request Time
check_command check_gc_cr_request_time! -i TESTDB
check_interval 2
max_check_attempts 5
retry_interval 2
}

The script is designed to compute some stats based on 2 different snapshots taken 60sec apart. So the overall runtime is around 90sec. I have the following global settings.

cat nagios.cfg |grep interval_length
interval_length=60

cat nagios.cfg |grep timeout
service_check_timeout=120

Any help or suggestion would be highly appreciated.

Regards,
Krish

[gormank]

Is the return value what you expect? No output means nothing was sent to stdout, not that there was necessarily a failure. I'd guess things are timing out.
Why cat file | grep string rather than grep string file? The latter is easier to type...

[ssax]

@gormank is correct, what you may want to do is try editing the command and add this onto the end of it to redirect stderr to stdout as well to see if that gives you any additional details:

Code: Select all

 2>&1

If the plugin isn't returning any output and you're not seeing a timeout message then it's likely something in the plugin that is failing on the first time (or just initializing a seek file or something depending on what you are doing). For example, when you run check_log3.pl the first time to check for text in a log file it won't show anything on the 1st run because it's creating a seek file and updating where it read to, it will work from the 2nd run onwards though.

Can you PM me your plugin so I can see what it actually does?

[nagios4krish]

The plugin is as below.

Code: Select all

#!/bin/bash

# Loading Database env vars
# ------------------------------
. /usr/local/nagios/.db_env
# ------------------------------

VERBOSE=0

usage() {
cat << EOF
usage: $0 -h  -u  -p  [-i ] [-s] [-v]

This script connects to the specified Oracle instance and executes a simple
statement. If that statement succeeds, the script returns 0.

OPTIONS:
   -h      Specify the host (required)
   -u      Oracle username (required)
   -p      Oracle password for the user (required)
   -i      SID of the instance (default: test01)
   -s      Force login AS SYSDBA
   -v      Verbose
EOF
}


while getopts "u:p:i:h:vs" OPTION; do
        case $OPTION in
                u)
                        ORACLE_USERNAME=$OPTARG
                        ;;
                p)
                        ORACLE_PASSWORD=$OPTARG
                        ;;
                i)
                        INSTANCE=$OPTARG
                        ;;
                v)
                        VERBOSE=1
                        ;;
                h)
                        HOST=$OPTARG
                        ;;
                s)
                        ASSYSDBA=1
                        ;;
                ?)
                        usage
                        exit 1
                        ;;
        esac
done

if [ -z "$INSTANCE" ]; then
                echo "You must specify an Oracle instance or DB Service (-i)!"
                usage
                exit 1
fi

CONNECT_STRING=$(echo /@\"$INSTANCE\")

VAR_GC_CR_REQ_TIME=`sqlplus -S "$CONNECT_STRING" << EOF
SET HEADING OFF
SET SERVEROUTPUT ON;
SET FEEDBACK OFF;
undef sleep
declare
        type number_table is table of number index by binary_integer;
        b_inst_id  number_table ;
        b_cr_tm_recv    number_table ;
        b_cr_blks_recv  number_table ;
        
        e_inst_id  number_table ;
        e_cr_blks_recv  number_table ;
        e_cr_tm_recv    number_table ;

        v_ver number;
        v_tot_instances number;
        v_dbname varchar2(5 char);
        v_inst_id number;
        v_instance_name varchar2(10 char);

        v_cr_blks_serv  varchar2(256); 
        v_cr_blks_recv    varchar2(256);
        v_cur_blks_serv  varchar2(256); 
        v_cur_blks_recv   varchar2(256);
        v_cr_blks_rcv_time varchar2(256);
        v_cur_blks_rcv_time varchar2(256);
    
        l_sleep number;

        v_gc_cr_req_time_calc float;

begin
          select count(*) into v_tot_instances from sys.gv_\\\$instance;
          V_DBNAME := UPPER(SYS_CONTEXT('USERENV','DB_NAME'));

          v_cr_blks_serv :='gc cr blocks served';
          v_cr_blks_recv := 'gc cr blocks received';
          v_cur_blks_serv := 'gc current blocks served';
          v_cur_blks_recv := 'gc current blocks received';
          v_cr_blks_rcv_time :='gc cr block receive time';
          v_cur_blks_rcv_time := 'gc current block receive time';

          select 
                evt_cr_recv.inst_id,
                evt_cr_tm.value cr_tm_recv,
                evt_cr_recv.value cr_blks_recv
                bulk collect into 
                b_inst_id, b_cr_tm_recv,b_cr_blks_recv
          from 
                sys.gv_\\\$sysstat evt_cr_tm, 
                sys.gv_\\\$sysstat evt_cr_recv
          where
                evt_cr_tm.name = v_cr_blks_rcv_time
                and evt_cr_recv.name = v_cr_blks_recv
                and evt_cr_tm.inst_id = evt_cr_recv.inst_id
                order by inst_id
                ;
          

         sys.dbms_lock.sleep(60);
          

         select 
                evt_cr_recv.inst_id,
                evt_cr_tm.value cr_tm_recv,
                evt_cr_recv.value cr_blks_recv
         bulk collect into 
                e_inst_id,e_cr_tm_recv,e_cr_blks_recv 
         from 
                sys.gv_\\\$sysstat evt_cr_tm, 
                sys.gv_\\\$sysstat evt_cr_recv                
         where
                    evt_cr_tm.name = v_cr_blks_rcv_time
                and evt_cr_recv.name = v_cr_blks_recv
                and evt_cr_tm.inst_id = evt_cr_recv.inst_id
                order by inst_id
                ;
          

          FOR i IN  1 .. v_tot_instances
          LOOP
              V_INSTANCE_NAME  := CONCAT(V_DBNAME,i);
                           
              IF (e_cr_blks_recv (i) - b_cr_blks_recv (i)) >0 THEN
                  --V_GC_CR_REQ_TIME_CALC := lpad(to_char(round(10*( e_cr_tm_recv(i) - b_cr_tm_recv(i) )/(e_cr_blks_recv (i) - b_cr_blks_recv (i)), 2)),9);
                  --V_GC_CR_REQ_TIME_CALC := round(10*((e_cr_tm_recv(i) - b_cr_tm_recv(i))/(e_cr_blks_recv (i) - b_cr_blks_recv (i))), 0);
                  V_GC_CR_REQ_TIME_CALC := trunc((10*((e_cr_tm_recv(i) - b_cr_tm_recv(i))/(e_cr_blks_recv (i) - b_cr_blks_recv (i)))),2);
                  --V_GC_CR_REQ_TIME_CALC := 10*((e_cr_tm_recv(i) - b_cr_tm_recv(i))/(e_cr_blks_recv (i) - b_cr_blks_recv (i)));
 
                  IF V_GC_CR_REQ_TIME_CALC >= 2  THEN
                                DBMS_OUTPUT.PUT_LINE('criticalNagiosFlag Average GC CR Request Time for '||V_DBNAME||':'||V_INSTANCE_NAME||' is '||V_GC_CR_REQ_TIME_CALC||'ms.');
                  ELSIF V_GC_CR_REQ_TIME_CALC >= 1 AND V_GC_CR_REQ_TIME_CALC < 2 THEN
                                DBMS_OUTPUT.PUT_LINE('warningNagiosFlag Average GC CR Request Time for '||V_DBNAME||':'||V_INSTANCE_NAME||' is '||V_GC_CR_REQ_TIME_CALC||'ms.');
                  ELSIF V_GC_CR_REQ_TIME_CALC < 1 THEN 
                                DBMS_OUTPUT.PUT_LINE('okNagiosFlag Average GC CR Request Time for '||V_DBNAME||':'||V_INSTANCE_NAME||' is '||V_GC_CR_REQ_TIME_CALC||'ms.'); 
                  ELSE        
                                DBMS_OUTPUT.PUT_LINE('warningNagiosFlag Error while checking Average GC CR Request Time for '||V_DBNAME||':'||V_INSTANCE_NAME||'.');
                  END IF;
             ELSE
                   V_GC_CR_REQ_TIME_CALC := lpad('0',9);
                                   
                   IF V_GC_CR_REQ_TIME_CALC > 2  THEN
                                DBMS_OUTPUT.PUT_LINE('criticalNagiosFlag Average GC CR Request Time for '||V_DBNAME||':'||V_INSTANCE_NAME||' is '||V_GC_CR_REQ_TIME_CALC||'ms.');
                   ELSIF V_GC_CR_REQ_TIME_CALC >= 1 AND V_GC_CR_REQ_TIME_CALC < 2 THEN
                                DBMS_OUTPUT.PUT_LINE('warningNagiosFlag Average GC CR Request Time for '||V_DBNAME||':'||V_INSTANCE_NAME||' is '||V_GC_CR_REQ_TIME_CALC||'ms.');
                   ELSIF V_GC_CR_REQ_TIME_CALC < 1 THEN 
                                DBMS_OUTPUT.PUT_LINE('okNagiosFlag Average GC CR Request Time for '||V_DBNAME||':'||V_INSTANCE_NAME||' is '||V_GC_CR_REQ_TIME_CALC||'ms.'); 
                   ELSE        
                                DBMS_OUTPUT.PUT_LINE('warningNagiosFlag Error while checking Average GC CR Request Time for '||V_DBNAME||':'||V_INSTANCE_NAME||'.');
                   END IF;
             END IF;  
           END LOOP;
END;
/

exit;
EOF`

VAR_CR_GET_TIME_CRITICAL_EXIT=`echo $VAR_GC_CR_REQ_TIME | grep criticalNagiosFlag | wc -l`;
VAR_CR_GET_TIME_WARNING_EXIT=`echo $VAR_GC_CR_REQ_TIME | grep warningNagiosFlag | wc -l`;
VAR_CR_GET_TIME_OK_EXIT=`echo $VAR_GC_CR_REQ_TIME | grep okNagiosFlag | wc -l`;

if [ ${VAR_CR_GET_TIME_CRITICAL_EXIT} -ge 1 ] ; then
    VAR_GC_CR_REQ_TIME=`echo "$VAR_GC_CR_REQ_TIME" | grep criticalNagiosFlag`;
    echo $VAR_GC_CR_TIME | sed -e "s/criticalNagiosFlag //g" ;
    exit 2
elif [ ${VAR_CR_GET_TIME_WARNING_EXIT} -ge 1 ] ; then
    VAR_GC_CR_REQ_TIME=`echo "$VAR_GC_CR_REQ_TIME" | grep warningNagiosFlag`;
    echo $VAR_GC_CR_TIME | sed -e "s/warningNagiosFlag //g" ;
    exit 1    
elif [ ${VAR_CR_GET_TIME_OK_EXIT} -ge 1 ] ; then
    echo $VAR_GC_CR_REQ_TIME | sed -e "s/okNagiosFlag //g" ;
    exit 0 
else 
    echo $VAR_GC_CR_REQ_TIME ;
    exit 1
fi

[nagios4krish]

Yes when I run the bash script manually the return value is 0 which is a success and as expected. So yes there is no obvious failure.