No output returned from plugin status

[elinagios]

Hello

i have a question about a script and its result view. It shows correct info on OK status but when its critical, then status information is not showing most of the times.

script: -rwxrwxr-x 1 apache nagios 472 Apr 26 2017 check_cmd.sh

When i run from cli:
./check_cmd.sh someip
ERROR - test error

When i run from service (run check command)
[nagios@mynagios ~]$ /usr/local/nagios/libexec/check_cmd.sh someip
ERROR - test error

From nagios main web in critical state (service detail) i get (No output returned from plugin) in status information. And here is the funny thing, once a while (not often), the status information shows the same result as CLI. When the status is OK, then the status information gives correct result.

Any ideas what to check?

Thank you!

[scottwilkerson]

Can you share the contents of check_cmd.sh

Also, can you show what you have in the check command and arguments for the service you are putting this on?

[elinagios]

Script:
#!/bin/bash

HOST=$1
RESPONSE=`ssh nagios@$HOST /home/user/script.sh`
STATUS=$?

if [ $STATUS -eq 0 ]; then
echo "Server is OK"
exit 0
elif [ $STATUS -eq 2 ]; then
#error
echo "$RESPONSE"
exit 2
elif [ $STATUS -eq 3 ]; then
#unknown/overload
echo "$RESPONSE"
exit 3
fi

echo "Script failed - unknown error"
exit 3


Check command:
$USER1$/check_cmd.sh $HOSTADDRESS$

remote /home/user/script.sh is basically checking for a file content, if its content is OK, then it says OK and exitcode 0, if content not OK, then print the content and exitcode 2, other instances unknown error and exitcode3.

[scottwilkerson]

Based on how this is setup I could see one possible scenerio that would give you the results you are getting

If the value of $RESPONSE doesn't contain any chars on the first line, you would get the error "No output returned from plugin"

Nagios requires the first line of text returned from a plugin to be a non empty string.

[elinagios]

Sadly its not the case, tested the file with 1 line in it:
ERROR - test error

The cli got correct info, also from services when you run "run check command" and there were rare cases when status info was shown correctly in service status information for critical state. OK status information is always printed correctly.

[scottwilkerson]

Well, somehow this command is coming back sometimes with non-0 exit codes

Code: Select all

ssh nagios@$HOST /home/user/script.sh

can you post the full contents of /home/user/script.sh ?

[elinagios]

Not sure the status code could differ because it always gets the correct status, either OK or Critical, and cli always shows the correct status information value.
Requested script:
#!/bin/bash

LENGTH_LIMIT=256

CURRENT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
HOST="$(hostname)"
RESULT_FILE="$CURRENT_DIR/$HOST-monitoring-output.txt"
LOG_FILE="/home/nagios/monitoring.$(date "+%Y%m%d").log"

if [ ! -f $LOG_FILE ]; then
touch $LOG_FILE
fi

log () {
echo "$(date "+%Y-%m-%d %H:%M:%S") - $1" >> $LOG_FILE
}

if [ ! -f $RESULT_FILE ]; then
log "MONITORING FILE NOT FOUND"
echo "MONITORING FILE NOT FOUND"
exit 3
fi

MESSAGE=$(tail -n 1 $RESULT_FILE)

if [ "$MESSAGE" = "OK" ]; then
log "returning status code 0, message:$MESSAGE"
echo "$MESSAGE"
exit 0
else
if [ "${#MESSAGE}" -gt $LENGTH_LIMIT ]; then
log "Truncating excessively long message from ${#MESSAGE} characters to $LENGTH_LIMIT"
MESSAGE=${MESSAGE:0:$LENGTH_LIMIT}
fi
log "returning status code 2, message: $MESSAGE"
echo "$MESSAGE"
exit 2
fi

log "??? Unown exit ???, message: $MESSAGE"
exit 3

[scottwilkerson]

I wonder if it is possible that the ssh session is erroring

What if you replace this

Code: Select all

RESPONSE=`ssh nagios@$HOST /home/user/script.sh`

with this

Code: Select all

RESPONSE=`ssh nagios@$HOST /home/user/script.sh 2>&1`

This should redirect any error output to $RESPONSE and then you could see the error if it happens

[elinagios]

Sadly it didnt make any difference. It understands that the status needs to be changed, it changes from OK status to Critical, but status information posts no output returned. When i changed back to OK, it understands the status needs to be changed from critical to ok and shows correct status information. From CLI i get the correct result message each time.

[scottwilkerson]

What lines are you getting in LOG_FILE when you do not get the correct output from nagios?