Service returning longoutput

[jwelch]

I forgot this check used NRDS. I don't use it so I don't know if it's capable of accepting multiline output.
I suspect it is (older versions may not), but don't see anything obvious to back that up.
I'd suggest configuring an NRDS check using a standard plugin and and if that works run it
manually and look at the output format.

Since this is passive check data going directly into a checkresult file, it probably requires a single line/string
with newlines (\n) as delimiters:
print "cond - sum|hung=1\nThis is detail line 1.\nThis is detail line 2.\nThis is detail line 3.";

[dangladesh]

Is this happening only with NDRS checks?

Have you tired creating a simple bash script (or similar) to run locally to test?

Currently, I am only using passive checks with NRDS. You mean run a bash script locally on the Nagios server?

I forgot this check used NRDS. I don't use it so I don't know if it's capable of accepting multiline output.
I suspect it is (older versions may not), but don't see anything obvious to back that up.
I'd suggest configuring an NRDS check using a standard plugin and and if that works run it
manually and look at the output format.

Since this is passive check data going directly into a checkresult file, it probably requires a single line/string
with newlines (\n) as delimiters:
print "cond - sum|hung=1\nThis is detail line 1.\nThis is detail line 2.\nThis is detail line 3.";

Thanks jwelch. I haven't tried getting multiline output from a standard plugin. I'll try that. I will also try condensing everything down to one line within my custom plugins.

[Box293]

You mean run a bash script locally on the Nagios server?

Yes I was just thinking that we could determine if it is only happening with NRDS checks.

[jwelch]

Here is a (relatively) easy way to test multiline output using a normal nagios check script. (not NRDS)
Create the following in /usr/local/nagios/libexec/check_multiline.pl

Code: Select all

#!/usr/bin/perl
#
#
use strict;

my %ERRORS=('OK'=>0,'WARNING'=>1,'CRITICAL'=>2,'UNKNOWN'=>3,'DEPENDENT'=>4);

my $cond;
my $sum;
my $perfdata;
my @data;

$cond="OK";
$sum="multiline nagios test";
$perfdata="a=0;1;2;0;10 b=1;3;5;0;10 c=2;3;4;0;10";
my @data;
push(@data,"This is line 1.");
push(@data,"This is line 2.");
push(@data,"This is line 3.");

print "$cond - $sum|$perfdata\n";
print "<pre>";
foreach my $line (@data) {
 print "$line\n";
}
print "</pre>";
exit $ERRORS{$cond};

Go into the Nagios XI Core Configuration Manager (CCM) and create a command called 'check_multiline' that runs
$USER1$/check_multiline.pl

Create a new service called 'Check_Multiline' that uses the 'check_multiline' command, apply the configuration, then go to the new service and hit 'Schedule a forced immediate check'

You should see something like:

check_multiline.png

[dangladesh]

Looks like multiline is working when the check is run directly on the Nagios Server (and HTML is escaped properly).

nagios_screen - multiline test.png

[dangladesh]

I have crated a test NRDS plugin that only outputs a multiline response to the Nagios Server.

Code: Select all

#!/usr/bin/ksh
#Nagios Plugin - check_multiline.sh
#     Tests multiline output

echo "MULTILINE OK - Testing Multiline output|a=0;1;2;0;10 b=1;3;5;0;10 c=2;3;4;0;10\nThis is line 1\nThis is line 2\nThis is line 3"
exit 0

Here is the output when run from command line on the client:

nagios_screen - multiline test3.png

Here is the service check in the Nagios Server:

nagios_screen - multiline test2.png

Unfortunately, multiline output isn't registered. Perfdata was successful. It seems to be something with the NRDS causing multiline output to be dropped.

[dangladesh]

I have done some digging. It looks like send_nrdp.sh is piped a tab delimited line from each service check. Formatted like the following:

Code: Select all

Format:  <config> \t <service> \t <state> \t <output>
Example: host.domain.com       Uptime     0     UPTIME OK - up for 10 day(s) | upDays=10

If there is a newline in the <output>, the content after that newline is treated as another service check. Since the line is without tabs, it is interpreted as an improperly formatted service check (with one field instead of 3 or 4) and triggers the error I mentioned in an earlier post.

This might be useful. I seem to get the following error in my var/mail directory from the cron job when multiple lines are returned by a service:

Code: Select all

/opt/nagios/nrdp/clients/nrds/nrds.pl -H 'host.domain.com' 2>&1

produced the following output:

ERROR: STDIN must be either 3 or 4 fields long, I found 1
ERROR: STDIN must be either 3 or 4 fields long, I found 1
Sent 25 checks to http://xxx.xxx.xxx.xxx/nrdp/

Here are the lines of code that deal with and output this error in the send_nrdp.sh file:

Code: Select all

205     while read -r line ; do
206         arr=($line)
207         if [ ${#arr[@]} != 0 ];then
208             if [[ ${#arr[@]} < 3 ]] || [[ ${#arr[@]} > 4 ]];then
209                 echo "ERROR: STDIN must be either 3 or 4 fields long, I found "${#arr[@]}
210             else
211                 if [ ${#arr[@]} == 4 ]; then
212                     xml="$xml<checkresult type='service' checktype='$checktype'>
213                     <servicename>${arr[1]}</servicename>
214                     <hostname>${arr[0]}</hostname>
215                     <state>${arr[2]}</state>
216                     <output>${arr[3]}</output>"
217                 else
218                     xml="$xml<checkresult type='host' checktype='$checktype'>
219                     <hostname>${arr[0]}</hostname>
220                     <state>${arr[1]}</state>
221                     <output>${arr[2]}</output>"
222                 fi
223
224                 xml="$xml</checkresult>"
225                 checkcount=$[checkcount+1]
226             fi
227         fi
228     done

I am working on a solution to this issue, but am open to any suggestions also.

[dangladesh]

Alright! I've got it working. The solution is to double escape the newline characters in the NRDS plugin outputs.

The nrds.pl script runs the commands in the nrds.cfg file and concatenates their output to a string, $senddata, ending in a \n.

Code: Select all

 86 my $senddata = "";
 87 for( my $x = 0; $x < @{ $config->{ "commands"}}; $x++) {
 88     my $cmd = (( $config->{ "COMMAND_PREFIX"} eq "") ? "" :
 89             $config->{ "COMMAND_PREFIX"} . " ") .
 90             $config->{ "commands"}->[ $x]->{ "command"};
 91     my $output = `$cmd`;
 92     my $status = $?;
 93     $status >>= 8 if( $status);
 94     if( $status > 3) {
 95         $output = "Error code $status - check plugin";
 96         $status = 3;
 97     }
 98     if( $config->{ "commands"}->[ $x]->{ "service"} eq "__HOST__") {
 99         $senddata .= "$hostname\t$status\t$output\n";
100     }
101     else {
102         $senddata .= "$hostname\t" .
103                 $config->{ "commands"}->[ $x]->{ "service"} .
104                 "\t$status\t$output\n";
105     }
106 }

The first escape will leave another escape on these characters to be sent to send_nrdp.sh. This allows the read to also escape the newlines.

Code: Select all

205     while read -r line ; do
206         echo $line
207         arr=($line)
208         if [ ${#arr[@]} != 0 ];then
209             if [[ ${#arr[@]} < 3 ]] || [[ ${#arr[@]} > 4 ]];then
210                 echo "ERROR: STDIN must be either 3 or 4 fields long, I found "${#arr[@]}
211             else
212                 if [ ${#arr[@]} == 4 ]; then
213                     xml="$xml<checkresult type='service' checktype='$checktype'>
214                     <servicename>${arr[1]}</servicename>
215                     <hostname>${arr[0]}</hostname>
216                     <state>${arr[2]}</state>
217                     <output>${arr[3]}</output>"
218                 else
219                     xml="$xml<checkresult type='host' checktype='$checktype'>
220                     <hostname>${arr[0]}</hostname>
221                     <state>${arr[1]}</state>
222                     <output>${arr[2]}</output>"
223                 fi
224
225                 xml="$xml</checkresult>"
226                 checkcount=$[checkcount+1]
227             fi
228         fi
229     done

The end result is properly escaped newlines, allowing the Nagios Server to receive multiline output. Example plugin and result below:

Code: Select all

  1 #!/usr/bin/ksh
  2
  3 #Nagios Plugin - check_multiline.sh
  4 #     Tests multiline output
  5
  6 echo "MULTILINE OK - Testing Multiline output|a=0;1;2;0;10 b=1;3;5;0;10 c=2;3;4;0;10\\\\nThis is line 1\\\\nThis is line 2\\\\n    This is line 3"
  7 exit 0

nagios_screen - multiline test4.png

[Box293]

This is really helpful information, I'm glad you got to the bottom of it.