nrdp not processing all data

[BanditBBS]

So I have a weird NRDP issue. I have NRDP installed on an appliance that runs every 5 minutes and sends 14 check results.

Code: Select all

hdfs-chi-usm01:/tmp# /usr/local/nrdp/clients/nrds/nrds.pl -H 'hdfs-chi-usm01'
Sent 14 checks to https://xxxx.xxxx.com/nrdp/
hdfs-chi-usm01:/tmp# 

As you can see in this image the host plus 13 services highlighted are what is sent:

Capture1.PNG

However, look at the Last Check column. Quite often that "Check ossec-syscheckd" doesn't report in, even though it is sent. It is the last line in the config. Every so often another 1 or two services don't get updated. Here is the config:

Code: Select all

hdfs-chi-usm01:/tmp# cat /usr/local/nrdp/clients/nrds/nrds.cfg
CONFIG_VERSION="5.3"
CONFIG_OS="Linux"
CONFIG_NAME="hdfs_usm"
URL="https://xxxxxx.xxxxx.com/nrdp/"
TOKEN="xxxxxxxxxx"
PLUGIN_DIR="/usr/local/nagios/libexec"
SEND_NRDP="/usr/local/nrdp/clients/send_nrdp.sh"
TMPDIR="/usr/local/nrdp/clients/tmp"
COMMAND_PREFIX=""
LOG_FILE=""
UPDATE_CONFIG="1"
UPDATE_PLUGINS="1"


command[__HOST__]=/usr/local/nagios/libexec/check_ping -H localhost -w 3000.0,80% -c 5000.0,100% -p 1
command[Check Load]=/usr/local/nagios/libexec/check_load -w 23,18,13 -c 27,22,17
command[Check Disk]=/usr/local/nagios/libexec/check_disk -w 20% -c 10%
command[Check CPU]=/usr/local/nagios/libexec/check_cpu_stats.sh -w 90,85,60 -c 95,95,70
command[Check Memory]=/usr/local/nagios/libexec/check_mem.pl -w 5 -c 1 -f -C
command[Check Sensor]=/usr/local/nagios/libexec/check_ping -H 10.67.69.10 -w 3000.0,80% -c 5000.0,100% -p 1
command[Check Swap]=/usr/local/nagios/libexec/check_swap -w 50% -c 30%
command[Check Update URL]=/usr/local/nagios/libexec/check_xx_curl_response.sh 10 http://data.alienvault.com
command[Check Internet]=/usr/local/nagios/libexec/check_xx_curl_response.sh 10 http://www.google.com
command[Check OTX URL]=/usr/local/nagios/libexec/check_xx_curl_response.sh 10 https://otx.alienvault.com
command[Check ossec-remoted]=/usr/local/nagios/libexec/check_procs_usm -C ossec-remoted -w 1: -c 1:
command[Check ossec-logcollector]=/usr/local/nagios/libexec/check_procs_usm -a ossec-logcollector -w 1: -c 1:
command[Check ossec-analysisd]=/usr/local/nagios/libexec/check_procs_usm -C ossec-analysisd -w 1: -c 1:
command[Check ossec-syscheckd]=/usr/local/nagios/libexec/check_procs_usm -C ossec-syscheckd -w 1: -c 1:

Then, I added a second config to this machine and have it running 1 time per day in cron:

Code: Select all

hdfs-chi-usm01:/tmp# /usr/local/nrdp/clients/nrds/nrds.pl -H 'hdfs-chi-usm01' -c /usr/local/nrdp/clients/nrds/nrds2.cfg
Sent 1 checks to https://xxxx.xxxx.com/nrdp/
hdfs-chi-usm01:/tmp#

Here is that config:

Code: Select all

hdfs-chi-usm01:/tmp# cat /usr/local/nrdp/clients/nrds/nrds2.cfg
CONFIG_VERSION="0.5"
CONFIG_OS="Linux"
CONFIG_NAME="hdfs_chi_usm_logs"
URL="https://xxxx.xxxx.com/nrdp/"
TOKEN="sifo85kli9uh"
PLUGIN_DIR="/usr/local/nagios/libexec"
SEND_NRDP="/usr/local/nrdp/clients/send_nrdp.sh"
TMPDIR="/usr/local/nrdp/clients/tmp"
COMMAND_PREFIX=""
LOG_FILE="/tmp/nagios.log"
UPDATE_CONFIG="1"
UPDATE_PLUGINS="1"


command[Check Oracle Logs]=/usr/local/nagios/libexec/check_xx_usm_log.sh /var/log/oracle.log.1 "hdfs-chi-sdb01 hdfs-chi-pdb01 hdfs-chi-pdb11"

It is not updating Nagios. I try to send manually with his command:

Code: Select all

/usr/local/nrdp/clients/send_nrdp.sh -u https://xxxx.xxxx.com/nrdp/ -t xxxxxxxx -H "hdfs-chi-usm01" -s "Check Oracle Logs" -S 0 -o "This is a test6"

and maybe 1 out of every 30 tries it actually updates in Nagios. I change the number on the output to know.

If I use the external command file to submit a passive result or the /nrdp url directly it updates XI everytime without fail. Running XI 5.6.14

[ssax]

Please attach this file:

/usr/local/nrdp/server/config.inc.php

If you set $cfg["debug"] = true; in that file and then run these commands:

Code: Select all

touch /usr/local/nrdp/server/debug.log
chown apache.nagios /usr/local/nrdp/server/debug.log
chmod 664 /usr/local/nrdp/server/debug.log

Then run this tail command:

Code: Select all

tail -Fn0 /usr/local/nrdp/server/debug.log

Then submit the checks, do you see any errors in the debug.log output?

If you leave the NRDP debugging enabled it will likely fill up your disk if left on so turn off debugging once you are done.

Do you see any related errors in your /usr/local/nagios/var/nagios.log file when the checks are missing?

[BanditBBS]

Finally got back form vacation(BOO) and trying this. The debug log looks good except one issue in the end:

Code: Select all

[Fri, 23 Oct 2020 12:24:05 -0500] nrdp_write_check_output_to_cmd(hostname=hdfs-chi-usm01, servicename=Check Oracle Logs, state=0, type=service, output=This is a test6
[Fri, 23 Oct 2020 12:24:05 -0500] nagios_command_group=nagcmd does not exist, not chgrp()ing
[Fri, 23 Oct 2020 12:24:05 -0500] nrdp_write_check_output_to_cmd() completed
[Fri, 23 Oct 2020 12:24:05 -0500] all nrdp checks have been written

That nagcmd error is the only thing that seems odd because it does exist:

Code: Select all

grep '^nagcmd:' /etc/group
nagcmd:x:501:nagios,apache,snmptt

and

Code: Select all

prw-rw---- 1 nagios nagcmd 0 Oct 23 12:09 /usr/local/nagios/var/rw/nagios.cmd

and

Code: Select all

<?php
// NRDP Config File
// Copyright (c) 2010 Nagios Enterprises, LLC.
// License: Nagios Open Software License <http://www.nagios.com/legal/licenses>
//
// $Id: config.inc.php 454 2011-01-13 22:52:28Z egalstad $


// an array of one or more tokens that are valid for this NRDP install
// a client request must contain a valid token in order for the NRDP to response or honor the request
// NOTE: tokens are just alphanumeric strings - make them hard to guess!
$cfg['authorized_tokens']=array("sifo85kli9uh",);

// do we require that HTTPS be used to access NRDP?
// set this value to 'false' to disable HTTPS requirement
$cfg["require_https"]=false;

// do we require that basic authentication be used to access NRDP?
// set this value to 'false' to disable basic auth requirement
$cfg["require_basic_auth"]=false;

// what basic authentication users are allowed to access NRDP?
// comment this variable out to allow all authenticated users access to the NRDP
$cfg["valid_basic_auth_users"]=array();

// the name of the system group that has write permissions to the external command file
// this group is also used to set file permissions when writing bulk commands or passive check results
// NOTE: both the Apache and Nagios users must be a member of this group
$cfg["nagios_command_group"]="nagcmd";

// full path to Nagios external command file
$cfg["command_file"]="/usr/local/nagios/var/rw/nagios.cmd";

// full path to check results spool directory
$cfg["check_results_dir"]="/var/nagiosramdisk/spool/checkresults";

// full path to directory where temp scratch files can be written
// NOTE: the Apache user need to be able create files here, and the Nagios user needs to read/delete those same files, so the /tmp system directory won't work (it has a sticky bit on it)
$cfg["tmp_dir"]="/usr/local/nagiosxi/tmp";

$cfg["debug"] = false;

[ssax]

Please send the rest of the logs if you would, also check to make sure the time doesn't say 0 as that's a bug in NRDP.

Is the OS AD/LDAP integrated? What is the output of this command?

Code: Select all

getent group nagcmd

[BanditBBS]

Please send the rest of the logs if you would, also check to make sure the time doesn't say 0 as that's a bug in NRDP.

Is the OS AD/LDAP integrated? What is the output of this command?

Code: Select all

getent group nagcmd

Sent you a PM with the log file. The OS does use IPA for authentication but the nagios user is a local user.

As for the command output:

Code: Select all

nagcmd:x:501:nagios,apache,snmptt

[ssax]

Please attach this file:

Code: Select all

/usr/local/nrdp/server/plugins/nagioscorepassivecheck/nagioscorepassivecheck.inc.php

You may want to edit this file:

Code: Select all

/usr/local/nrdp/server/config.inc.php

And set this:

Code: Select all

$cfg["allow_old_results"] = true;

See if that helps.

[BanditBBS]

Tried adding that setting, made no change. I sent 4 passive results and only 1 processed.

Attached file:

nagioscorepassivecheck.inc.php.txt

[ssax]

That looks good.

If you run this tail command as root and leave it running:

Code: Select all

tail -Fn0 /var/log/httpd/*error_log /usr/local/nagios/var/nagios.log /usr/local/nagios/var/cmdsubsys.log

Then run this command to send a check:

Code: Select all

/usr/local/nrdp/clients/send_nrdp.sh -u https://xxxx.xxxx.com/nrdp/ -t xxxxxxxx -H "hdfs-chi-usm01" -s "Check Oracle Logs" -S 0 -o "This is a test6"

Send us the full output from the still running tail command after a couple minutes so we can see what it is showing.

If that doesn't look like it'll give any clues, please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:

https://support.nagios.com/tickets/

Thanks!

[BanditBBS]

Ticket created, nothing related to this issue seemed to be popping up.

[ssax]

Hmm, not sure then, we'll need to debug on the remote.

Locking thread, ticket received, we will continue support through the ticket.

Thank you!