Nagios cmdsubsys.log Growing Excessively

[spperry]

Starting sometime late Monday afternoon/evening, we started having issues with our Nagios system. I realized yesterday morning that the server was out of disk space, so we cleaned up space and extended the logical volume by 10GB. Between the two, a considerable amount of free space was added. Disk space was sort of sporadic through the day, jumping up and down by quite a bit, but it never hit zero. Overnight, I got a couple of errors, and found this morning that the disk had achieved filling up the volume entirely again. After poking around, I discovered that the cmdsubsys.log file in /usr/local/nagiosxi/var was 56GB.

I cleared the log file in question and the system does return to a functional state; however, the cmdsubsys.log file (with a yesterday's date addition to the name, i.e., cmdsubsys.log-20180821) keeps growing at an exceptionally rapid pace. It is filling up the drive and having to be cleared roughly every two hours. That being said, what would cause that particular log file to grow that large repeatedly? I have also noted in a second post here regarding the data in the log itself.

We were running 5.4.13 until Saturday, when I upgraded to 5.5.2. There were no issues from Saturday-Monday afternoon, and now it's all happens very quickly.

Any assistance in what I need to do to evaluate this would be greatly appreciated.

Thanks!
Scott

[spperry]

Quick update - I tailed the log and found hundreds of lines of this repeated:

PHP Notice: Uninitialized string offset: 0 in /usr/local/nagiosxi/html/includes/components/nagiosbpi/api_tool.php on line 138
PHP Warning: Illegal string offset 'is_currently_running' in /usr/local/nagiosxi/html/includes/components/nagiosbpi/api_tool.php on line 138

I cleared the log (because the system was totally broken and is an integral part of our audited systems) and it cleared 56GB as the file went away. It has currently chewed up about 40GB of that 56GB this morning (since about 8AM EST), but it no longer appears to be in the cmdsubsys.log.

What am I missing here?

[spperry]

This is the code from that area (just for the record):


// Verify ndo and core are running and loaded
$data = get_program_status_xml_output(array(), true);
if ($data['is_currently_running'] == 1) {
$is_running = true;
}

Lines 135-141.

[scottwilkerson]

I'm going to have a dev look into that however for now lets edit /etc/cron.d/nagiosxi
and change this line

Code: Select all

*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1

to this

Code: Select all

*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1

removing one of the >

This will trim the log every minute

[scottwilkerson]

Also, can you verify if you go to Admin -> Monitoring Engine Status all of the items are green?

[spperry]

I tried removing the > but it is still increasing the size of the file named:

cmdsubsys.log-20180821

As far as the monitoring engine and environment, everything looks good. The only issue is that one file that grows ever larger until it fills the disk and everything stops because it can't process.

[tgriep]

Is it only the cmdsubsys.log file that is growing really large or are there other log files growing as well.

Can you post 200 lines of the cmdsubsys.log file so I can see if there are any other errors that can help troubleshoot this?

Can you run the following commands as root and post the output here?

Code: Select all

echo 'SELECT TABLE_NAME AS "Table", round(((data_length + index_length) / 1024 / 1024), 2) AS Size_in_MB FROM information_schema.TABLES WHERE table_schema = "nagiosxi" ORDER BY Size_in_MB DESC;' |mysql -t -u root -pnagiosxi nagiosxi

Code: Select all

ps -ef --cols=300

[spperry]

Results attached. The cmdsubset.log only has the one line...

Definitely still filling up the other file, too.

[spperry]

Just to be clear, the cmdsubset.log file doesn't grow - it's the cmdsubset.log-20180821 file that continually grows. It doesn't appear that anything else is, though. If I clear that one file with the 2018 date appended to it it drops to a reasonable number used on the volume.

So, cmdsubset.log is okay. It's cmdsubset.log-20180821

Thanks.

Scott

[spperry]

Since the original cmdsubset.log file isn't growing (it's the additional), I added the > back into the cron job piece. I have let that file grow a bit (it's only 58K), and I've added it here. This will give more than the one line from the other one.

Thanks.
Scott