Multiple dbmaint running and /var/lib/mysql filling up 100 %

This support forum board is for questions relating to Nagios Fusion.
Locked
bpennin
Posts: 65
Joined: Tue Feb 20, 2018 2:23 pm

Multiple dbmaint running and /var/lib/mysql filling up 100 %

Post by bpennin »

Running on RHEL 7 and Fusion 4.1.9. Issue is my /var/lib/mysql will fill up to 100 percent in a day or two. I did notice that there are multiple dbmaint jobs running so I assume that is the issue in some way.
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Multiple dbmaint running and /var/lib/mysql filling up 1

Post by ssax »

Do you see anything in /usr/local/nagiosfusion/var/log/dberrors.log or /usr/local/nagiosfusion/var/log/dbmaint_subsys.log or in Admin > Fusion Logs?

I would reboot the server to get rid of the old processes and see if that helps.

How many XI or Core servers do you have being monitored?

How many XI Users do your XI systems have?

Send a screenshot of Admin > System Settings > General Tab AND send one of the Data & Polling tab.

Attach this file as well:

Code: Select all

/etc/php.ini
What file was consuming all the space in /var/lib/mysql/fusion?
bpennin
Posts: 65
Joined: Tue Feb 20, 2018 2:23 pm

Re: Multiple dbmaint running and /var/lib/mysql filling up 1

Post by bpennin »

php.ini
FusionGeneral.jpg
Data_PollingTab.jpg
I see the following in the dberror log:
exec(): SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transactionSQL: [429] DELETE
FROM polled_extras
WHERE polled_data_id < :polled_data_id
AND poll_key = :poll_key
AND polled_data_id
IN (
SELECT polled_data_id
FROM polled_data
WHERE server_id = :server_id
AND username = :username
)
AND
): Failed to execute 'DELETE
FROM polled_extras
WHERE polled_data_id < :polled_data_id
AND poll_key = :poll_key
AND polled_data_id
IN (
SELECT polled_data_id
FROM polled_data
WHERE server_id = :server_id
AND username = :username
)'SQL: [429] DELETE
FROM polled_extras
WHERE polled_data_id < :polled_data_id
AND poll_key = :poll_key
AND polled_data_id
IN (
SELECT polled_data_id
FROM polled_data
WHERE server_id = :server_id
AND username = :username
)
Params: 4

dbmaint log:
2021-06-30 08:15:55[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_data
WHERE server_id NOT IN (3,5,4,9)
2021-06-30 08:16:46[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_data
WHERE UNIX_TIMESTAMP(polled_time) < 1624968955
2021-06-30 08:17:37[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-06-30 08:20:55[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_data
WHERE server_id NOT IN (3,5,4,9)
2021-06-30 08:23:33[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
2021-07-01 02:20:54[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 05:10:58[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 09:01:03[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 09:26:31[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:06:06[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:15:59[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:20:55[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:26:07[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:36:07[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:41:18[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:56:17[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:01:09[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:05:56[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:10:58[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:20:54[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:26:10[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:36:12[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:45:53[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 12:55:37[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593


I would reboot the server to get rid of the old processes and see if that helps.

The server was rebooted and continue to see multiple dbmaint_subsys.php running. I changed the cron job to every 9 min to try to give it more time to complete but that did not help. I killed them off from last night and below is what it looks like so far:
nagios 45305 45304 0 08:09 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/dbmaint_subsys.php >>/usr/local/nagiosfusion/var/log/dbmaint_subsys.log 2>&1
nagios 45308 45305 0 08:09 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/dbmaint_subsys.php
nagios 47935 47934 0 08:18 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/dbmaint_subsys.php >>/usr/local/nagiosfusion/var/log/dbmaint_subsys.log 2>&1
nagios 47944 47935 0 08:18 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/dbmaint_subsys.php

How many XI or Core servers do you have being monitored?

We have 3XI servers and a pair of Log servers defined to Fusion

The files taking up the space in /var/lib/mysql are the below:
-rw-rw---- 1 mysql mysql 268435456 Jul 2 08:36 ib_logfile0
-rw-rw---- 1 mysql mysql 268435456 Jul 2 08:36 ib_logfile1
The file taking up the most space in the fusion directory:
6006243328 Jul 2 08:40 polled_extras.ibd
You do not have the required permissions to view the files attached to this post.
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Multiple dbmaint running and /var/lib/mysql filling up 1

Post by ssax »

Please check the box in Admin > System Settings > Data & Polling for "Only poll data for mapped users when a user is logged in" and click the Update Settings button.

Then run this command:

Code: Select all

/usr/local/nagiosfusion/scripts/truncate_polled.php
See if that helps.

If it doesn't, please create a ticket for this and include a link back to this forum thread so we can get a remote session setup to debug further:

https://support.nagios.com/tickets/
Locked