Multiple dbmaint running and /var/lib/mysql filling up 100 %
Multiple dbmaint running and /var/lib/mysql filling up 100 %
Running on RHEL 7 and Fusion 4.1.9. Issue is my /var/lib/mysql will fill up to 100 percent in a day or two. I did notice that there are multiple dbmaint jobs running so I assume that is the issue in some way.
Re: Multiple dbmaint running and /var/lib/mysql filling up 1
Do you see anything in /usr/local/nagiosfusion/var/log/dberrors.log or /usr/local/nagiosfusion/var/log/dbmaint_subsys.log or in Admin > Fusion Logs?
I would reboot the server to get rid of the old processes and see if that helps.
How many XI or Core servers do you have being monitored?
How many XI Users do your XI systems have?
Send a screenshot of Admin > System Settings > General Tab AND send one of the Data & Polling tab.
Attach this file as well:
What file was consuming all the space in /var/lib/mysql/fusion?
I would reboot the server to get rid of the old processes and see if that helps.
How many XI or Core servers do you have being monitored?
How many XI Users do your XI systems have?
Send a screenshot of Admin > System Settings > General Tab AND send one of the Data & Polling tab.
Attach this file as well:
Code: Select all
/etc/php.ini
Re: Multiple dbmaint running and /var/lib/mysql filling up 1
exec(): SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transactionSQL: [429] DELETE
FROM polled_extras
WHERE polled_data_id < :polled_data_id
AND poll_key = :poll_key
AND polled_data_id
IN (
SELECT polled_data_id
FROM polled_data
WHERE server_id = :server_id
AND username = :username
)
AND
): Failed to execute 'DELETE
FROM polled_extras
WHERE polled_data_id < :polled_data_id
AND poll_key = :poll_key
AND polled_data_id
IN (
SELECT polled_data_id
FROM polled_data
WHERE server_id = :server_id
AND username = :username
)'SQL: [429] DELETE
FROM polled_extras
WHERE polled_data_id < :polled_data_id
AND poll_key = :poll_key
AND polled_data_id
IN (
SELECT polled_data_id
FROM polled_data
WHERE server_id = :server_id
AND username = :username
)
Params: 4
dbmaint log:
2021-06-30 08:15:55[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_data
WHERE server_id NOT IN (3,5,4,9)
2021-06-30 08:16:46[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_data
WHERE UNIX_TIMESTAMP(polled_time) < 1624968955
2021-06-30 08:17:37[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-06-30 08:20:55[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_data
WHERE server_id NOT IN (3,5,4,9)
2021-06-30 08:23:33[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
2021-07-01 02:20:54[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 05:10:58[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 09:01:03[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 09:26:31[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:06:06[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:15:59[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:20:55[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:26:07[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:36:07[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:41:18[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 10:56:17[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:01:09[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:05:56[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:10:58[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:20:54[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:26:10[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:36:12[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 11:45:53[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2021-07-01 12:55:37[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 593
I would reboot the server to get rid of the old processes and see if that helps.
The server was rebooted and continue to see multiple dbmaint_subsys.php running. I changed the cron job to every 9 min to try to give it more time to complete but that did not help. I killed them off from last night and below is what it looks like so far:
nagios 45305 45304 0 08:09 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/dbmaint_subsys.php >>/usr/local/nagiosfusion/var/log/dbmaint_subsys.log 2>&1
nagios 45308 45305 0 08:09 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/dbmaint_subsys.php
nagios 47935 47934 0 08:18 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/dbmaint_subsys.php >>/usr/local/nagiosfusion/var/log/dbmaint_subsys.log 2>&1
nagios 47944 47935 0 08:18 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/dbmaint_subsys.php
How many XI or Core servers do you have being monitored?
We have 3XI servers and a pair of Log servers defined to Fusion
The files taking up the space in /var/lib/mysql are the below:
-rw-rw---- 1 mysql mysql 268435456 Jul 2 08:36 ib_logfile0
-rw-rw---- 1 mysql mysql 268435456 Jul 2 08:36 ib_logfile1
The file taking up the most space in the fusion directory:
6006243328 Jul 2 08:40 polled_extras.ibd
You do not have the required permissions to view the files attached to this post.
Re: Multiple dbmaint running and /var/lib/mysql filling up 1
Please check the box in Admin > System Settings > Data & Polling for "Only poll data for mapped users when a user is logged in" and click the Update Settings button.
Then run this command:
See if that helps.
If it doesn't, please create a ticket for this and include a link back to this forum thread so we can get a remote session setup to debug further:
https://support.nagios.com/tickets/
Then run this command:
Code: Select all
/usr/local/nagiosfusion/scripts/truncate_polled.php
If it doesn't, please create a ticket for this and include a link back to this forum thread so we can get a remote session setup to debug further:
https://support.nagios.com/tickets/