massive updates cause disk full

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
axvaster
Posts: 21
Joined: Tue Jan 08, 2019 11:54 am

massive updates cause disk full

Post by axvaster »

Hi support,

Last week, we found a Nagios XI server CPU load gets high, disk size decreases fast, suddenly and without a reason.

There's no any significant server changes or env changes, all things just as usual.

And finally found that it is the xi_meta table increasing size extremely fast (1G/min).

At the first moment all we can do is to restart database.

It seems like a tmp file, every restart database would release size, but never resolve issue.

It could not slow down the speed until we TRUNCATE-ed the table

Code: Select all

truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;
(Found from forum; link: https://support.nagios.com/forum/viewto ... =6&t=46425)

And the root cause seems that database wants to update a table with 17438358 rows in the table "xi_mata".

Code: Select all

---TRANSACTION 2BEBA539, ACTIVE 15355 sec fetching rows
mysql tables in use 1, locked 1
1107859 lock struct(s), heap size 98662840, 17438358 row lock(s), undo log entries 1
MySQL thread id 863, OS thread handle 0x7f832dffb700, query id 2875205 localhost nagiosxi Updating
UPDATE xi_meta SET keyvalue='1605194641' WHERE metatype_id='0' AND metaobj_id='0' AND keyname='last_nom_nagioscore_checkpoint'

---TRANSACTION 2BEBA539, ACTIVE 15483 sec fetching rows
mysql tables in use 1, locked 1
1114497 lock struct(s), heap size 99252664, 17545150 row lock(s), undo log entries 1
MySQL thread id 863, OS thread handle 0x7f832dffb700, query id 2875205 localhost nagiosxi Updating
UPDATE xi_meta SET keyvalue='1605194641' WHERE metatype_id='0' AND metaobj_id='0' AND keyname='last_nom_nagioscore_checkpoint'
Please advise how to fix this.

Thank you!
dchurch
Posts: 858
Joined: Wed Oct 07, 2020 12:46 pm
Location: Yo mama

Re: massive updates cause disk full

Post by dchurch »

It couldn't hurt, but I'd suggest running the running the repair database routine.

If you're still having issues, we can gain some insight into what is driving that table to become so large by running the following command:

Code: Select all

mysql -unagiosxi -pn@gweb nagiosxi -e 'select keyname, count(*) from xi_meta group by keyname'
Also if you PM me a system profile I can diagnose further. Get one by going to Admin (top menu) => System Profile (in the left menu), then clicking the blue button.
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.

Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
axvaster
Posts: 21
Joined: Tue Jan 08, 2019 11:54 am

Re: massive updates cause disk full

Post by axvaster »

Okay, I will keep following the server.

If it happens error again, I will try these commands.


The profile I sent is generated on Nov 2, 2 weeks before the issues happened.

Not sure this profile could help or not.


Anyway, we are planing to upgrade Nagios XI to latest version recently.
(To avoid potential DB bugs.)

If there is no issue before then, I think it would be good to close this ticket that time.
benjaminsmith
Posts: 5324
Joined: Wed Aug 22, 2018 4:39 pm
Location: saint paul

Re: massive updates cause disk full

Post by benjaminsmith »

HI,
If there is no issue before then, I think it would be good to close this ticket that time.
Sounds good. We'll keep this open for now.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.

Be sure to check out our Knowledgebase for helpful articles and solutions!
Locked