Fusion database lock issue

[hbouma]

PM sent with the ps -ef --cols=300 output and an updated mariadb.log file.

the trimpolled componant is still installed from the last time that we went through a similar issue.

We had another issue last night with locked DB files. We see no issues in the /var/log/messages. We did not run out of RAM and SWAP like we did over the weekend.

I also supplied some additional files from the Fusion logs.

Can we reopen the case from the last time this happened?

[tgriep]

Lets stop all of the Nagios Fusion processes and run a dbmaint on the server.
There are dbmaint processes that are stuck and they need to be stopped.
Depending on how much data there is, this could take a while.
Also, if the dbmaint_subsys shows that there is a locking issue, post them here.

Code: Select all

pkill -9 -u nagios
service crond stop
/usr/bin/php /usr/local/nagiosfusion/cron/dbmaint_subsys.php

Then start up the crond process.

Code: Select all

service crond start

Let the system run for 10 minutes and run the following commands and post the output.

Code: Select all

echo 'show full processlist;' |mysql -t -u fusion -pfusion
echo "show variables like 'innodb_lock_wait_timeout';" |mysql -t -u fusion -pfusion fusion

Thanks

[hbouma]

I will work with my company for a time when we can preform this work today and will get you the results.

This looks like the same thing we did from posts viewtopic.php?f=17&t=49945 and viewtopic.php?f=17&t=50159 as well as ticket 100103. Is there any way to find out the cause as to why this keeps happening? I don't want to be in this position again in another few weeks or months.

[tgriep]

I have gone through the previous posts and the tickets and with the number of users logging in to the GUI and the number of polls generated, the system cannot process them in time and they are stacking up.
The previous post I said that there are stuck dbmaint processes that are locking the database and causing the issue so they need to be stopped and the process ran manually so it can finish.

I suggest increasing the amount or RAM allocated to the system to 24 Gig.

Login to the Fusion GUI and go the Admin > System Settings menu.
Double the Global Authentication Interval: and the Global Polling Interval: options as well as the Polling Subsystem Memory Limit:.

Also, increase the Simultaneous Pollers: to the same number of servers you have Fused and that is 9.
If you add any new servers, increase that settings to match.

[hbouma]

I have preformed the requested steps. I am working with my server team to up the RAM and will test to see how that goes.

Code: Select all

pkill -9 -u nagios
service crond stop
/usr/bin/php /usr/local/nagiosfusion/cron/dbmaint_subsys.php
service crond start

Let the system run for 10 minutes and run the following commands and post the output.

Code: Select all

echo 'show full processlist;' |mysql -t -u fusion -pfusion
echo "show variables like 'innodb_lock_wait_timeout';" |mysql -t -u fusion -pfusion fusion

Here is the wait timeout. I will send the processlist in a PM per my IT Security policy as it has usernames in it.

Code: Select all

+--------------------------+-------+
| Variable_name            | Value |
+--------------------------+-------+
| innodb_lock_wait_timeout | 50    |
+--------------------------+-------+

[tgriep]

Thanks, I received the file. Did you see my PM?

[hbouma]

I did get the PM. I need to run it by my manager for the OK

[tgriep]

OK, let us know if you have any other questions.
The files you sent in look OK so hopefully clearing the locks manually will give you some time to upgrade to the Alpha copy.