PM sent with the ps -ef --cols=300 output and an updated mariadb.log file.
the trimpolled componant is still installed from the last time that we went through a similar issue.
We had another issue last night with locked DB files. We see no issues in the /var/log/messages. We did not run out of RAM and SWAP like we did over the weekend.
I also supplied some additional files from the Fusion logs.
Can we reopen the case from the last time this happened?
Fusion database lock issue
Re: Fusion database lock issue
Lets stop all of the Nagios Fusion processes and run a dbmaint on the server.
There are dbmaint processes that are stuck and they need to be stopped.
Depending on how much data there is, this could take a while.
Also, if the dbmaint_subsys shows that there is a locking issue, post them here.
Then start up the crond process.
Let the system run for 10 minutes and run the following commands and post the output.
Thanks
There are dbmaint processes that are stuck and they need to be stopped.
Depending on how much data there is, this could take a while.
Also, if the dbmaint_subsys shows that there is a locking issue, post them here.
Code: Select all
pkill -9 -u nagios
service crond stop
/usr/bin/php /usr/local/nagiosfusion/cron/dbmaint_subsys.php
Code: Select all
service crond start
Code: Select all
echo 'show full processlist;' |mysql -t -u fusion -pfusion
echo "show variables like 'innodb_lock_wait_timeout';" |mysql -t -u fusion -pfusion fusion
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: Fusion database lock issue
I will work with my company for a time when we can preform this work today and will get you the results.
This looks like the same thing we did from posts viewtopic.php?f=17&t=49945 and viewtopic.php?f=17&t=50159 as well as ticket 100103. Is there any way to find out the cause as to why this keeps happening? I don't want to be in this position again in another few weeks or months.
This looks like the same thing we did from posts viewtopic.php?f=17&t=49945 and viewtopic.php?f=17&t=50159 as well as ticket 100103. Is there any way to find out the cause as to why this keeps happening? I don't want to be in this position again in another few weeks or months.
Re: Fusion database lock issue
I have gone through the previous posts and the tickets and with the number of users logging in to the GUI and the number of polls generated, the system cannot process them in time and they are stacking up.
The previous post I said that there are stuck dbmaint processes that are locking the database and causing the issue so they need to be stopped and the process ran manually so it can finish.
I suggest increasing the amount or RAM allocated to the system to 24 Gig.
Login to the Fusion GUI and go the Admin > System Settings menu.
Double the Global Authentication Interval: and the Global Polling Interval: options as well as the Polling Subsystem Memory Limit:.
Also, increase the Simultaneous Pollers: to the same number of servers you have Fused and that is 9.
If you add any new servers, increase that settings to match.
The previous post I said that there are stuck dbmaint processes that are locking the database and causing the issue so they need to be stopped and the process ran manually so it can finish.
I suggest increasing the amount or RAM allocated to the system to 24 Gig.
Login to the Fusion GUI and go the Admin > System Settings menu.
Double the Global Authentication Interval: and the Global Polling Interval: options as well as the Polling Subsystem Memory Limit:.
Also, increase the Simultaneous Pollers: to the same number of servers you have Fused and that is 9.
If you add any new servers, increase that settings to match.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: Fusion database lock issue
I have preformed the requested steps. I am working with my server team to up the RAM and will test to see how that goes.
Here is the wait timeout. I will send the processlist in a PM per my IT Security policy as it has usernames in it.Let the system run for 10 minutes and run the following commands and post the output.Code: Select all
pkill -9 -u nagios service crond stop /usr/bin/php /usr/local/nagiosfusion/cron/dbmaint_subsys.php service crond start
Code: Select all
echo 'show full processlist;' |mysql -t -u fusion -pfusion echo "show variables like 'innodb_lock_wait_timeout';" |mysql -t -u fusion -pfusion fusion
Code: Select all
+--------------------------+-------+
| Variable_name | Value |
+--------------------------+-------+
| innodb_lock_wait_timeout | 50 |
+--------------------------+-------+
Re: Fusion database lock issue
Thanks, I received the file. Did you see my PM?
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: Fusion database lock issue
I did get the PM. I need to run it by my manager for the OK
Re: Fusion database lock issue
OK, let us know if you have any other questions.
The files you sent in look OK so hopefully clearing the locks manually will give you some time to upgrade to the Alpha copy.
The files you sent in look OK so hopefully clearing the locks manually will give you some time to upgrade to the Alpha copy.
Be sure to check out our Knowledgebase for helpful articles and solutions!