Page 2 of 3
Re: FUSION disk full 5-6 days after starting new server
Posted: Tue Mar 27, 2018 11:09 am
by eloyd
I need to clarify something: I do not work for, nor do I represent, Nagios. I work for a company that provides professional services related to Nagios software. So please don't say "you" when talking to me and meaning "Nagios."
Re: FUSION disk full 5-6 days after starting new server
Posted: Tue Mar 27, 2018 12:34 pm
by bheden
@dlukinski,
As stated by
@eloyd, this problem you're having is not a common one. I sent you a new OVA download link via PM, and you never responded to me or the forum until this latest reply. I would like very much to help you resolve your issue.
Can you walk me through the steps you took once I sent you the new OVA, please?
Or if possible, can we schedule a remote assistance at some point today or tomorrow so that I can review the system?
Thanks.
Re: FUSION disk full 5-6 days after starting new server
Posted: Tue Mar 27, 2018 1:34 pm
by dlukinski
bheden wrote:@dlukinski,
As stated by
@eloyd, this problem you're having is not a common one. I sent you a new OVA download link via PM, and you never responded to me or the forum until this latest reply. I would like very much to help you resolve your issue.
Can you walk me through the steps you took once I sent you the new OVA, please?
Or if possible, can we schedule a remote assistance at some point today or tomorrow so that I can review the system?
Thanks.
Yes we could schedule the session tomorrow. Would 9 AM your time work for you?
Once OVA got spinned, we've added the servers and set few accounts (which should be there already for others to connect)
Just like with previous cases everything looked OK (still does) except for the storage consumed.
I expect FUSION to run out of storage in a few days on Easter.
Haven't heard from you (5 PM for me now) / any time you may have on Thursday?
Re: FUSION disk full 5-6 days after starting new server
Posted: Wed Mar 28, 2018 10:31 am
by dlukinski
No responses from Nagios Support.
DB have grown by 3 GB in a single day.
login as: root
root@fikc-nagfsprod01's password:
Last login: Tue Mar 27 15:13:45 2018 from 10.67.2.58
[root@fikc-nagfsprod01 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
37G 29G 6.8G 81% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 477M 122M 330M 27% /boot
/dev/sr0 56M 56M 0 100% /mnt
[root@fikc-nagfsprod01 ~]#
Calling support at this point. Issue is not resolved, no responses.
Re: FUSION disk full 5-6 days after starting new server
Posted: Wed Mar 28, 2018 11:23 am
by tgriep
I found these errors in the mysqld.log file.
InnoDB: ERROR: the age of the last checkpoint is 9443781,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
What this means is that the log buffer and file sizes are too small so the tables will not get truncated, causing them to grow.
To increase, the size
Add the following to the /etc/my.cnf file
Code: Select all
[mysqld]
innodb_log_buffer_size = 32M
innodb_buffer_pool_size = 1G
innodb_log_file_size = 256M
Stop MYSQL by running
Remove the log files
/var/lib/mysql/ib_logfile0
/var/lib/mysql/ib_logfile1
And restart the MYSQL by running
service mysqld start
If you still see the error again in the /var/log/mysqld.log file, increase the following entry in the /etc/my.cnf file
repeat the process by stopping mysqld, removing the files and starting mysqld.
Re: FUSION disk full 5-6 days after starting new server
Posted: Tue Apr 03, 2018 10:23 am
by dlukinski
tgriep wrote:I found these errors in the mysqld.log file.
InnoDB: ERROR: the age of the last checkpoint is 9443781,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
What this means is that the log buffer and file sizes are too small so the tables will not get truncated, causing them to grow.
To increase, the size
Add the following to the /etc/my.cnf file
Code: Select all
[mysqld]
innodb_log_buffer_size = 32M
innodb_buffer_pool_size = 1G
innodb_log_file_size = 256M
Stop MYSQL by running
Remove the log files
/var/lib/mysql/ib_logfile0
/var/lib/mysql/ib_logfile1
And restart the MYSQL by running
service mysqld start
If you still see the error again in the /var/log/mysqld.log file, increase the following entry in the /etc/my.cnf file
repeat the process by stopping mysqld, removing the files and starting mysqld.
Hello Tom
We are back to square one with 23 GB consumed over the Easter Holidays
[root@fikc-nagfsprod01 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
37G 23G 13G 65% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 477M 122M 330M 27% /boot
/dev/sr0 56M 56M 0 100% /mnt
[root@fikc-nagfsprod01 ~]#
- Removing log files did not fix the problem.
Re: FUSION disk full 5-6 days after starting new server
Posted: Tue Apr 03, 2018 10:42 am
by tgriep
Can you run the following commands on the Fusion server and post the /tmp/info.txt file?
Code: Select all
tail -200 /var/log/mysqld.log >/tmp/info.txt
ls -lR /var/lib/mysql/ >>/tmp/info.txt
ps -ef --cols=300 >>/tmp/info.txt
tail -200 /usr/local/nagiosfusion/var/log/dbmaint_subsys.log >>/tmp/info.txt
Or PM it to me if you prefer, thanks.
Re: FUSION disk full 5-6 days after starting new server
Posted: Tue Apr 03, 2018 11:03 am
by dlukinski
tgriep wrote:Can you run the following commands on the Fusion server and post the /tmp/info.txt file?
Code: Select all
tail -200 /var/log/mysqld.log >/tmp/info.txt
ls -lR /var/lib/mysql/ >>/tmp/info.txt
ps -ef --cols=300 >>/tmp/info.txt
tail -200 /usr/local/nagiosfusion/var/log/dbmaint_subsys.log >>/tmp/info.txt
Or PM it to me if you prefer, thanks.
Attached
Please review
Re: FUSION disk full 5-6 days after starting new server
Posted: Tue Apr 03, 2018 2:03 pm
by tgriep
I did not see the output of this command in the info.txt file.
Code: Select all
tail -200 /usr/local/nagiosfusion/var/log/dbmaint_subsys.log >>/tmp/info.txt
Can you run it again and post the info.txt file?
Re: FUSION disk full 5-6 days after starting new server
Posted: Tue Apr 03, 2018 3:47 pm
by dlukinski
tgriep wrote:I did not see the output of this command in the info.txt file.
Code: Select all
tail -200 /usr/local/nagiosfusion/var/log/dbmaint_subsys.log >>/tmp/info.txt
Can you run it again and post the info.txt file?
Looks like command in question does not produce any entries to the output file
- it is 0 size since install date on Mar 13th (install date seem to explain our troubles with the server :-0)