FUSION disk full 5-6 days after starting new server

[eloyd]

I need to clarify something: I do not work for, nor do I represent, Nagios. I work for a company that provides professional services related to Nagios software. So please don't say "you" when talking to me and meaning "Nagios."

[bheden]

@dlukinski,

As stated by @eloyd, this problem you're having is not a common one. I sent you a new OVA download link via PM, and you never responded to me or the forum until this latest reply. I would like very much to help you resolve your issue.

Can you walk me through the steps you took once I sent you the new OVA, please?

Or if possible, can we schedule a remote assistance at some point today or tomorrow so that I can review the system?

Thanks.

[dlukinski]

@dlukinski,

As stated by @eloyd, this problem you're having is not a common one. I sent you a new OVA download link via PM, and you never responded to me or the forum until this latest reply. I would like very much to help you resolve your issue.

Can you walk me through the steps you took once I sent you the new OVA, please?

Or if possible, can we schedule a remote assistance at some point today or tomorrow so that I can review the system?

Thanks.

Yes we could schedule the session tomorrow. Would 9 AM your time work for you?

Once OVA got spinned, we've added the servers and set few accounts (which should be there already for others to connect)
Just like with previous cases everything looked OK (still does) except for the storage consumed.
I expect FUSION to run out of storage in a few days on Easter.

Haven't heard from you (5 PM for me now) / any time you may have on Thursday?

[dlukinski]

No responses from Nagios Support.

DB have grown by 3 GB in a single day.

login as: root
root@fikc-nagfsprod01's password:
Last login: Tue Mar 27 15:13:45 2018 from 10.67.2.58
[root@fikc-nagfsprod01 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
37G 29G 6.8G 81% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 477M 122M 330M 27% /boot
/dev/sr0 56M 56M 0 100% /mnt
[root@fikc-nagfsprod01 ~]#

Calling support at this point. Issue is not resolved, no responses.

[tgriep]

I found these errors in the mysqld.log file.

InnoDB: ERROR: the age of the last checkpoint is 9443781,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.

What this means is that the log buffer and file sizes are too small so the tables will not get truncated, causing them to grow.
To increase, the size

Add the following to the /etc/my.cnf file

Code: Select all

[mysqld]
innodb_log_buffer_size          = 32M
innodb_buffer_pool_size         = 1G
innodb_log_file_size            = 256M

Stop MYSQL by running

Code: Select all

service mysqld stop

Remove the log files
/var/lib/mysql/ib_logfile0
/var/lib/mysql/ib_logfile1

And restart the MYSQL by running
service mysqld start

If you still see the error again in the /var/log/mysqld.log file, increase the following entry in the /etc/my.cnf file

Code: Select all

innodb_log_file_size            = 256M

repeat the process by stopping mysqld, removing the files and starting mysqld.

[dlukinski]

I found these errors in the mysqld.log file.

InnoDB: ERROR: the age of the last checkpoint is 9443781,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.

What this means is that the log buffer and file sizes are too small so the tables will not get truncated, causing them to grow.
To increase, the size

Add the following to the /etc/my.cnf file

Code: Select all

[mysqld]
innodb_log_buffer_size          = 32M
innodb_buffer_pool_size         = 1G
innodb_log_file_size            = 256M

Stop MYSQL by running

Code: Select all

service mysqld stop

Remove the log files
/var/lib/mysql/ib_logfile0
/var/lib/mysql/ib_logfile1

And restart the MYSQL by running
service mysqld start

If you still see the error again in the /var/log/mysqld.log file, increase the following entry in the /etc/my.cnf file

Code: Select all

innodb_log_file_size            = 256M

repeat the process by stopping mysqld, removing the files and starting mysqld.

Hello Tom

We are back to square one with 23 GB consumed over the Easter Holidays

[root@fikc-nagfsprod01 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
37G 23G 13G 65% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 477M 122M 330M 27% /boot
/dev/sr0 56M 56M 0 100% /mnt
[root@fikc-nagfsprod01 ~]#

- Removing log files did not fix the problem.

[tgriep]

Can you run the following commands on the Fusion server and post the /tmp/info.txt file?

Code: Select all

tail -200 /var/log/mysqld.log >/tmp/info.txt
ls -lR /var/lib/mysql/ >>/tmp/info.txt
ps -ef --cols=300 >>/tmp/info.txt
tail -200 /usr/local/nagiosfusion/var/log/dbmaint_subsys.log >>/tmp/info.txt

Or PM it to me if you prefer, thanks.

[dlukinski]

Can you run the following commands on the Fusion server and post the /tmp/info.txt file?

Code: Select all

tail -200 /var/log/mysqld.log >/tmp/info.txt
ls -lR /var/lib/mysql/ >>/tmp/info.txt
ps -ef --cols=300 >>/tmp/info.txt
tail -200 /usr/local/nagiosfusion/var/log/dbmaint_subsys.log >>/tmp/info.txt

Or PM it to me if you prefer, thanks.

Attached

Please review

[tgriep]

I did not see the output of this command in the info.txt file.

Code: Select all

tail -200 /usr/local/nagiosfusion/var/log/dbmaint_subsys.log >>/tmp/info.txt

Can you run it again and post the info.txt file?

[dlukinski]

I did not see the output of this command in the info.txt file.

Code: Select all

tail -200 /usr/local/nagiosfusion/var/log/dbmaint_subsys.log >>/tmp/info.txt

Can you run it again and post the info.txt file?

Looks like command in question does not produce any entries to the output file
- it is 0 size since install date on Mar 13th (install date seem to explain our troubles with the server :-0)