Nagios Backups consuming a lot of disk space.

[arthurkroth]

Hi all,

I hope you are doing well.

We use Nagios XI 2024R1.3.3 to monitor 71 hosts and 850+ services across all hosts.
Recently, we migrated our NagiosXI server from EOL CentOS to Ubuntu 22.04. Everything is working great, and I have no issues with the monitoring.

On CentOS, we only had the SSH backup enabled, which is still working with no issues; my manager requested that we also have a local backup running weekly, and here's where the problem started.

Currently, our Nagios backup takes around 25GB after being compacted. I'm here to clarify this to ensure we're not backing up stuff we shouldn't. Is 25GB a standard backup size for the number of hosts and services we monitor?

The root partition of the disk currently has 40GB available. However, the backup has failed for the second weekend in a row due to a lack of space. I wonder if there's a way to delete the existing NagiosXI backup on the disk before starting the new one directly from the backup script without setting up a new Cron job manually.

My data retention period is 365 days (default), and we need to keep that config.

Any help/suggestion is appreciated.

Kind regards,
Arthur Kroth

[arthurkroth]

Quick update:

The error over the weekend(last Sunday) was that the backup failed because the disk reached 100% and the server restarted. When checking the logs, this is what I get:

Code: Select all

mysqldump: Got error: 2013: Lost connection to MySQL server during query when using LOCK TABLES
	    Error backing up MySQL database 'nagios' - check the password in this script!

I then deleted the existing backup from /home/nagios and again triggered the scheduled local backup. Surprisingly, the backup was completed quickly, and the file size was only 1.38GB.

I wonder if the server crash could've caused this. Going from 25GB to 1.38GB doesn't sound healthy.

Regards,
Arthur

[jsimon]

Hi @arthurkroth,

It is generally pretty harmful for your disk to reach 100% storage and from the error in your backup logs it sounds likely that your normal backup wasn't able to run as a crash caused a lock on one or more tables. I would recommend restoring a backup to get your database back to normal working condition.

Generally speaking depending on the size of your monitoring setup, 25GB does not sound like an abnormally large backup size, as all of your rrds need to be backed up, and with 850+ at roughly 1MB each, that's roughly 1/3rd of your backup size just for rrds. I would say dropping to 1.38GB is definitely not healthy and is part of why I'm recommending restoring from a backup. The size difference makes sense given the log message indicating MYSQL connection issues, as database entries are a large portion of what is being backed up.

I don't see any functionality in the backup script to rotate the directory you back up to, so I would recommend adding a cron job to do this, but also to increase the size of the disk you're running on to provide more than one single backup worth of leeway for you, as again, filling up your disk to 100% is very bad for your disk health.

Let us know if you have any additional questions.

[arthurkroth]

Hi @jsimon,

I believe I have found the "issue". Looking at the scheduledbackups.log I can see the following lines:

Code: Select all

Backing up nagios user home dir...
tar: Removing leading `/' from member names
tar: /home/nagios/xi-backup-1738745111-ec175faa72110f36c4f5/home-nagios.tar.gz: file changed as we read it

The last line of the log after the completion states the following:

Code: Select all

===============
BACKUP COMPLETE
===============
Backup stored in /home/nagios/nagiosxi.1738745111.tar.gz

So, I concluded that I recursively backed up the previous week's backup every week when creating a new backup since the place where my new backup gets saved will be part of the following week's backup. Does that make sense?

The huge drop in the file size was probably because I had cleared the existing local backup on /home/nagios before starting the scheduled backup again, so it didn't have the 20GB+ file on /home/nagios to be backed up.

To fix this, I will create a new disk on my VM to store my backups rather than keep it on /home/nagios.

Thanks for your suggestions there.

Kind regards,
Arthur Kroth

[jsimon]

Glad you were able to get the issue figured out! I'll go ahead and lock this topic.