Nagios not deleting old backups

[emartine]

MOD EDIT: Split from https://support.nagios.com/forum/viewto ... 16&t=43679

Ok. This is embarrassing. We were using a custom check for checking disk space and it didn't account for XFS which was created for the /usr/local partition.
1.1T of disk space and I didn't notice that it was full because I was looking at the web page noting that disk check was OK. I cleared up space and everything seems to be working.

Nagios XI is backing up its configuration files in this same location as well as to another server and for some reason it isn't keeping just 5 backupos as we told it to.

[mcapra]

Can you share the output of this sql? Replace the credentials if you're not using the default ones.

Code: Select all

echo "SELECT * FROM xi_options WHERE name LIKE '%backup%';" | mysql -uroot -pnagiosxi nagiosxi

Can you also share the contents of this file:

Code: Select all

/usr/local/nagiosxi/var/components/scheduledbackups.log

[emartine]

I haven't changed any defaults.


echo "SELECT * FROM xi_options WHERE name LIKE '%backup%';" | mysql -uroot -pnagiosxi nagiosxi
ERROR 1049 (42000): Unknown database 'nagiosxi'






Sample output

Code: Select all

04-22-2017 11:02:01 DEBUG: Running scheduled local backup ...
04-22-2017 11:02:01 INFO: Creating a local backup: nagiosxi.1492876921
04-22-2017 11:02:01 INFO: Too many backups! Limit is 7. Removing: nagiosxi.1462118522.tar.gz before proceeding with backup.
04-22-2017 11:02:02 DEBUG: Sending create local backup command to CmdSubsystem
04-23-2017 05:01:02 DEBUG: Running scheduled SSH backup ...
04-23-2017 05:01:02 DEBUG: Sending create local backup command to CmdSubsystem
04-23-2017 11:02:02 DEBUG: Running scheduled local backup ...
04-23-2017 11:02:02 INFO: Creating a local backup: nagiosxi.1492963322
04-23-2017 11:02:02 INFO: Too many backups! Limit is 7. Removing: nagiosxi.1462118522.tar.gz before proceeding with backup.
04-23-2017 11:02:03 DEBUG: Sending create local backup command to CmdSubsystem
04-24-2017 05:01:01 DEBUG: Running scheduled SSH backup ...
04-24-2017 05:01:01 DEBUG: Sending create local backup command to CmdSubsystem
04-24-2017 11:02:01 DEBUG: Running scheduled local backup ...
04-24-2017 11:02:01 INFO: Creating a local backup: nagiosxi.1493049721
04-24-2017 11:02:01 INFO: Too many backups! Limit is 7. Removing: nagiosxi.1462118522.tar.gz before proceeding with backup.
04-24-2017 11:02:02 DEBUG: Sending create local backup command to CmdSubsystem
04-25-2017 05:01:02 DEBUG: Running scheduled SSH backup ...
04-25-2017 05:01:02 DEBUG: Sending create local backup command to CmdSubsystem
04-25-2017 11:02:01 DEBUG: Running scheduled local backup ...
04-25-2017 11:02:01 INFO: Creating a local backup: nagiosxi.1493136121
04-25-2017 11:02:01 INFO: Too many backups! Limit is 7. Removing: nagiosxi.1462118522.tar.gz before proceeding with backup.
04-25-2017 11:02:03 DEBUG: Sending create local backup command to CmdSubsystem
04-26-2017 05:01:01 DEBUG: Running scheduled SSH backup ...
04-26-2017 05:01:01 DEBUG: Sending create local backup command to CmdSubsystem
04-26-2017 11:02:01 DEBUG: Running scheduled local backup ...
04-26-2017 11:02:01 INFO: Creating a local backup: nagiosxi.1493222521
04-26-2017 11:02:01 INFO: Too many backups! Limit is 7. Removing: nagiosxi.1462118522.tar.gz before proceeding with backup.
04-26-2017 11:02:03 DEBUG: Sending create local backup command to CmdSubsystem
04-27-2017 05:01:01 DEBUG: Running scheduled SSH backup ...
04-27-2017 05:01:01 DEBUG: Sending create local backup command to CmdSubsystem
04-27-2017 11:02:01 DEBUG: Running scheduled local backup ...
04-27-2017 11:02:01 INFO: Creating a local backup: nagiosxi.1493308921
04-27-2017 11:02:01 INFO: Too many backups! Limit is 7. Removing: nagiosxi.1462118522.tar.gz before proceeding with backup.
04-27-2017 11:02:02 DEBUG: Sending create local backup command to CmdSubsystem
04-28-2017 05:01:01 DEBUG: Running scheduled SSH backup ...
04-28-2017 05:01:01 DEBUG: Sending create local backup command to CmdSubsystem
04-28-2017 11:02:01 DEBUG: Running scheduled local backup ...
04-28-2017 11:02:01 INFO: Creating a local backup: nagiosxi.1493395321
04-28-2017 11:02:01 INFO: Too many backups! Limit is 7. Removing: nagiosxi.1462118522.tar.gz before proceeding with backup.
04-28-2017 11:02:03 DEBUG: Sending create local backup command to CmdSubsystem
04-29-2017 05:01:01 DEBUG: Running scheduled SSH backup ...
04-29-2017 05:01:01 DEBUG: Sending create local backup command to CmdSubsystem
04-29-2017 11:02:01 DEBUG: Running scheduled local backup ...
04-29-2017 11:02:01 INFO: Creating a local backup: nagiosxi.1493481721
04-29-2017 11:02:01 INFO: Too many backups! Limit is 7. Removing: nagiosxi.1462118522.tar.gz before proceeding with backup.
04-29-2017 11:02:03 DEBUG: Sending create local backup command to CmdSubsystem
04-30-2017 05:01:01 DEBUG: Running scheduled SSH backup ...
04-30-2017 05:01:01 DEBUG: Sending create local backup command to CmdSubsystem
04-30-2017 11:02:01 DEBUG: Running scheduled local backup ...
04-30-2017 11:02:01 INFO: Creating a local backup: nagiosxi.1493568121
04-30-2017 11:02:01 INFO: Too many backups! Limit is 7. Removing: nagiosxi.1462118522.tar.gz before proceeding with backup.
04-30-2017 11:02:11 DEBUG: Sending create local backup command to CmdSubsystem
05-01-2017 10:40:01 DEBUG: Running scheduled SSH backup ...
05-01-2017 10:40:01 DEBUG: Sending create local backup command to CmdSubsystem
05-01-2017 11:02:01 DEBUG: Running scheduled local backup ...
05-01-2017 11:02:01 INFO: Creating a local backup: nagiosxi.1493654521
05-01-2017 11:02:01 INFO: Too many backups! Limit is 7. Removing: nagiosxi.1483290121.tar.gz before proceeding with backup.
05-01-2017 11:02:02 DEBUG: Sending create local backup command to CmdSubsystem

[ssax]

Try this one:

Code: Select all

echo "SELECT * FROM xi_options WHERE name LIKE '%backup%';" | psql nagiosxi nagiosxi

Please run this command and send me the output:
- Point it to wherever you're storing the backups

Code: Select all

ls -l /store/backups/nagiosxi
ls -ld /store/backups/nagiosxi

Also, sometimes if the XI server doesn't have enough disk space left for the temporary files it needs for the .tar.gz file and for the mysql temporary tables it can cause this issue.


Thank you

[ssax]

In addition to my previous post, please send screenshots of your Admin > Scheduled Backups page and any of the tabs that are active.

Thank you

[emartine]

PM send and attached is a screenshot. The ssh is sending to the DR server we created.

[ssax]

So is it the SSH one or the local backup one that is failing to delete them? If it's the SSH ones, please send a screenshot of that SSH tab as well.

Thank you

[emartine]

it is the local backup on the host that is failing to delete them. The remote host is fine.

[SteveBeauchemin]

I had a similar situation a short while ago.

On the remote system where the tar.gz files are landing, can you extract the data? Basically, is the tar file complete and okay?

Mine were not. It turn out with my site that I had a large amount of Perf files to tar up. It took too long, and the system tried to move the tar file to the remote host before it was actually ready.

I needed to add some variable to one of my php files to fix it. Once that was done, the tar files on the remote server were complete and I could extract them.

This may not be your situation, but if your remote backup files will not extract, you may want to make this change.
https://support.nagios.com/forum/viewto ... 16&t=42075

Good luck... I'll be watching this for answers too.

Thanks

Steve B

[ssax]

Please run these commands and see if the issue is resolved when it runs as scheduled:

Code: Select all

mkdir /usr/local/backups/OLD
mv /usr/local/backups/* OLD/

You may want to delete what you don't need to free up space from this directory so that the backup scripts have enough room to work:

Code: Select all

mkdir /usr/local/backups/OLD

I believe this may fix the problem (I saw this exact issue before) but we'll have to wait and see.

Thank you