Page 1 of 5
Backup directory consuming large amounts of disk space
Posted: Sun Sep 20, 2015 12:02 pm
by jkinning
I had our NagiosXI VM instance go south due to lack of disk space. When I examined what was taking up the space it was the /store/backup/nagiosxi directory which was using 118G, see image attached. I was able to add more space to the root LVM but was wondering if these are all needed and if there was a cleanup job that may not be working or a cleanup job that needs to be created? I do have a scheduled backup task to ssh to a different server that will then get backed up to tape by our NetBackup system.
I am just trying to prevent this issue in the future.
I noticed this is actually the local backup archive but I don't have anything configured under the Local tab within the scheduled backup just the ssh tab which I have set to 7 days. Would setting the Local tab with the /store/backup/nagiosxi directory and only keep X amount of backups eliminate this issue?
Re: Backup directory consuming large amounts of disk space
Posted: Mon Sep 21, 2015 10:02 am
by jdalrymple
The backups at play are ones we build in for you to saver your behind if you blow up your databases. If you're confident that you have good backups of your databases offline you can safely remove the backup jobs and the backups themselves.
You might dig in beforehand just to make sure that you only have a weeks worth - it's possible that the cleanup isn't happening.
The jobs are here:
/etc/cron.d/nagiosxi
automysqlbackup and autopostgresqlbackup
Re: Backup directory consuming large amounts of disk space
Posted: Mon Sep 21, 2015 2:45 pm
by jkinning
Those directories /store/backups/mysql and /store/backups/postgresql are fine. It is the /store/backups/nagiosxi which has a lot of nagios.xi.<number>.tar.gz files in them. Where is that cleanup script at?
Re: Backup directory consuming large amounts of disk space
Posted: Mon Sep 21, 2015 4:49 pm
by jdalrymple
Those would be "local backups." You said that you don't have any scheduled though.
How old are these?
The "cleanup script" is part of routing maintenance, not callable at the CLI to the best of my knowledge. Cleanup can be handled in the Admin section of the XI interface under "Local Backups Archive"
Re: Backup directory consuming large amounts of disk space
Posted: Tue Sep 22, 2015 6:01 am
by jkinning
They are several months old.
Code: Select all
ls -l /store/backups/nagiosxi/
total 122468780
-rw-r--r-- 1 nagios nagios 633960092 Apr 29 11:35 autoupgrade_backup..tar.gz
-rw-r--r-- 1 nagios nagios 708437620 May 7 00:29 nagiosxi.1430970601.tar.gz
-rw-r--r-- 1 nagios nagios 716844388 May 8 00:22 nagiosxi.1431057002.tar.gz
-rw-r--r-- 1 nagios nagios 801918903 May 16 00:41 nagiosxi.1431748202.tar.gz
-rw-r--r-- 1 nagios nagios 821061045 May 19 00:17 nagiosxi.1432007401.tar.gz
-rw-r--r-- 1 nagios nagios 859258223 May 21 00:21 nagiosxi.1432180202.tar.gz
-rw-r--r-- 1 nagios nagios 943817196 May 30 00:31 nagiosxi.1432957802.tar.gz
-rw-r--r-- 1 nagios nagios 981802581 Jun 5 00:17 nagiosxi.1433476201.tar.gz
-rw-r--r-- 1 nagios nagios 988506980 Jun 6 00:21 nagiosxi.1433562601.tar.gz
-rw-r--r-- 1 nagios nagios 1029082878 Jun 13 00:20 nagiosxi.1434167401.tar.gz
-rw-r--r-- 1 nagios nagios 1032426155 Jun 14 00:17 nagiosxi.1434253802.tar.gz
-rw-r--r-- 1 nagios nagios 1081603695 Jun 20 00:20 nagiosxi.1434772201.tar.gz
-rw-r--r-- 1 nagios nagios 1086431754 Jun 21 00:16 nagiosxi.1434858601.tar.gz
-rw-r--r-- 1 nagios nagios 1104600989 Jun 24 00:21 nagiosxi.1435117801.tar.gz
-rw-r--r-- 1 nagios nagios 1118482058 Jun 25 00:21 nagiosxi.1435204202.tar.gz
-rw-r--r-- 1 nagios nagios 1126164150 Jun 26 00:24 nagiosxi.1435290601.tar.gz
-rw-r--r-- 1 nagios nagios 1132862441 Jun 27 00:23 nagiosxi.1435377002.tar.gz
-rw-r--r-- 1 nagios nagios 1136418456 Jun 28 00:20 nagiosxi.1435463401.tar.gz
-rw-r--r-- 1 nagios nagios 1138704044 Jun 29 00:18 nagiosxi.1435549802.tar.gz
-rw-r--r-- 1 nagios nagios 1131099527 Jun 30 00:18 nagiosxi.1435636201.tar.gz
-rw-r--r-- 1 nagios nagios 1151329753 Jul 1 00:21 nagiosxi.1435722601.tar.gz
-rw-r--r-- 1 nagios nagios 1165562044 Jul 2 00:22 nagiosxi.1435809001.tar.gz
-rw-r--r-- 1 nagios nagios 1173981850 Jul 3 00:20 nagiosxi.1435895401.tar.gz
-rw-r--r-- 1 nagios nagios 1179951773 Jul 4 00:43 nagiosxi.1435981801.tar.gz
-rw-r--r-- 1 nagios nagios 1183425706 Jul 5 00:56 nagiosxi.1436068201.tar.gz
-rw-r--r-- 1 nagios nagios 1186308130 Jul 6 00:45 nagiosxi.1436154603.tar.gz
-rw-r--r-- 1 nagios nagios 1173412830 Jul 7 00:20 nagiosxi.1436241001.tar.gz
-rw-r--r-- 1 nagios nagios 1190384743 Jul 8 00:24 nagiosxi.1436327401.tar.gz
-rw-r--r-- 1 nagios nagios 1202922944 Jul 9 00:21 nagiosxi.1436413801.tar.gz
-rw-r--r-- 1 nagios nagios 1208354538 Jul 10 00:19 nagiosxi.1436500201.tar.gz
-rw-r--r-- 1 nagios nagios 1214434998 Jul 11 00:49 nagiosxi.1436586601.tar.gz
-rw-r--r-- 1 nagios nagios 1215561128 Jul 12 00:22 nagiosxi.1436673001.tar.gz
-rw-r--r-- 1 nagios nagios 1216160312 Jul 13 00:23 nagiosxi.1436759402.tar.gz
-rw-r--r-- 1 nagios nagios 1200685871 Jul 14 00:23 nagiosxi.1436845801.tar.gz
-rw-r--r-- 1 nagios nagios 1218738470 Jul 15 00:31 nagiosxi.1436932201.tar.gz
-rw-r--r-- 1 nagios nagios 1230393344 Jul 16 00:20 nagiosxi.1437018601.tar.gz
-rw-r--r-- 1 nagios nagios 1236342402 Jul 17 00:21 nagiosxi.1437105001.tar.gz
-rw-r--r-- 1 nagios nagios 1241322503 Jul 18 00:57 nagiosxi.1437191401.tar.gz
-rw-r--r-- 1 nagios nagios 1241393977 Jul 19 00:28 nagiosxi.1437277801.tar.gz
-rw-r--r-- 1 nagios nagios 1242883091 Jul 20 00:25 nagiosxi.1437364201.tar.gz
-rw-r--r-- 1 nagios nagios 1228473130 Jul 21 00:24 nagiosxi.1437450602.tar.gz
-rw-r--r-- 1 nagios nagios 1245413178 Jul 22 00:25 nagiosxi.1437537001.tar.gz
-rw-r--r-- 1 nagios nagios 1255992718 Jul 23 00:21 nagiosxi.1437623402.tar.gz
-rw-r--r-- 1 nagios nagios 1259240596 Jul 24 00:22 nagiosxi.1437709802.tar.gz
-rw-r--r-- 1 nagios nagios 1263501602 Jul 25 01:09 nagiosxi.1437796201.tar.gz
-rw-r--r-- 1 nagios nagios 1264952464 Jul 26 01:03 nagiosxi.1437882601.tar.gz
-rw-r--r-- 1 nagios nagios 1265339326 Jul 27 00:55 nagiosxi.1437969001.tar.gz
-rw-r--r-- 1 nagios nagios 1247569060 Jul 28 00:25 nagiosxi.1438055401.tar.gz
-rw-r--r-- 1 nagios nagios 1262107009 Jul 29 00:26 nagiosxi.1438141801.tar.gz
-rw-r--r-- 1 nagios nagios 1271746018 Jul 30 00:24 nagiosxi.1438228202.tar.gz
-rw-r--r-- 1 nagios nagios 1273991696 Jul 31 00:24 nagiosxi.1438314601.tar.gz
-rw-r--r-- 1 nagios nagios 1278528705 Aug 1 01:05 nagiosxi.1438401003.tar.gz
-rw-r--r-- 1 nagios nagios 1278948117 Aug 2 00:24 nagiosxi.1438487401.tar.gz
-rw-r--r-- 1 nagios nagios 1278025549 Aug 3 00:18 nagiosxi.1438573802.tar.gz
-rw-r--r-- 1 nagios nagios 1263203494 Aug 4 00:24 nagiosxi.1438660202.tar.gz
-rw-r--r-- 1 nagios nagios 1279753862 Aug 5 00:24 nagiosxi.1438746601.tar.gz
-rw-r--r-- 1 nagios nagios 1288843998 Aug 6 00:24 nagiosxi.1438833002.tar.gz
-rw-r--r-- 1 nagios nagios 1291919993 Aug 7 00:26 nagiosxi.1438919401.tar.gz
-rw-r--r-- 1 nagios nagios 1295615929 Aug 8 01:06 nagiosxi.1439005801.tar.gz
-rw-r--r-- 1 nagios nagios 1297024919 Aug 9 00:56 nagiosxi.1439092201.tar.gz
-rw-r--r-- 1 nagios nagios 1297419703 Aug 10 00:40 nagiosxi.1439178601.tar.gz
-rw-r--r-- 1 nagios nagios 1280285021 Aug 11 00:23 nagiosxi.1439265001.tar.gz
-rw-r--r-- 1 nagios nagios 1293591149 Aug 12 00:25 nagiosxi.1439351401.tar.gz
-rw-r--r-- 1 nagios nagios 1300711572 Aug 13 00:24 nagiosxi.1439437801.tar.gz
-rw-r--r-- 1 nagios nagios 1299684058 Aug 14 00:21 nagiosxi.1439524201.tar.gz
-rw-r--r-- 1 nagios nagios 1302333407 Aug 15 00:26 nagiosxi.1439610601.tar.gz
-rw-r--r-- 1 nagios nagios 1301607952 Aug 16 00:25 nagiosxi.1439697001.tar.gz
-rw-r--r-- 1 nagios nagios 1298039721 Aug 17 00:20 nagiosxi.1439783401.tar.gz
-rw-r--r-- 1 nagios nagios 1281647903 Aug 18 00:21 nagiosxi.1439869801.tar.gz
-rw-r--r-- 1 nagios nagios 1297065387 Aug 19 00:23 nagiosxi.1439956201.tar.gz
-rw-r--r-- 1 nagios nagios 1305194743 Aug 20 00:22 nagiosxi.1440042601.tar.gz
-rw-r--r-- 1 nagios nagios 1303103503 Aug 21 00:22 nagiosxi.1440129001.tar.gz
-rw-r--r-- 1 nagios nagios 1306666076 Aug 22 00:28 nagiosxi.1440215401.tar.gz
-rw-r--r-- 1 nagios nagios 1306626965 Aug 23 00:23 nagiosxi.1440301802.tar.gz
-rw-r--r-- 1 nagios nagios 1305429337 Aug 24 00:23 nagiosxi.1440388201.tar.gz
-rw-r--r-- 1 nagios nagios 1287557412 Aug 25 00:21 nagiosxi.1440474601.tar.gz
-rw-r--r-- 1 nagios nagios 1302470655 Aug 26 00:22 nagiosxi.1440561001.tar.gz
-rw-r--r-- 1 nagios nagios 1309993219 Aug 27 00:20 nagiosxi.1440647402.tar.gz
-rw-r--r-- 1 nagios nagios 1312302383 Aug 28 00:21 nagiosxi.1440733801.tar.gz
-rw-r--r-- 1 nagios nagios 1316051651 Aug 29 00:31 nagiosxi.1440820201.tar.gz
-rw-r--r-- 1 nagios nagios 1316458735 Aug 30 00:23 nagiosxi.1440906602.tar.gz
-rw-r--r-- 1 nagios nagios 1315253208 Aug 31 00:23 nagiosxi.1440993001.tar.gz
-rw-r--r-- 1 nagios nagios 1299175375 Sep 1 00:21 nagiosxi.1441079401.tar.gz
-rw-r--r-- 1 nagios nagios 1315203285 Sep 2 00:23 nagiosxi.1441165801.tar.gz
-rw-r--r-- 1 nagios nagios 1324693306 Sep 3 00:23 nagiosxi.1441252201.tar.gz
-rw-r--r-- 1 nagios nagios 1326749835 Sep 4 00:23 nagiosxi.1441338601.tar.gz
-rw-r--r-- 1 nagios nagios 1330517028 Sep 5 00:34 nagiosxi.1441425001.tar.gz
-rw-r--r-- 1 nagios nagios 1330538804 Sep 6 00:25 nagiosxi.1441511401.tar.gz
-rw-r--r-- 1 nagios nagios 1329630539 Sep 7 00:23 nagiosxi.1441597802.tar.gz
-rw-r--r-- 1 nagios nagios 1314025069 Sep 8 00:19 nagiosxi.1441684201.tar.gz
-rw-r--r-- 1 nagios nagios 1327938919 Sep 9 00:20 nagiosxi.1441770601.tar.gz
-rw-r--r-- 1 nagios nagios 1334871930 Sep 10 01:02 nagiosxi.1441857002.tar.gz
-rw-r--r-- 1 nagios nagios 1338088978 Sep 11 00:21 nagiosxi.1441943401.tar.gz
-rw-r--r-- 1 nagios nagios 1342149209 Sep 12 01:01 nagiosxi.1442029803.tar.gz
-rw-r--r-- 1 nagios nagios 1338724922 Sep 13 00:22 nagiosxi.1442116201.tar.gz
-rw-r--r-- 1 nagios nagios 1337246546 Sep 14 00:15 nagiosxi.1442202601.tar.gz
-rw-r--r-- 1 nagios nagios 1322252053 Sep 15 00:22 nagiosxi.1442289001.tar.gz
-rw-r--r-- 1 nagios nagios 1336310264 Sep 16 00:23 nagiosxi.1442375401.tar.gz
-rw-r--r-- 1 nagios nagios 1344095035 Sep 17 00:18 nagiosxi.1442461801.tar.gz
-rw-r--r-- 1 nagios nagios 1348464350 Sep 18 00:19 nagiosxi.1442548201.tar.gz
-rw-r--r-- 1 nagios nagios 1146195968 Sep 19 00:31 nagiosxi.1442634601.tar.gz
drwxr-xr-x 4 nagios users 4096 Sep 20 00:21 nagiosxi.1442721001
-rw-r--r-- 1 nagios nagios 1330374090 Sep 21 00:24 nagiosxi.1442807402.tar.gz
-rw-r--r-- 1 nagios nagios 1313347751 Sep 22 00:22 nagiosxi.1442893801.tar.gz
I see them but I is there a more efficient method to delete these, cleanup, without having to hit the delete on each one under the Admin section?
Re: Backup directory consuming large amounts of disk space
Posted: Tue Sep 22, 2015 9:01 am
by tgriep
You could delete them from the unix command line by running the command below from the /store/backups/nagiosxi folder.
It will delete all of the backups from May to August 19th without you asking you to confirm you want them deleted.
If this is not what you want, then don't run it.
After that, make sure your XI system is setup to only keep so many backups and you shouldn't run out of space in the future.
Re: Backup directory consuming large amounts of disk space
Posted: Tue Sep 22, 2015 9:03 am
by lmiltchev
I noticed this is actually the local backup archive but I don't have anything configured under the Local tab within the scheduled backup just the ssh tab which I have set to 7 days. Would setting the Local tab with the /store/backup/nagiosxi directory and only keep X amount of backups eliminate this issue?
What is the version of Nagios XI and the "Scheduled Backups" component that you are currently using? Do "SSH Scheduled Backups" work as expected for you? Are the backups transferred to the remote server? Is the the "Backup Limit" value of 7 honored?
Can you show screenshots of the "SSH" and "Local" tab pages under the "Scheduled Backups"? Hide sensitive info.
Re: Backup directory consuming large amounts of disk space
Posted: Tue Sep 22, 2015 2:52 pm
by jkinning
Nagios XI 2014R2.7 is the version I am running.
Well, Nagios shows that it was working but I checked the server which I was sending the SSH backups to and looks like it stopped back in June but shows it ran.
Do I need to configure the local backup in order for the cleanup stuff to work?
Re: Backup directory consuming large amounts of disk space
Posted: Tue Sep 22, 2015 2:58 pm
by tgriep
The ssh backup creates a local copy and then it copies it over to the remote system. If the local backup and the ssh backup happen at the same time, that may cause a problem.
To be sure, can you look at this log file for any errors and or post it here?
Code: Select all
/usr/local/nagiosxi/var/components/scheduledbackups.log
Re: Backup directory consuming large amounts of disk space
Posted: Tue Sep 22, 2015 3:00 pm
by jkinning
Code: Select all
09-13-2015 23:50:01 DEBUG: Running scheduled SSH backup ...
09-13-2015 23:50:01 DEBUG: Sending create local backup command to CmdSubsystem
09-14-2015 00:12:40 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist
09-14-2015 23:50:01 DEBUG: Running scheduled SSH backup ...
09-14-2015 23:50:01 DEBUG: Sending create local backup command to CmdSubsystem
09-15-2015 00:12:40 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist
09-15-2015 23:50:01 DEBUG: Running scheduled SSH backup ...
09-15-2015 23:50:01 DEBUG: Sending create local backup command to CmdSubsystem
09-16-2015 00:12:44 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist
09-16-2015 23:50:01 DEBUG: Running scheduled SSH backup ...
09-16-2015 23:50:01 DEBUG: Sending create local backup command to CmdSubsystem
09-17-2015 00:12:41 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist
09-17-2015 23:50:01 DEBUG: Running scheduled SSH backup ...
09-17-2015 23:50:01 DEBUG: Sending create local backup command to CmdSubsystem
09-18-2015 00:12:39 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist
09-18-2015 23:50:01 DEBUG: Running scheduled SSH backup ...
09-18-2015 23:50:01 DEBUG: Sending create local backup command to CmdSubsystem
09-19-2015 00:15:54 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist
09-19-2015 23:50:01 DEBUG: Running scheduled SSH backup ...
09-19-2015 23:50:01 DEBUG: Sending create local backup command to CmdSubsystem
09-20-2015 00:15:48 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist
09-20-2015 23:50:01 DEBUG: Running scheduled SSH backup ...
09-20-2015 23:50:02 DEBUG: Sending create local backup command to CmdSubsystem
09-21-2015 00:15:49 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist
09-21-2015 23:50:01 DEBUG: Running scheduled SSH backup ...
09-21-2015 23:50:01 DEBUG: Sending create local backup command to CmdSubsystem
09-22-2015 00:15:51 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist
Log is full of these messages