Backup directory consuming large amounts of disk space

[lmiltchev]

Your backup directory is in root... (/nagios/nagiosl1p). Either, change the ownership of "/nagios/nagiosl1p" to root:root or create a different backup directory in "/home/nagiosl/", and test the SCP transfer again.
You have some backups already in this folder. I wonder if you (at one point of time) changed the permissions of the backup directory...

[jkinning]

I've changed the /nagios/nagiosl1p directory to root:root still doesn't work. I even chmod'd 777 the /nagios and still nothing. Finally tried the nagiosl /home directory and still nothing.

[tgriep]

Try changing the permissions on the remote system to the following.

Code: Select all

chmod a+rwx /home/nagiosl/
chown nagiosl.users /home/nagiosl/

Then in the XI system, click on the Test SCP button and if you get any errors, take a look at the following log files on the remote system to see what is wrong.

Code: Select all

/var/log/messages
/var/log/secure

[jkinning]

Same thing. My test nagios instance is working just fine.

Code: Select all

 /nagios/nagiosl1t/
total 2608292
-rwxrwxr-x 1 nagiosl users  196422630 Oct  1 23:51 nagiosxi.1443757801.tar.gz
-rwxrwxr-x 1 nagiosl users  196239501 Oct  2 23:51 nagiosxi.1443844201.tar.gz
-rwxrwxr-x 1 nagiosl users  196800391 Oct  3 23:51 nagiosxi.1443930601.tar.gz
-rwxrwxr-x 1 nagiosl users  196581568 Oct  4 23:51 nagiosxi.1444017001.tar.gz
-rwxrwxr-x 1 nagiosl users  255865828 Oct  5 23:50 nagiosxi.1444103401.tar.gz
-rw-r--r-- 1 nagiosl users  256422042 Oct  6 23:50 nagiosxi.1444189801.tar.gz
-rw-r--r-- 1 nagiosl users 1369900363 Oct  7 00:33 nagiosxi.1444190281.tar.gz

Same server with directories having the same permissions.

Code: Select all

ll /nagios/
total 12
drwxrwxr-x 2 nagiosl users 4096 Jun 30  2014 drnagiosl1p
drwxrwxr-x 2 nagiosl users 4096 Oct  5 12:44 nagiosl1p
drwxrwxr-x 2 nagiosl users 4096 Oct  7 00:33 nagiosl1t

[jkinning]

Could all these old backup files be creating the problem? I thought these go away but all they do is keep accumulating and taking up space. My test instance doesn't have all these.

Test

Code: Select all

ll /store/backups/nagiosxi/
total 191816
-rw-r--r-- 1 nagios nagios 196415360 Oct  5 12:10 autoupgrade_backup..tar.gz

Production

Code: Select all

ll /store/backups/nagiosxi/
total 138888688
-rw-r--r-- 1 nagios nagios  633960092 Apr 29 11:35 autoupgrade_backup..tar.gz
-rw-r--r-- 1 nagios nagios  821061045 May 19 00:17 nagiosxi.1432007401.tar.gz
-rw-r--r-- 1 nagios nagios  859258223 May 21 00:21 nagiosxi.1432180202.tar.gz
-rw-r--r-- 1 nagios nagios  943817196 May 30 00:31 nagiosxi.1432957802.tar.gz
-rw-r--r-- 1 nagios nagios  981802581 Jun  5 00:17 nagiosxi.1433476201.tar.gz
-rw-r--r-- 1 nagios nagios  988506980 Jun  6 00:21 nagiosxi.1433562601.tar.gz
-rw-r--r-- 1 nagios nagios 1029082878 Jun 13 00:20 nagiosxi.1434167401.tar.gz
-rw-r--r-- 1 nagios nagios 1032426155 Jun 14 00:17 nagiosxi.1434253802.tar.gz
-rw-r--r-- 1 nagios nagios 1081603695 Jun 20 00:20 nagiosxi.1434772201.tar.gz
-rw-r--r-- 1 nagios nagios 1086431754 Jun 21 00:16 nagiosxi.1434858601.tar.gz
-rw-r--r-- 1 nagios nagios 1104600989 Jun 24 00:21 nagiosxi.1435117801.tar.gz
-rw-r--r-- 1 nagios nagios 1118482058 Jun 25 00:21 nagiosxi.1435204202.tar.gz
-rw-r--r-- 1 nagios nagios 1126164150 Jun 26 00:24 nagiosxi.1435290601.tar.gz
-rw-r--r-- 1 nagios nagios 1132862441 Jun 27 00:23 nagiosxi.1435377002.tar.gz
-rw-r--r-- 1 nagios nagios 1136418456 Jun 28 00:20 nagiosxi.1435463401.tar.gz
-rw-r--r-- 1 nagios nagios 1138704044 Jun 29 00:18 nagiosxi.1435549802.tar.gz
-rw-r--r-- 1 nagios nagios 1131099527 Jun 30 00:18 nagiosxi.1435636201.tar.gz
-rw-r--r-- 1 nagios nagios 1151329753 Jul  1 00:21 nagiosxi.1435722601.tar.gz
-rw-r--r-- 1 nagios nagios 1165562044 Jul  2 00:22 nagiosxi.1435809001.tar.gz
-rw-r--r-- 1 nagios nagios 1173981850 Jul  3 00:20 nagiosxi.1435895401.tar.gz
-rw-r--r-- 1 nagios nagios 1179951773 Jul  4 00:43 nagiosxi.1435981801.tar.gz
-rw-r--r-- 1 nagios nagios 1183425706 Jul  5 00:56 nagiosxi.1436068201.tar.gz
-rw-r--r-- 1 nagios nagios 1186308130 Jul  6 00:45 nagiosxi.1436154603.tar.gz
-rw-r--r-- 1 nagios nagios 1173412830 Jul  7 00:20 nagiosxi.1436241001.tar.gz
-rw-r--r-- 1 nagios nagios 1190384743 Jul  8 00:24 nagiosxi.1436327401.tar.gz
-rw-r--r-- 1 nagios nagios 1202922944 Jul  9 00:21 nagiosxi.1436413801.tar.gz
-rw-r--r-- 1 nagios nagios 1208354538 Jul 10 00:19 nagiosxi.1436500201.tar.gz
-rw-r--r-- 1 nagios nagios 1214434998 Jul 11 00:49 nagiosxi.1436586601.tar.gz
-rw-r--r-- 1 nagios nagios 1215561128 Jul 12 00:22 nagiosxi.1436673001.tar.gz
-rw-r--r-- 1 nagios nagios 1216160312 Jul 13 00:23 nagiosxi.1436759402.tar.gz
-rw-r--r-- 1 nagios nagios 1200685871 Jul 14 00:23 nagiosxi.1436845801.tar.gz
-rw-r--r-- 1 nagios nagios 1218738470 Jul 15 00:31 nagiosxi.1436932201.tar.gz
-rw-r--r-- 1 nagios nagios 1230393344 Jul 16 00:20 nagiosxi.1437018601.tar.gz
-rw-r--r-- 1 nagios nagios 1236342402 Jul 17 00:21 nagiosxi.1437105001.tar.gz
-rw-r--r-- 1 nagios nagios 1241322503 Jul 18 00:57 nagiosxi.1437191401.tar.gz
-rw-r--r-- 1 nagios nagios 1241393977 Jul 19 00:28 nagiosxi.1437277801.tar.gz
-rw-r--r-- 1 nagios nagios 1242883091 Jul 20 00:25 nagiosxi.1437364201.tar.gz
-rw-r--r-- 1 nagios nagios 1228473130 Jul 21 00:24 nagiosxi.1437450602.tar.gz
-rw-r--r-- 1 nagios nagios 1245413178 Jul 22 00:25 nagiosxi.1437537001.tar.gz
-rw-r--r-- 1 nagios nagios 1255992718 Jul 23 00:21 nagiosxi.1437623402.tar.gz
-rw-r--r-- 1 nagios nagios 1259240596 Jul 24 00:22 nagiosxi.1437709802.tar.gz
-rw-r--r-- 1 nagios nagios 1263501602 Jul 25 01:09 nagiosxi.1437796201.tar.gz
-rw-r--r-- 1 nagios nagios 1264952464 Jul 26 01:03 nagiosxi.1437882601.tar.gz
-rw-r--r-- 1 nagios nagios 1265339326 Jul 27 00:55 nagiosxi.1437969001.tar.gz
-rw-r--r-- 1 nagios nagios 1247569060 Jul 28 00:25 nagiosxi.1438055401.tar.gz
-rw-r--r-- 1 nagios nagios 1262107009 Jul 29 00:26 nagiosxi.1438141801.tar.gz
-rw-r--r-- 1 nagios nagios 1271746018 Jul 30 00:24 nagiosxi.1438228202.tar.gz
-rw-r--r-- 1 nagios nagios 1273991696 Jul 31 00:24 nagiosxi.1438314601.tar.gz
-rw-r--r-- 1 nagios nagios 1278528705 Aug  1 01:05 nagiosxi.1438401003.tar.gz
-rw-r--r-- 1 nagios nagios 1278948117 Aug  2 00:24 nagiosxi.1438487401.tar.gz
-rw-r--r-- 1 nagios nagios 1278025549 Aug  3 00:18 nagiosxi.1438573802.tar.gz
-rw-r--r-- 1 nagios nagios 1263203494 Aug  4 00:24 nagiosxi.1438660202.tar.gz
-rw-r--r-- 1 nagios nagios 1279753862 Aug  5 00:24 nagiosxi.1438746601.tar.gz
-rw-r--r-- 1 nagios nagios 1288843998 Aug  6 00:24 nagiosxi.1438833002.tar.gz
-rw-r--r-- 1 nagios nagios 1291919993 Aug  7 00:26 nagiosxi.1438919401.tar.gz
-rw-r--r-- 1 nagios nagios 1295615929 Aug  8 01:06 nagiosxi.1439005801.tar.gz
-rw-r--r-- 1 nagios nagios 1297024919 Aug  9 00:56 nagiosxi.1439092201.tar.gz
-rw-r--r-- 1 nagios nagios 1297419703 Aug 10 00:40 nagiosxi.1439178601.tar.gz
-rw-r--r-- 1 nagios nagios 1280285021 Aug 11 00:23 nagiosxi.1439265001.tar.gz
-rw-r--r-- 1 nagios nagios 1293591149 Aug 12 00:25 nagiosxi.1439351401.tar.gz
-rw-r--r-- 1 nagios nagios 1300711572 Aug 13 00:24 nagiosxi.1439437801.tar.gz
-rw-r--r-- 1 nagios nagios 1299684058 Aug 14 00:21 nagiosxi.1439524201.tar.gz
-rw-r--r-- 1 nagios nagios 1302333407 Aug 15 00:26 nagiosxi.1439610601.tar.gz
-rw-r--r-- 1 nagios nagios 1301607952 Aug 16 00:25 nagiosxi.1439697001.tar.gz
-rw-r--r-- 1 nagios nagios 1298039721 Aug 17 00:20 nagiosxi.1439783401.tar.gz
-rw-r--r-- 1 nagios nagios 1281647903 Aug 18 00:21 nagiosxi.1439869801.tar.gz
-rw-r--r-- 1 nagios nagios 1297065387 Aug 19 00:23 nagiosxi.1439956201.tar.gz
-rw-r--r-- 1 nagios nagios 1305194743 Aug 20 00:22 nagiosxi.1440042601.tar.gz
-rw-r--r-- 1 nagios nagios 1303103503 Aug 21 00:22 nagiosxi.1440129001.tar.gz
-rw-r--r-- 1 nagios nagios 1306666076 Aug 22 00:28 nagiosxi.1440215401.tar.gz
-rw-r--r-- 1 nagios nagios 1306626965 Aug 23 00:23 nagiosxi.1440301802.tar.gz
-rw-r--r-- 1 nagios nagios 1305429337 Aug 24 00:23 nagiosxi.1440388201.tar.gz
-rw-r--r-- 1 nagios nagios 1287557412 Aug 25 00:21 nagiosxi.1440474601.tar.gz
-rw-r--r-- 1 nagios nagios 1302470655 Aug 26 00:22 nagiosxi.1440561001.tar.gz
-rw-r--r-- 1 nagios nagios 1309993219 Aug 27 00:20 nagiosxi.1440647402.tar.gz
-rw-r--r-- 1 nagios nagios 1312302383 Aug 28 00:21 nagiosxi.1440733801.tar.gz
-rw-r--r-- 1 nagios nagios 1316051651 Aug 29 00:31 nagiosxi.1440820201.tar.gz
-rw-r--r-- 1 nagios nagios 1316458735 Aug 30 00:23 nagiosxi.1440906602.tar.gz
-rw-r--r-- 1 nagios nagios 1315253208 Aug 31 00:23 nagiosxi.1440993001.tar.gz
-rw-r--r-- 1 nagios nagios 1299175375 Sep  1 00:21 nagiosxi.1441079401.tar.gz
-rw-r--r-- 1 nagios nagios 1315203285 Sep  2 00:23 nagiosxi.1441165801.tar.gz
-rw-r--r-- 1 nagios nagios 1324693306 Sep  3 00:23 nagiosxi.1441252201.tar.gz
-rw-r--r-- 1 nagios nagios 1326749835 Sep  4 00:23 nagiosxi.1441338601.tar.gz
-rw-r--r-- 1 nagios nagios 1330517028 Sep  5 00:34 nagiosxi.1441425001.tar.gz
-rw-r--r-- 1 nagios nagios 1330538804 Sep  6 00:25 nagiosxi.1441511401.tar.gz
-rw-r--r-- 1 nagios nagios 1329630539 Sep  7 00:23 nagiosxi.1441597802.tar.gz
-rw-r--r-- 1 nagios nagios 1314025069 Sep  8 00:19 nagiosxi.1441684201.tar.gz
-rw-r--r-- 1 nagios nagios 1327938919 Sep  9 00:20 nagiosxi.1441770601.tar.gz
-rw-r--r-- 1 nagios nagios 1334871930 Sep 10 01:02 nagiosxi.1441857002.tar.gz
-rw-r--r-- 1 nagios nagios 1338088978 Sep 11 00:21 nagiosxi.1441943401.tar.gz
-rw-r--r-- 1 nagios nagios 1342149209 Sep 12 01:01 nagiosxi.1442029803.tar.gz
-rw-r--r-- 1 nagios nagios 1338724922 Sep 13 00:22 nagiosxi.1442116201.tar.gz
-rw-r--r-- 1 nagios nagios 1337246546 Sep 14 00:15 nagiosxi.1442202601.tar.gz
-rw-r--r-- 1 nagios nagios 1322252053 Sep 15 00:22 nagiosxi.1442289001.tar.gz
-rw-r--r-- 1 nagios nagios 1336310264 Sep 16 00:23 nagiosxi.1442375401.tar.gz
-rw-r--r-- 1 nagios nagios 1344095035 Sep 17 00:18 nagiosxi.1442461801.tar.gz
-rw-r--r-- 1 nagios nagios 1348464350 Sep 18 00:19 nagiosxi.1442548201.tar.gz
-rw-r--r-- 1 nagios nagios 1146195968 Sep 19 00:31 nagiosxi.1442634601.tar.gz
-rw-r--r-- 1 nagios nagios 1330374090 Sep 21 00:24 nagiosxi.1442807402.tar.gz
-rw-r--r-- 1 nagios nagios 1313347751 Sep 22 00:22 nagiosxi.1442893801.tar.gz
-rw-r--r-- 1 nagios nagios 1341042662 Sep 23 00:28 nagiosxi.1442980201.tar.gz
-rw-r--r-- 1 nagios nagios 1341038650 Sep 23 00:36 nagiosxi.1442980741.tar.gz
-rw-r--r-- 1 nagios nagios 1349360652 Sep 24 00:25 nagiosxi.1443066601.tar.gz
-rw-r--r-- 1 nagios nagios 1349353969 Sep 24 00:32 nagiosxi.1443067141.tar.gz
-rw-r--r-- 1 nagios nagios 1351415861 Sep 25 00:21 nagiosxi.1443153002.tar.gz
-rw-r--r-- 1 nagios nagios 1355051847 Sep 26 00:25 nagiosxi.1443239401.tar.gz
-rw-r--r-- 1 nagios nagios 1354982591 Sep 27 00:25 nagiosxi.1443325801.tar.gz
-rw-r--r-- 1 nagios nagios 1337636072 Sep 29 00:23 nagiosxi.1443498601.tar.gz
-rw-r--r-- 1 nagios nagios 1351165197 Sep 30 00:23 nagiosxi.1443585001.tar.gz
-rw-r--r-- 1 nagios nagios   84339241 Sep 30 12:32 nagiosxi.1443629521.tar.gz
-rw-r--r-- 1 nagios nagios 1369639190 Oct  2 08:01 nagiosxi.1443785401.tar.gz
-rw-r--r-- 1 nagios nagios 1373580341 Oct  3 08:16 nagiosxi.1443871801.tar.gz
-rw-r--r-- 1 nagios nagios 1374094915 Oct  4 08:09 nagiosxi.1443958202.tar.gz
-rw-r--r-- 1 nagios nagios 1350256789 Oct  5 08:15 nagiosxi.1444044603.tar.gz
-rw-r--r-- 1 nagios nagios 1358153599 Oct  6 08:24 nagiosxi.1444131002.tar.gz

[tgriep]

You can remove the old backups from the XI system and see if it works for you.
Can you login to the XI system in a shell and see if you can scp a file to that remote folder using the nagiosl credentials?
Maybe that will give us a better idea on why it is not working.

[jkinning]

I was able to manually scp the file over. yet it looks like now the daily backups aren't happening on my prod instance. Last one was the 6th.

View from the remote server

Code: Select all

ll nagios/nagiosl1p/
total 5610876
-rwxrwxr-x 1 nagiosl users  927645696 Jun 19 00:13 nagiosxi.1434685802.tar.gz
-rwxrwxr-x 1 nagiosl users 1032716288 Jun 22 00:14 nagiosxi.1434945002.tar.gz
-rwxrwxr-x 1 nagiosl users  459210752 Jun 23 00:14 nagiosxi.1435031401.tar.gz
-rwxrwxr-x 1 nagiosl users  509640704 Sep 28 00:15 nagiosxi.1443412201.tar.gz
-rwxrwxr-x 1 nagiosl users   84340376 Sep 30 12:32 nagiosxi.1443629401.tar.gz
-rwxrwxr-x 1 nagiosl users 1368170581 Oct  1 07:54 nagiosxi.1443699001.tar.gz
-rw-r--r-- 1 nagiosl users 1358153599 Oct  6 08:24 nagiosxi.1444131002.tar.gz

ll /nagios/nagiosl1t/
total 3822776
-rwxrwxr-x 1 nagiosl users  196800391 Oct  3 23:51 nagiosxi.1443930601.tar.gz
-rwxrwxr-x 1 nagiosl users  196581568 Oct  4 23:51 nagiosxi.1444017001.tar.gz
-rwxrwxr-x 1 nagiosl users  255865828 Oct  5 23:50 nagiosxi.1444103401.tar.gz
-rw-r--r-- 1 nagiosl users  256422042 Oct  6 23:50 nagiosxi.1444189801.tar.gz
-rw-r--r-- 1 nagiosl users 1369900363 Oct  7 00:33 nagiosxi.1444190281.tar.gz
-rw-r--r-- 1 nagiosl users  256255148 Oct  7 23:50 nagiosxi.1444276201.tar.gz
-rw-r--r-- 1 nagiosl users 1378822381 Oct  8 00:48 nagiosxi.1444276681.tar.gz

Last few backups in the /store/backups/nagiosxi directory

Code: Select all

-rw-r--r-- 1 nagios nagios 1337636072 Sep 29 00:23 nagiosxi.1443498601.tar.gz
-rw-r--r-- 1 nagios nagios 1351165197 Sep 30 00:23 nagiosxi.1443585001.tar.gz
-rw-r--r-- 1 nagios nagios   84339241 Sep 30 12:32 nagiosxi.1443629521.tar.gz
-rw-r--r-- 1 nagios nagios 1369639190 Oct  2 08:01 nagiosxi.1443785401.tar.gz
-rw-r--r-- 1 nagios nagios 1373580341 Oct  3 08:16 nagiosxi.1443871801.tar.gz
-rw-r--r-- 1 nagios nagios 1374094915 Oct  4 08:09 nagiosxi.1443958202.tar.gz
-rw-r--r-- 1 nagios nagios 1350256789 Oct  5 08:15 nagiosxi.1444044603.tar.gz
-rw-r--r-- 1 nagios nagios 1358153599 Oct  6 08:24 nagiosxi.1444131002.tar.gz

I'm going to delete them all in the /store/backups/nagiosxi directory and see what happens. So, these should be cleaning themselves up, right? Wondering why these keep increasing.

[tgriep]

The Scheduled backups are controlled by the cron daemon. Try restarting it by running

Code: Select all

service crond restart

The way the backups are deleted, it will only delete the last 7 (In your settings) from the time it was run. It will not go back and delete the backups older than that setting.

[jkinning]

I removed all but the last 14 days of backups in /store/backups/nagiosxi, restarted cron, and today's backup went to my remote server. WooHoo!

I'll give it a couple more days just to make sure all is working well and let you know when this thread can be locked. I just wish I knew what caused the problem and what actually fixed it.

Once all is well I'll remove all the other backup files in the /store/backups/nagiosxi as I don't have local backups enabled so the backups should get exported to my remote server.

[tgriep]

That is good to hear. We'll keep it open if you have any more issues.