Backup directory consuming large amounts of disk space

[tgriep]

Take a look at the local directory on the XI system and see if there is a directory leftover dated June 23rd.
Delete it and see if the ssh backup works on the next schedule.
The local directory should be the default of /store/backups/nagiosxi

[jkinning]

I don't see anything

Code: Select all

ll /store/backups/nagiosxi
total 124396172
-rw-r--r-- 1 nagios nagios  633960092 Apr 29 11:35 autoupgrade_backup..tar.gz
-rw-r--r-- 1 nagios nagios  716844388 May  8 00:22 nagiosxi.1431057002.tar.gz
-rw-r--r-- 1 nagios nagios  801918903 May 16 00:41 nagiosxi.1431748202.tar.gz
-rw-r--r-- 1 nagios nagios  821061045 May 19 00:17 nagiosxi.1432007401.tar.gz
-rw-r--r-- 1 nagios nagios  859258223 May 21 00:21 nagiosxi.1432180202.tar.gz
-rw-r--r-- 1 nagios nagios  943817196 May 30 00:31 nagiosxi.1432957802.tar.gz
-rw-r--r-- 1 nagios nagios  981802581 Jun  5 00:17 nagiosxi.1433476201.tar.gz
-rw-r--r-- 1 nagios nagios  988506980 Jun  6 00:21 nagiosxi.1433562601.tar.gz
-rw-r--r-- 1 nagios nagios 1029082878 Jun 13 00:20 nagiosxi.1434167401.tar.gz
-rw-r--r-- 1 nagios nagios 1032426155 Jun 14 00:17 nagiosxi.1434253802.tar.gz
-rw-r--r-- 1 nagios nagios 1081603695 Jun 20 00:20 nagiosxi.1434772201.tar.gz
-rw-r--r-- 1 nagios nagios 1086431754 Jun 21 00:16 nagiosxi.1434858601.tar.gz
-rw-r--r-- 1 nagios nagios 1104600989 Jun 24 00:21 nagiosxi.1435117801.tar.gz
-rw-r--r-- 1 nagios nagios 1118482058 Jun 25 00:21 nagiosxi.1435204202.tar.gz
-rw-r--r-- 1 nagios nagios 1126164150 Jun 26 00:24 nagiosxi.1435290601.tar.gz
-rw-r--r-- 1 nagios nagios 1132862441 Jun 27 00:23 nagiosxi.1435377002.tar.gz
-rw-r--r-- 1 nagios nagios 1136418456 Jun 28 00:20 nagiosxi.1435463401.tar.gz
-rw-r--r-- 1 nagios nagios 1138704044 Jun 29 00:18 nagiosxi.1435549802.tar.gz
-rw-r--r-- 1 nagios nagios 1131099527 Jun 30 00:18 nagiosxi.1435636201.tar.gz
-rw-r--r-- 1 nagios nagios 1151329753 Jul  1 00:21 nagiosxi.1435722601.tar.gz
-rw-r--r-- 1 nagios nagios 1165562044 Jul  2 00:22 nagiosxi.1435809001.tar.gz
-rw-r--r-- 1 nagios nagios 1173981850 Jul  3 00:20 nagiosxi.1435895401.tar.gz
-rw-r--r-- 1 nagios nagios 1179951773 Jul  4 00:43 nagiosxi.1435981801.tar.gz
-rw-r--r-- 1 nagios nagios 1183425706 Jul  5 00:56 nagiosxi.1436068201.tar.gz
-rw-r--r-- 1 nagios nagios 1186308130 Jul  6 00:45 nagiosxi.1436154603.tar.gz
-rw-r--r-- 1 nagios nagios 1173412830 Jul  7 00:20 nagiosxi.1436241001.tar.gz
-rw-r--r-- 1 nagios nagios 1190384743 Jul  8 00:24 nagiosxi.1436327401.tar.gz
-rw-r--r-- 1 nagios nagios 1202922944 Jul  9 00:21 nagiosxi.1436413801.tar.gz
-rw-r--r-- 1 nagios nagios 1208354538 Jul 10 00:19 nagiosxi.1436500201.tar.gz
-rw-r--r-- 1 nagios nagios 1214434998 Jul 11 00:49 nagiosxi.1436586601.tar.gz
-rw-r--r-- 1 nagios nagios 1215561128 Jul 12 00:22 nagiosxi.1436673001.tar.gz
-rw-r--r-- 1 nagios nagios 1216160312 Jul 13 00:23 nagiosxi.1436759402.tar.gz
-rw-r--r-- 1 nagios nagios 1200685871 Jul 14 00:23 nagiosxi.1436845801.tar.gz
-rw-r--r-- 1 nagios nagios 1218738470 Jul 15 00:31 nagiosxi.1436932201.tar.gz
-rw-r--r-- 1 nagios nagios 1230393344 Jul 16 00:20 nagiosxi.1437018601.tar.gz
-rw-r--r-- 1 nagios nagios 1236342402 Jul 17 00:21 nagiosxi.1437105001.tar.gz
-rw-r--r-- 1 nagios nagios 1241322503 Jul 18 00:57 nagiosxi.1437191401.tar.gz
-rw-r--r-- 1 nagios nagios 1241393977 Jul 19 00:28 nagiosxi.1437277801.tar.gz
-rw-r--r-- 1 nagios nagios 1242883091 Jul 20 00:25 nagiosxi.1437364201.tar.gz
-rw-r--r-- 1 nagios nagios 1228473130 Jul 21 00:24 nagiosxi.1437450602.tar.gz
-rw-r--r-- 1 nagios nagios 1245413178 Jul 22 00:25 nagiosxi.1437537001.tar.gz
-rw-r--r-- 1 nagios nagios 1255992718 Jul 23 00:21 nagiosxi.1437623402.tar.gz
-rw-r--r-- 1 nagios nagios 1259240596 Jul 24 00:22 nagiosxi.1437709802.tar.gz
-rw-r--r-- 1 nagios nagios 1263501602 Jul 25 01:09 nagiosxi.1437796201.tar.gz
-rw-r--r-- 1 nagios nagios 1264952464 Jul 26 01:03 nagiosxi.1437882601.tar.gz
-rw-r--r-- 1 nagios nagios 1265339326 Jul 27 00:55 nagiosxi.1437969001.tar.gz
-rw-r--r-- 1 nagios nagios 1247569060 Jul 28 00:25 nagiosxi.1438055401.tar.gz
-rw-r--r-- 1 nagios nagios 1262107009 Jul 29 00:26 nagiosxi.1438141801.tar.gz
-rw-r--r-- 1 nagios nagios 1271746018 Jul 30 00:24 nagiosxi.1438228202.tar.gz
-rw-r--r-- 1 nagios nagios 1273991696 Jul 31 00:24 nagiosxi.1438314601.tar.gz
-rw-r--r-- 1 nagios nagios 1278528705 Aug  1 01:05 nagiosxi.1438401003.tar.gz
-rw-r--r-- 1 nagios nagios 1278948117 Aug  2 00:24 nagiosxi.1438487401.tar.gz
-rw-r--r-- 1 nagios nagios 1278025549 Aug  3 00:18 nagiosxi.1438573802.tar.gz
-rw-r--r-- 1 nagios nagios 1263203494 Aug  4 00:24 nagiosxi.1438660202.tar.gz
-rw-r--r-- 1 nagios nagios 1279753862 Aug  5 00:24 nagiosxi.1438746601.tar.gz
-rw-r--r-- 1 nagios nagios 1288843998 Aug  6 00:24 nagiosxi.1438833002.tar.gz
-rw-r--r-- 1 nagios nagios 1291919993 Aug  7 00:26 nagiosxi.1438919401.tar.gz
-rw-r--r-- 1 nagios nagios 1295615929 Aug  8 01:06 nagiosxi.1439005801.tar.gz
-rw-r--r-- 1 nagios nagios 1297024919 Aug  9 00:56 nagiosxi.1439092201.tar.gz
-rw-r--r-- 1 nagios nagios 1297419703 Aug 10 00:40 nagiosxi.1439178601.tar.gz
-rw-r--r-- 1 nagios nagios 1280285021 Aug 11 00:23 nagiosxi.1439265001.tar.gz
-rw-r--r-- 1 nagios nagios 1293591149 Aug 12 00:25 nagiosxi.1439351401.tar.gz
-rw-r--r-- 1 nagios nagios 1300711572 Aug 13 00:24 nagiosxi.1439437801.tar.gz
-rw-r--r-- 1 nagios nagios 1299684058 Aug 14 00:21 nagiosxi.1439524201.tar.gz
-rw-r--r-- 1 nagios nagios 1302333407 Aug 15 00:26 nagiosxi.1439610601.tar.gz
-rw-r--r-- 1 nagios nagios 1301607952 Aug 16 00:25 nagiosxi.1439697001.tar.gz
-rw-r--r-- 1 nagios nagios 1298039721 Aug 17 00:20 nagiosxi.1439783401.tar.gz
-rw-r--r-- 1 nagios nagios 1281647903 Aug 18 00:21 nagiosxi.1439869801.tar.gz
-rw-r--r-- 1 nagios nagios 1297065387 Aug 19 00:23 nagiosxi.1439956201.tar.gz
-rw-r--r-- 1 nagios nagios 1305194743 Aug 20 00:22 nagiosxi.1440042601.tar.gz
-rw-r--r-- 1 nagios nagios 1303103503 Aug 21 00:22 nagiosxi.1440129001.tar.gz
-rw-r--r-- 1 nagios nagios 1306666076 Aug 22 00:28 nagiosxi.1440215401.tar.gz
-rw-r--r-- 1 nagios nagios 1306626965 Aug 23 00:23 nagiosxi.1440301802.tar.gz
-rw-r--r-- 1 nagios nagios 1305429337 Aug 24 00:23 nagiosxi.1440388201.tar.gz
-rw-r--r-- 1 nagios nagios 1287557412 Aug 25 00:21 nagiosxi.1440474601.tar.gz
-rw-r--r-- 1 nagios nagios 1302470655 Aug 26 00:22 nagiosxi.1440561001.tar.gz
-rw-r--r-- 1 nagios nagios 1309993219 Aug 27 00:20 nagiosxi.1440647402.tar.gz
-rw-r--r-- 1 nagios nagios 1312302383 Aug 28 00:21 nagiosxi.1440733801.tar.gz
-rw-r--r-- 1 nagios nagios 1316051651 Aug 29 00:31 nagiosxi.1440820201.tar.gz
-rw-r--r-- 1 nagios nagios 1316458735 Aug 30 00:23 nagiosxi.1440906602.tar.gz
-rw-r--r-- 1 nagios nagios 1315253208 Aug 31 00:23 nagiosxi.1440993001.tar.gz
-rw-r--r-- 1 nagios nagios 1299175375 Sep  1 00:21 nagiosxi.1441079401.tar.gz
-rw-r--r-- 1 nagios nagios 1315203285 Sep  2 00:23 nagiosxi.1441165801.tar.gz
-rw-r--r-- 1 nagios nagios 1324693306 Sep  3 00:23 nagiosxi.1441252201.tar.gz
-rw-r--r-- 1 nagios nagios 1326749835 Sep  4 00:23 nagiosxi.1441338601.tar.gz
-rw-r--r-- 1 nagios nagios 1330517028 Sep  5 00:34 nagiosxi.1441425001.tar.gz
-rw-r--r-- 1 nagios nagios 1330538804 Sep  6 00:25 nagiosxi.1441511401.tar.gz
-rw-r--r-- 1 nagios nagios 1329630539 Sep  7 00:23 nagiosxi.1441597802.tar.gz
-rw-r--r-- 1 nagios nagios 1314025069 Sep  8 00:19 nagiosxi.1441684201.tar.gz
-rw-r--r-- 1 nagios nagios 1327938919 Sep  9 00:20 nagiosxi.1441770601.tar.gz
-rw-r--r-- 1 nagios nagios 1334871930 Sep 10 01:02 nagiosxi.1441857002.tar.gz
-rw-r--r-- 1 nagios nagios 1338088978 Sep 11 00:21 nagiosxi.1441943401.tar.gz
-rw-r--r-- 1 nagios nagios 1342149209 Sep 12 01:01 nagiosxi.1442029803.tar.gz
-rw-r--r-- 1 nagios nagios 1338724922 Sep 13 00:22 nagiosxi.1442116201.tar.gz
-rw-r--r-- 1 nagios nagios 1337246546 Sep 14 00:15 nagiosxi.1442202601.tar.gz
-rw-r--r-- 1 nagios nagios 1322252053 Sep 15 00:22 nagiosxi.1442289001.tar.gz
-rw-r--r-- 1 nagios nagios 1336310264 Sep 16 00:23 nagiosxi.1442375401.tar.gz
-rw-r--r-- 1 nagios nagios 1344095035 Sep 17 00:18 nagiosxi.1442461801.tar.gz
-rw-r--r-- 1 nagios nagios 1348464350 Sep 18 00:19 nagiosxi.1442548201.tar.gz
-rw-r--r-- 1 nagios nagios 1146195968 Sep 19 00:31 nagiosxi.1442634601.tar.gz
drwxr-xr-x 4 nagios users        4096 Sep 20 00:21 nagiosxi.1442721001
-rw-r--r-- 1 nagios nagios 1330374090 Sep 21 00:24 nagiosxi.1442807402.tar.gz
-rw-r--r-- 1 nagios nagios 1313347751 Sep 22 00:22 nagiosxi.1442893801.tar.gz
-rw-r--r-- 1 nagios nagios 1341042662 Sep 23 00:28 nagiosxi.1442980201.tar.gz
-rw-r--r-- 1 nagios nagios 1341038650 Sep 23 00:36 nagiosxi.1442980741.tar.gz

There is one dated Sep 20.

[tgriep]

Everything does look good.
Try removing the following bad backup just so it doesn't become an issue in the future.

Code: Select all

drwxr-xr-x 4 nagios users        4096 Sep 20 00:21 nagiosxi.1442721001

Disable the ssh backups, and update the settings, re-enable it and update the settings and see if that starts them running again.

[jkinning]

Still getting the same error message.

[tgriep]

Can you show the permissions of the folder of the remote system by running the following on the remote system?

Code: Select all

ls -l /nagios

[jkinning]

Code: Select all

 ls -l /nagios/
total 12
drwxrwxr-x 2 nagiosl users 4096 Jun 30  2014 drnagiosl1p
drwxrwxr-x 2 nagiosl users 4096 Sep 24 00:16 nagiosl1p
drwxrwxr-x 2 nagiosl users 4096 Sep 23 23:50 nagiosl1t

Code: Select all

ll nagios/nagiosl1p/
total 7257252
-rw-r--r-- 1 nagiosl users 1016064337 Jun 11 00:14 nagiosxi.1433994601.tar.gz
-rw-r--r-- 1 nagiosl users 1021691555 Jun 12 00:14 nagiosxi.1434081002.tar.gz
-rw-r--r-- 1 nagiosl users  388120576 Jun 15 00:15 nagiosxi.1434340201.tar.gz
-rw-r--r-- 1 nagiosl users  790855680 Jun 16 00:13 nagiosxi.1434426601.tar.gz
-rw-r--r-- 1 nagiosl users 1050623122 Jun 17 00:13 nagiosxi.1434513001.tar.gz
-rw-r--r-- 1 nagiosl users  737181696 Jun 18 00:14 nagiosxi.1434599401.tar.gz
-rw-r--r-- 1 nagiosl users  927645696 Jun 19 00:13 nagiosxi.1434685802.tar.gz
-rw-r--r-- 1 nagiosl users 1032716288 Jun 22 00:14 nagiosxi.1434945002.tar.gz
-rw-r--r-- 1 nagiosl users  459210752 Jun 23 00:14 nagiosxi.1435031401.tar.gz
-rw-r--r-- 1 nagiosl users        110 Aug  6  2014 ssh_test_secure_copy.txt

Code: Select all

ll nagios/nagiosl1t/
total 2111544
-rw-r--r-- 1 nagiosl users 196306901 Sep 13 23:51 nagiosxi.1442202601.tar.gz
-rw-r--r-- 1 nagiosl users 196172136 Sep 14 23:51 nagiosxi.1442289001.tar.gz
-rw-r--r-- 1 nagiosl users 196020886 Sep 15 23:51 nagiosxi.1442375402.tar.gz
-rw-r--r-- 1 nagiosl users 196608439 Sep 16 23:51 nagiosxi.1442461802.tar.gz
-rw-r--r-- 1 nagiosl users 196451852 Sep 17 23:51 nagiosxi.1442548201.tar.gz
-rw-r--r-- 1 nagiosl users 196291417 Sep 18 23:50 nagiosxi.1442634601.tar.gz
-rw-r--r-- 1 nagiosl users 196139505 Sep 19 23:50 nagiosxi.1442721001.tar.gz
-rw-r--r-- 1 nagiosl users 196709223 Sep 20 23:53 nagiosxi.1442807401.tar.gz
-rw-r--r-- 1 nagiosl users 196512300 Sep 21 23:50 nagiosxi.1442893801.tar.gz
-rw-r--r-- 1 nagiosl users 196135393 Sep 22 23:50 nagiosxi.1442980201.tar.gz
-rw-r--r-- 1 nagiosl users 196688673 Sep 23 23:50 nagiosxi.1443066601.tar.gz
-rw-r--r-- 1 nagiosl users       110 Jun 30  2014 ssh_test_secure_copy.txt

Strange thing is my test instance is running fine and uses the same exact setup.

[tgriep]

Can you run the following tail command, change the schedule for the backup to happen 5 minutes in to the future, wait for the tail command to show the backup has happened and post the results?

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

[jkinning]

Code: Select all

tailf /usr/local/nagiosxi/var/cmdsubsys.log
.............................................................
PROCESSED 0 COMMANDS
.
PROCESSED 0 COMMANDS

PROCESSED 0 COMMANDS
.PROCESSING COMMAND ID 3706...
PROCESS COMMAND: CMD=1117, DATA=a:2:{i:0;s:19:"nagiosxi.1443629401";i:1;s:0:"";}
CMDLINE=/usr/local/nagiosxi/scripts/backup_xi.sh -n nagiosxi.1443629401
Backing up Core Config Manager (NagiosQL)...
tar: Removing leading `/' from member names

tar: Removing leading `/' from member names
Backing up Nagios Core...
tar: Removing leading `/' from member names
tar: /usr/local/nagios/libexec/check_services: Cannot open: Permission denied
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
tar: /usr/local/nagios/var/ndo.sock: socket ignored
tar: /usr/local/nagios/var: file changed as we read it
.......................tar: /usr/local/nagios/share/perfdata/MQPROD1P: file changed as we read it
tar: /usr/local/nagios/share/perfdata/IPAM3T: file changed as we read it
tar: /usr/local/nagios/share/perfdata/MRSW1D: file changed as we read it
.PROCESSING COMMAND ID 3707...
PROCESS COMMAND: CMD=1119, DATA=nagiosxi.1431748202
CMDLINE=rm -rf /store/backups/nagiosxi/nagiosxi.1431748202.tar.gz
OUTPUT=
RETURNCODE=0
.PROCESSING COMMAND ID 3708...
PROCESS COMMAND: CMD=1117, DATA=a:2:{i:0;s:19:"nagiosxi.1443629521";i:1;s:24:"/store/backups/nagiosxi/";}
CMDLINE=/usr/local/nagiosxi/scripts/backup_xi.sh -n nagiosxi.1443629521 -d /store/backups/nagiosxi
Backing up Core Config Manager (NagiosQL)...
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up Nagios Core...
tar: Removing leading `/' from member names
tar: /usr/local/nagios/libexec/check_services: Cannot open: Permission denied
tar: /usr/local/nagios/var/nagios.tmpddgwPU: File removed before we read it
tar: /usr/local/nagios/var/service-perfdata: file changed as we read it
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
tar: /usr/local/nagios/var/ndo.sock: socket ignored
tar: /usr/local/nagios/var: file changed as we read it

[lmiltchev]

You have some permissions issues:

tar: /usr/local/nagios/libexec/check_services: Cannot open: Permission denied

Can you run the following commands as root to see if this is going to fix the issue?

Code: Select all

cd /usr/local/nagiosxi/scripts/
./reset_config_perms.sh

[jkinning]

Code: Select all

[root@nagios]# /usr/local/nagiosxi/scripts/reset_config_perms.sh 
RESETTING PERMS
[root@nagios]# tailf /usr/local/nagiosxi/var/cmdsubsys.log
...............PROCESSED 0 COMMANDS
............................................................
PROCESSED 0 COMMANDS
............................................................
PROCESSED 0 COMMANDS
.............................................................
PROCESSED 0 COMMANDS

PROCESSED 0 COMMANDS
............................................................
PROCESSED 0 COMMANDS
.............................................................
PROCESSED 0 COMMANDS
osxi.1443699001";i:1;s:0:"";}
CMDLINE=/usr/local/nagiosxi/scripts/backup_xi.sh -n nagiosxi.1443699001
Backing up Core Config Manager (NagiosQL)...
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up Nagios Core...
tar: Removing leading `/' from member names
tar: /usr/local/nagios/libexec/check_services: Cannot open: Permission denied
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
tar: /usr/local/nagios/var/ndo.sock: socket ignored
tar: /usr/local/nagios/var: file changed as we read it
...............................................tar: /usr/local/nagios/share/perfdata/UNITY2T: file changed as we read it
.
PROCESSED 0 COMMANDS