XI local back up appears to be going insane

[JakeHatMacys]

So we were using the local back up function to store our back ups in the default directory... but it was filling up our root file system so we wanted to move it to our /usr/local directory...

Upon doing this and kicking off a back up in XI we get these random directory created in /store/backups/nagiosxi

Capture.JPG

It's the 3 random number directories (I moved all my old back ups to a different directory to save space). When entering a directory I see:

Capture1.JPG

These directories took up several gigs within 5 minutes... the only thing I changed in XI was the directory for the back ups and CHMOD 777'd the directory:

Capture2.JPG

And the back ups never make it to the new directory.

Thoughts?

[tgriep]

Could you run the following command, then schedule a backup to happen a few minutes in the future and then post the output?

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

[JakeHatMacys]

So here it is, from the looks of the end failure it didn't take the directory change from the GUI. Do I need to apply the config after??? Now that I think about it I didn't... but I also bolded around the time it really started to fill up the disc space. We have about 8 gigs available on root before running this...

[root@esu1l268 backups]# tail -f /usr/local/nagiosxi/var/cmdsubsys.log
........................................................
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
..........................................................
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
...........PROCESSING COMMAND ID 320...
PROCESS COMMAND: CMD=1117, DATA=
CMDLINE=/usr/local/nagiosxi/scripts/backup_xi.sh
Backing up Core Config Manager (NagiosQL)...
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up Nagios Core...
tar: Removing leading `/' from member names
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
....................................................tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated

PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.......................................................tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated

PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
tar: /usr/local/nagios/share/perfdata/esu2v213: file changed as we read it
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
..
PROCESSED 0 COMMANDS
tar: /usr/local/nagios/var/ndo.sock: socket ignored
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
tar: /usr/local/nagios/var: file changed as we read it
Backing up Nagios XI...
tar: Removing leading `/' from member names
Backing up MRTG...
tar: Removing leading `/' from member names
Backing up NRDP...
tar: Removing leading `/' from member names
Backing up MySQL databases...
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated

PROCESSED 0 COMMANDS
Backing up PostgresQL databases...
Backing up logrotate config files...
Backing up Apache config files...
Compressing backup...
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
..
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS

gzip: stdout: No space left on device
tar: 1426620493.tar.gz: Wrote only 2048 of 10240 bytes
tar: Error is not recoverable: exiting now
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS

===============
BACKUP COMPLETE
===============
Backup stored in /store/backups/nagiosxi/1426620493.tar.gz
OUTPUT=Backup stored in /store/backups/nagiosxi/1426620493.tar.gz
RETURNCODE=0

PROCESSED 1 COMMANDS

[JakeHatMacys]

Oddly enough it created a back up file I wouldn't count on being vaild in the older directory /store/backups/nagiosxi/1426620493.tar.gz

But it's also 3.99 gigs in size and our back ups before the move were between 500mb's to 1GB. So that's interesting.

[cmerchant]

The backup files get put into a temporary directory, and after successful gzip'ng the files into the backup directory the backup script will remove the temporary directory and its contents. You should manually remove the temporary backup directories and the contents of them. And then review what backups you have stored in the original location to see if you need to delete those as well (they won't be deleted automatically through the backup script).

[JakeHatMacys]

The backup files get put into a temporary directory, and after successful gzip'ng the files into the backup directory the backup script will remove the temporary directory and its contents. You should manually remove the temporary backup directories and the contents of them. And then review what backups you have stored in the original location to see if you need to delete those as well (they won't be deleted automatically through the backup script).

I guess where I'm confused is since changing the directory in XI off the the root file system is resulting in it getting filled by 1 back up and then ultimately failing. And we have 8gb's available:

l$ df -m
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/localvg00-lv_slash
16631 7433 8355 48% /

Which actually we had more like 12gb's available as a back up of 3.99 gb's is stored in the /store/backups/nagiosxi folder.

Which shouldn't of happened since I changed the local directory... so it seems like the Local file path change just doesn't work?

Edit: Looking at it again, when I change the local file path on the schedule I guess I assumed it would re-route all the local back ups... now I'm thinking that's only for the scheduled back ups and not the ones I'm manually kicking off?

[cmerchant]

It seemed too obvious, but the location of /usr/local/nagios/var/backups gets included in the scheduled backup, so the backup will start growing exponentially.

I replicated the process and got the same results. The backup is a (backup of the (of the backup (of the backup))....

Code: Select all

-rw-r--r-- 1 nagios nagios 108M Mar 18 10:50 nagiosxi.1426693743.tar.gz
-rw-r--r-- 1 nagios nagios 216M Mar 18 10:54 nagiosxi.1426693982.tar.gz
-rw-r--r-- 1 nagios nagios 755M Mar 18 10:58 nagiosxi.1426694103.tar.gz

Do not use the /usr/local/nagios/ as part of the path for your backups.

[JakeHatMacys]

It seemed too obvious, but the location of /usr/local/nagios/var/backups gets included in the scheduled backup, so the backup will start growing exponentially.

I replicated the process and got the same results. The backup is a (backup of the (of the backup (of the backup))....

Code: Select all

-rw-r--r-- 1 nagios nagios 108M Mar 18 10:50 nagiosxi.1426693743.tar.gz
-rw-r--r-- 1 nagios nagios 216M Mar 18 10:54 nagiosxi.1426693982.tar.gz
-rw-r--r-- 1 nagios nagios 755M Mar 18 10:58 nagiosxi.1426694103.tar.gz

Do not use the /usr/local/nagios/ as part of the path for your backups.

I changed it back and still having the issue, is there something I need to scrub to bring me back to normal?

What I'm thinking is just leaving it alone when I can get it fixed and just scripting a move to different directory at a later time.

[cmerchant]

You need to manually purge the backups that were created in /usr/local/nagios/var/backups:

Code: Select all

rm -rf /usr/local/nagios/var/backups/nagiosxi*tar.gz

because those backups located in the /usr/local/nagios/* are still embedded into your existing backups into the directory /store/backups/

[JakeHatMacys]

Okay, looks like that worked. Thanks I was mildly freaking out since we really needed a back up before we connect this to our BSM haha.

I still don't have my head wrapped around why it was nesting the back ups but I'll just manually move them off root Bi-Weekly. Eventually we can just script it.