Archiving Data

[CFT6Server]

I need to make sure I keep a couple days of indices to archive. I have the repository configured. What files do I need to grab to ensure that I get everything I need to pull from archive if needed? To be clear, I need to copy from repository to another archive for long term retention.

In the backup repository there is an index file and then metadata-logstash-<date> files and then indices directory of each daily index.

(Also from looking at the repository, looks like the older folders aren't removed. The indices and data is gone but the folders aren't ie. /indices/logstash-2015.05.09 etc.)

[jolson]

Copy all of the data stored in your backup directory to your archive. This will be your initial archive. For each archive that you need to do beyond the first archive, copy the 'index' file, located at /some/repository/index, as well as the appropriate metadata files and indices files.

Example for initial backup:
Change directory to your current backup:

Code: Select all

cd /mnt/nlsback

Copy all files recursively, retaining permissions:

Code: Select all

cp -Rp * /mnt/archive

Example for additional backups:
Change directory to your current backup:

Code: Select all

cd /mnt/nlsback

Copy select files, retaining permissions:

Code: Select all

cp -R snapshot-logstash-2015.09.07 index indices/logstash-2015.09.07/ /mnt/archive/

You could certainly setup a cron job to achieve this on a schedule. Keep in mind that the 'index' file *does* change sometimes, so it's necessary to copy it over and overwrite the old index file.

[CFT6Server]

I have about 35 backups in the repository, but I only require the middle 3 or 4 daily indices. Do I still need the original copy in the archive?

[jolson]

No you don't - the only files you need are the up-to-date 'index' file, the metadata-logstash-xxx.xx.xx file, and the indices/logstash-xxxx.xx.xx directory (including all of the files inside of that directory of course).

[CFT6Server]

Thanks. I will give this a try. I'll spin up a test instance and test the restore.

[jolson]

Let me know how it goes. Thanks!