Log Server Backup Snapshots no getting created

[oxigenservices]

No, the backup snapshots I am able to see is for last 15 days only, starting from current date. These indices I have in live too.

I agree that I have configured 'Delete backups older than' > 35 because as per this configuration it should backup as snapshot of 31st-35th days (5 Days) backup. I want to test it like this for storage space behaviour analysis.

Backup SS.JPG

[mcapra]

Lets backup a bit and discuss what a "snapshot" is in this context. Because the way backups are addressed by the software is confusing within the context of most organizations requirements.

A snapshot is, quite literally, a snapshot of the current elasticsearch database (not entirely unlike a VM snapshot). This means that when a snapshot is taken, almost every single open index is usually included. However, the current day's index is never included in a snapshot. You may notice your 11-16-2016 snapshot does not contain logstash-2016.11.16 for this reason.

Addressing your original requirements:

My requirement is to keep 15 days (1-15) indices live, 15 days (16-30) indices closed

You "Close indexes older than" 15 days, and "Delete indexes older than" 30 days.

30 days (31-60) indices as backup snapshot.

You would set the "Delete backups older than" to 46 days. This would cause some overlap since you would be basically maintaining days 1-60 in your snapshot repository. There isn't really a way to avoid this overlap.

Where I get the 46 days number: 15 days of indices included in each snapshot, excluding the current day's index for any given snapshot (15 - 1 = 14). If I want 60 days worth of data in my repository, 60 - 14 = 46.

Again, super confusing stuff. I'm working on a better way to visually describe retention periods in the Backup & Maintenance page. You're not the first person to have questions

[oxigenservices]

Thanks buddy for elaboration on this. Yes, it is bit confusing but I think we are in a direction that it would be cleared in couple of more discussions.

I understood the calculations for backup snapshots of indices and will take few more days' to analyze behaviour and configure in a way we require.

Please let me know that will it occupy multiple storage space for snapshot of each day's indices, which is already there in previous day's snapshot. And what would be the ideal practice to delete old snapshots.

[mcapra]

Please let me know that will it occupy multiple storage space for snapshot of each day's indices, which is already there in previous day's snapshot.

If i'm understanding your meaning correctly, no there will not be "duplicates" of each index in the repository. Each repository has an "indices" path that contains the indices referenced by each snapshot. So while each snapshot may have overlap in the indices it references, the data is not actually duplicated per snapshot; Just the reference itself.

Here's an example of a repository (located at /var/backups):

Code: Select all

[root@localhost backups]# tree -L 2
.
├── index
├── indices
│   ├── logstash-2016.10.24
│   ├── logstash-2016.10.25
│   ├── logstash-2016.10.26
│   ├── logstash-2016.10.27
│   ├── logstash-2016.10.28
│   ├── logstash-2016.10.29
│   ├── logstash-2016.10.30
│   ├── logstash-2016.10.31
│   ├── logstash-2016.11.01
│   └── logstash-2016.11.17
├── metadata-curator-20161027141546
├── metadata-curator-20161118174220
├── metadata-curator-20161118174359
├── snapshot-curator-20161027141546
├── snapshot-curator-20161118174220
└── snapshot-curator-20161118174359

While snapshot-curator-20161118174220 and snapshot-curator-20161118174359 both reference the exact same indices, there are no duplicate copies of the indices themselves.

And what would be the ideal practice to delete old snapshots.

It's hard to say what "ideal practice" is for deleting old snapshots. This depends entirely on your organization's retention requirements. Generally speaking, the "Delete backups older than X days" setting under Backup & Maintenance should handle removal of the old data.

[oxigenservices]

Thanks Team for the support on my posts. Yes, after observing the backup behavior for few days, it is cleared to me now that how to configure it for orgnization's policy requirement.

[mcapra]

I'm glad to hear that Is it alright if we close this thread and mark the issue as resolved?