Updated maintenance settings but indices still not deleting

[CFT6Server]

ok, this is missing the actual line ?

Code: Select all

python `find /usr/lib/python* -name curator.py` "$@"

[rkennedy]

Ah, looks like your path is hardcoded. Take what I originally posted, and replace curator "$@" with python `find /usr/lib/python* -name curator.py` "$@"

They are essentially the same, except it may have been hard coded to find the path for curator in previous versions.

[CFT6Server]

So looks like it is still working without the modifications. Here is the debug file.

Repository is currently holding data since August 14th.

Code: Select all

[root@~]# curl localhost:9200/_snapshot/Repository/_all?pretty | grep indices
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
    "indices" : [ "logstash-2016.08.14" ],  0 --:--:-- --:--:-- --:--:--     0
100    "indices" : [ "logstash-2016.08.15" ],
 2    "indices" : [ "logstash-2016.08.16" ],
39    "indices" : [ "logstash-2016.08.17" ],
80    "indices" : [ "logstash-2016.08.18" ],
      "indices" : [ "logstash-2016.08.19" ],
1    "indices" : [ "logstash-2016.08.20" ],
00    "indices" : [ "logstash-2016.08.21" ],
 2    "indices" : [ "logstash-2016.08.22" ],
39    "indices" : [ "logstash-2016.08.23" ],
80    "indices" : [ "logstash-2016.08.24" ],
      "indices" : [ "logstash-2016.08.25" ],
      "indices" : [ "logstash-2016.08.26" ],
0    "indices" : [ "logstash-2016.08.27" ],
      "indices" : [ "logstash-2016.08.28" ],
      "indices" : [ "logstash-2016.08.29" ],
 0    "indices" : [ "logstash-2016.08.30" ],
      "indices" : [ "logstash-2016.08.31" ],
 5    "indices" : [ "logstash-2016.09.01" ],
8    "indices" : [ "logstash-2016.09.02" ],
7    "indices" : [ "logstash-2016.09.03" ],
k    "indices" : [ "logstash-2016.09.04" ],
     "indices" : [ "logstash-2016.09.05" ],
     "indices" : [ "logstash-2016.09.06" ],
     "indices" : [ "logstash-2016.09.07" ],
     "indices" : [ "logstash-2016.09.08" ],
     "indices" : [ "logstash-2016.09.09" ],
     "indices" : [ "logstash-2016.09.10" ],
     "indices" : [ "logstash-2016.09.11" ],
:-    "indices" : [ "logstash-2016.09.12" ],
- -    "indices" : [ "logstash-2016.09.13" ],
-:    "indices" : [ "logstash-2016.09.14" ],
-    "indices" : [ "logstash-2016.09.15" ],
-:    "indices" : [ "logstash-2016.09.16" ],
--    "indices" : [ "logstash-2016.09.17" ],
     "indices" : [ "logstash-2016.09.18" ],
-    "indices" : [ "logstash-2016.09.19" ],
-:    "indices" : [ "logstash-2016.09.20" ],
--    "indices" : [ "logstash-2016.09.21" ],
:    "indices" : [ "logstash-2016.09.22" ],
--    "indices" : [ "logstash-2016.09.23" ],
     "indices" : [ "logstash-2016.09.24" ],
 6    "indices" : [ "logstash-2016.09.25" ],
1    "indices" : [ "logstash-2016.09.26" ],
6k    "indices" : [ "logstash-2016.09.27" ],
    "indices" : [ "logstash-2016.09.28" ],
    "indices" : [ "logstash-2016.09.29" ],

    "indices" : [ "logstash-2016.09.30" ],
    "indices" : [ "logstash-2016.10.01" ],
    "indices" : [ "logstash-2016.10.02" ],
    "indices" : [ "logstash-2016.10.03" ],
    "indices" : [ "logstash-2010.11.21" ],
    "indices" : [ "logstash-2013.11.14" ],
    "indices" : [ "logstash-2016.10.04" ],
    "indices" : [ "logstash-2016.10.05" ],

[jomann]

Basically the way the settings work is that each snapshot will hold the last 15 days of data, for 45 days - that is what your settings on the first page showed. So in theory you should see 45 "snapshots" and each of those snapshots will have your indexes that you were backing up. It's a little bit confusing how it is laid out in the GUI and even when using curator. Did changing from 60 -> 45 days cause actual snapshot deletion?

[CFT6Server]

Yes the setting was changed from 60 to 45, but the storage and usage remained the same and is what brought me to review this. There are currently 55 snapshots in the repo, so it is not clearing what it is supposed to. Prior to this, I did 'reset' all jobs in the command subsystem, which I needed to do in the past. So it is closing the indices at 15 days fine, just not purging the repository properly.

[mcapra]

From the guy who developed curator:

As long as the segment is referenced by any other snapshot in the system, deleting a snapshot will not delete the referenced data.

What I suspect is that, somehow, one of your snapshots is still referencing those indices for some reason. This is entirely possible if the maintenance settings were only changed recently. Can I see the full output of the following command without the grep?

Code: Select all

curl localhost:9200/_snapshot/Repository/_all?pretty

[CFT6Server]

From the guy who developed curator:

As long as the segment is referenced by any other snapshot in the system, deleting a snapshot will not delete the referenced data.

What I suspect is that, somehow, one of your snapshots is still referencing those indices for some reason. This is entirely possible if the maintenance settings were only changed recently. Can I see the full output of the following command without the grep?

Code: Select all

curl localhost:9200/_snapshot/Repository/_all?pretty

Here it is. Thanks.

[mcapra]

That confirms my suspicions.

So based on this information:

Previous settings: close index after 14 days / Delete after 30 days / Delete backups after 60 days.
New settings: close index after 7 days / Delete after 15 days / Delete backups after 45 days.

There are likely still indices in that repository that are on the previous "14 days" setting. Since those snapshots were created when "14 days" worth of indices were open, we would expect the total number of indices to be included in this repository to be 45 + 14 (45 days worth of snapshots, 14 indices max) which is 59 minus 1 for the current days index equals 58. I counted 58 total indices in that repository, which makes sense given the previous math and accounting for an exclusion of the current day's index.

So essentially, it is expected behavior that the repository contain 58 total indices. Once you reach a point in your maintenance settings where all of those old snapshots (with the previous settings) cycle out (which I imagine should be rapidly approaching since your first post was 15 days ago), you should be seeing 51 total indices in your repository moving forward.

[CFT6Server]

Thanks for Clarifying this. So meaning the repository setting takes into account the current indexes as well. So if I want to retain 45 days total and keeping the current closed index settings, I should actually use 30 days instead of 45 days. I suggest clarifying in the GUI or manual. Thanks.

[mcapra]

So meaning the repository setting takes into account the current indexes as well.

Every time a snapshot is created, it contains the data of every open index within elasticsearch. So a snapshot isn't really limited to a single day; It's a snapshot of the current elasticsearch system state.

So if I want to retain 45 days total and keeping the current closed index settings, I should actually use 30 days instead of 45 days.

That's one way to look at it, yeah. I agree the language is a bit vague in terms of how most end-users understand backups.