backup_maintenance is not clearing old backups

mike4vr · Post by **mike4vr** » Fri Jul 29, 2016 11:44 am

I recently changed my "Delete backups older than" option from 180 to 90 days because of policy change and to conserve disk usage. Since then, the 'backup_maintenance' job has ran twice (no errors) but we are not seeing any old indices removed. Here's what the /indices directory looks like:

Code: Select all

logstash-2015.12.06  logstash-2016.01.16  logstash-2016.02.24  logstash-2016.04.03  logstash-2016.05.12  logstash-2016.06.20
logstash-2015.12.07  logstash-2016.01.17  logstash-2016.02.25  logstash-2016.04.04  logstash-2016.05.13  logstash-2016.06.21
logstash-2015.12.08  logstash-2016.01.18  logstash-2016.02.26  logstash-2016.04.05  logstash-2016.05.14  logstash-2016.06.22
logstash-2015.12.09  logstash-2016.01.19  logstash-2016.02.27  logstash-2016.04.06  logstash-2016.05.15  logstash-2016.06.23
logstash-2015.12.10  logstash-2016.01.20  logstash-2016.02.28  logstash-2016.04.07  logstash-2016.05.16  logstash-2016.06.24
logstash-2015.12.11  logstash-2016.01.21  logstash-2016.02.29  logstash-2016.04.08  logstash-2016.05.17  logstash-2016.06.25
logstash-2015.12.12  logstash-2016.01.22  logstash-2016.03.01  logstash-2016.04.09  logstash-2016.05.18  logstash-2016.06.26
logstash-2015.12.13  logstash-2016.01.23  logstash-2016.03.02  logstash-2016.04.10  logstash-2016.05.19  logstash-2016.06.27
logstash-2015.12.14  logstash-2016.01.24  logstash-2016.03.03  logstash-2016.04.11  logstash-2016.05.20  logstash-2016.06.28
logstash-2015.12.15  logstash-2016.01.25  logstash-2016.03.04  logstash-2016.04.12  logstash-2016.05.21  logstash-2016.06.29
logstash-2015.12.16  logstash-2016.01.26  logstash-2016.03.05  logstash-2016.04.13  logstash-2016.05.22  logstash-2016.06.30
logstash-2015.12.17  logstash-2016.01.27  logstash-2016.03.06  logstash-2016.04.14  logstash-2016.05.23  logstash-2016.07.01
logstash-2015.12.18  logstash-2016.01.28  logstash-2016.03.07  logstash-2016.04.15  logstash-2016.05.24  logstash-2016.07.02
logstash-2015.12.19  logstash-2016.01.29  logstash-2016.03.08  logstash-2016.04.16  logstash-2016.05.25  logstash-2016.07.03
logstash-2015.12.20  logstash-2016.01.30  logstash-2016.03.09  logstash-2016.04.17  logstash-2016.05.26  logstash-2016.07.04
logstash-2015.12.21  logstash-2016.01.31  logstash-2016.03.10  logstash-2016.04.18  logstash-2016.05.27  logstash-2016.07.05
logstash-2015.12.22  logstash-2016.02.01  logstash-2016.03.11  logstash-2016.04.19  logstash-2016.05.28  logstash-2016.07.06
logstash-2015.12.23  logstash-2016.02.02  logstash-2016.03.12  logstash-2016.04.20  logstash-2016.05.29  logstash-2016.07.07
logstash-2015.12.24  logstash-2016.02.03  logstash-2016.03.13  logstash-2016.04.21  logstash-2016.05.30  logstash-2016.07.08
logstash-2015.12.25  logstash-2016.02.04  logstash-2016.03.14  logstash-2016.04.22  logstash-2016.05.31  logstash-2016.07.09
logstash-2015.12.26  logstash-2016.02.05  logstash-2016.03.15  logstash-2016.04.23  logstash-2016.06.01  logstash-2016.07.10
logstash-2015.12.27  logstash-2016.02.06  logstash-2016.03.16  logstash-2016.04.24  logstash-2016.06.02  logstash-2016.07.11
logstash-2015.12.28  logstash-2016.02.07  logstash-2016.03.17  logstash-2016.04.25  logstash-2016.06.03  logstash-2016.07.12
logstash-2015.12.31  logstash-2016.02.08  logstash-2016.03.18  logstash-2016.04.26  logstash-2016.06.04  logstash-2016.07.13
logstash-2016.01.01  logstash-2016.02.09  logstash-2016.03.19  logstash-2016.04.27  logstash-2016.06.05  logstash-2016.07.14
logstash-2016.01.02  logstash-2016.02.10  logstash-2016.03.20  logstash-2016.04.28  logstash-2016.06.06  logstash-2016.07.15
logstash-2016.01.03  logstash-2016.02.11  logstash-2016.03.21  logstash-2016.04.29  logstash-2016.06.07  logstash-2016.07.16
logstash-2016.01.04  logstash-2016.02.12  logstash-2016.03.22  logstash-2016.04.30  logstash-2016.06.08  logstash-2016.07.17
logstash-2016.01.05  logstash-2016.02.13  logstash-2016.03.23  logstash-2016.05.01  logstash-2016.06.09  logstash-2016.07.18
logstash-2016.01.06  logstash-2016.02.14  logstash-2016.03.24  logstash-2016.05.02  logstash-2016.06.10  logstash-2016.07.19
logstash-2016.01.07  logstash-2016.02.15  logstash-2016.03.25  logstash-2016.05.03  logstash-2016.06.11  logstash-2016.07.20
logstash-2016.01.08  logstash-2016.02.16  logstash-2016.03.26  logstash-2016.05.04  logstash-2016.06.12  logstash-2016.07.21
logstash-2016.01.09  logstash-2016.02.17  logstash-2016.03.27  logstash-2016.05.05  logstash-2016.06.13  logstash-2016.07.22
logstash-2016.01.10  logstash-2016.02.18  logstash-2016.03.28  logstash-2016.05.06  logstash-2016.06.14  logstash-2016.07.23
logstash-2016.01.11  logstash-2016.02.19  logstash-2016.03.29  logstash-2016.05.07  logstash-2016.06.15  logstash-2016.07.24
logstash-2016.01.12  logstash-2016.02.20  logstash-2016.03.30  logstash-2016.05.08  logstash-2016.06.16  logstash-2016.07.25
logstash-2016.01.13  logstash-2016.02.21  logstash-2016.03.31  logstash-2016.05.09  logstash-2016.06.17  logstash-2016.07.26
logstash-2016.01.14  logstash-2016.02.22  logstash-2016.04.01  logstash-2016.05.10  logstash-2016.06.18  logstash-2016.07.27
logstash-2016.01.15  logstash-2016.02.23  logstash-2016.04.02  logstash-2016.05.11  logstash-2016.06.19  logstash-2016.07.28

As you can see, there are plenty of directories older than 90 days that are still there. What can we do to see what's wrong?

Some additional info:
- NLS version: 1.4.2
- OS version: CentOS 6.6
- Backup repo is on an NFS mount
- Permissions look okay and backups are written daily, as you can see in the directory list
- The "older than 90 days" directories are not empty and contain data

hsmith · Post by **hsmith** » Fri Jul 29, 2016 1:25 pm

I'm going to look in to a few things on our git, and I'll git back to you.

mike4vr · Post by **mike4vr** » Fri Jul 29, 2016 1:40 pm

hsmith wrote:I'm going to look in to a few things on our git, and I'll git back to you.

Thanks! We're down to 9% available on the data storage unit so this is becoming a bit urgent.

hsmith · Post by **hsmith** » Fri Jul 29, 2016 1:46 pm

Code: Select all

curator delete snapshots --older-than 180 --time-unit days --timestring %Y%m%d --repository YOURREPOSITORYNAME

Give this command a shot. I'd start with a very high number on --older-than to verify it's working.

mike4vr · Post by **mike4vr** » Fri Jul 29, 2016 3:39 pm

hsmith wrote:
Code: Select all
curator delete snapshots --older-than 180 --time-unit days --timestring %Y%m%d --repository YOURREPOSITORYNAME
Give this command a shot. I'd start with a very high number on --older-than to verify it's working.

Tried this. Works for a while then spits out the following error:

Code: Select all

2016-07-29 13:38:12,865 INFO      Job starting: delete snapshots
2016-07-29 13:38:14,170 INFO      Deleting snapshot curator-20160401045823
Traceback (most recent call last):
  File "/usr/bin/curator", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/curator/curator.py", line 5, in main
    cli( obj={ "filters": [] } )
  File "/usr/lib/python2.6/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.6/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.6/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/curator/cli/snapshot_selection.py", line 93, in snapshots
    retval = delete_snapshot(client, snapshot=snap, repository=repository)
  File "/usr/lib/python2.6/site-packages/curator/api/snapshot.py", line 113, in delete_snapshot
    client.snapshot.delete(repository=repository, snapshot=snapshot)
  File "/usr/lib/python2.6/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/lib/python2.6/site-packages/elasticsearch/client/snapshot.py", line 40, in delete
    _make_path('_snapshot', repository, snapshot), params=params)
  File "/usr/lib/python2.6/site-packages/elasticsearch/transport.py", line 307, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/lib/python2.6/site-packages/elasticsearch/connection/http_urllib3.py", line 86, in perform_request
    raise ConnectionTimeout('TIMEOUT', str(e), e)
elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host=u'localhost', port=9200): Read timed out. (read timeout=30))

If I keep submitting the command, it'll go through a few but the error keeps happening.

hsmith · Post by **hsmith** » Mon Aug 01, 2016 9:20 am

What if we add a large timeout to it?

Code: Select all

curator --timeout 9999 delete snapshots --older-than 180 --time-unit days --timestring %Y%m%d --repository test

mike4vr · Post by **mike4vr** » Tue Aug 02, 2016 9:43 am

hsmith wrote:What if we add a large timeout to it?
Code: Select all
curator --timeout 9999 delete snapshots --older-than 180 --time-unit days --timestring %Y%m%d --repository test

That did it. Thank you!

hsmith · Post by **hsmith** » Tue Aug 02, 2016 9:50 am

Great, hopefully now the backup job will work correctly. Let us know if you find any other oddities!

Nagios Support Forum

backup_maintenance is not clearing old backups

backup_maintenance is not clearing old backups

Re: backup_maintenance is not clearing old backups

Re: backup_maintenance is not clearing old backups

Re: backup_maintenance is not clearing old backups

Re: backup_maintenance is not clearing old backups

Re: backup_maintenance is not clearing old backups

Re: backup_maintenance is not clearing old backups

Re: backup_maintenance is not clearing old backups