Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Updated maintenance settings but indices still not deleting

https://support.nagios.com/forum/viewtopic.php?t=40412

Page 1 of 2

Updated maintenance settings but indices still not deleting

Posted: Tue Sep 27, 2016 11:29 am
by CFT6Server
We recently reduced the retention period in our Maintenance settings, but I did not see any disk space usage changes. I checked backend and doesn't seem like the new settings took place.

Previous settings: close index after 14 days / Delete after 30 days / Delete backups after 60 days.
New settings: close index after 7 days / Delete after 15 days / Delete backups after 45 days.

Currently I am seeing about 53 indices listed in the backup maintenance page.

Here's the repository as a sample.

Code: Select all

[root@ repository]# ls
index                         metadata-logstash-2016.08.23  metadata-logstash-2016.09.14  snapshot-logstash-2016.08.13  snapshot-logstash-2016.09.04
indices                       metadata-logstash-2016.08.24  metadata-logstash-2016.09.15  snapshot-logstash-2016.08.14  snapshot-logstash-2016.09.05
metadata-logstash-2016.08.03  metadata-logstash-2016.08.25  metadata-logstash-2016.09.16  snapshot-logstash-2016.08.15  snapshot-logstash-2016.09.06
metadata-logstash-2016.08.04  metadata-logstash-2016.08.26  metadata-logstash-2016.09.17  snapshot-logstash-2016.08.16  snapshot-logstash-2016.09.07
metadata-logstash-2016.08.05  metadata-logstash-2016.08.27  metadata-logstash-2016.09.18  snapshot-logstash-2016.08.17  snapshot-logstash-2016.09.08
metadata-logstash-2016.08.06  metadata-logstash-2016.08.28  metadata-logstash-2016.09.19  snapshot-logstash-2016.08.18  snapshot-logstash-2016.09.09
metadata-logstash-2016.08.07  metadata-logstash-2016.08.29  metadata-logstash-2016.09.20  snapshot-logstash-2016.08.19  snapshot-logstash-2016.09.10
metadata-logstash-2016.08.08  metadata-logstash-2016.08.30  metadata-logstash-2016.09.21  snapshot-logstash-2016.08.20  snapshot-logstash-2016.09.11
metadata-logstash-2016.08.09  metadata-logstash-2016.08.31  metadata-logstash-2016.09.22  snapshot-logstash-2016.08.21  snapshot-logstash-2016.09.12
metadata-logstash-2016.08.10  metadata-logstash-2016.09.01  metadata-logstash-2016.09.23  snapshot-logstash-2016.08.22  snapshot-logstash-2016.09.13
metadata-logstash-2016.08.11  metadata-logstash-2016.09.02  metadata-logstash-2016.09.24  snapshot-logstash-2016.08.23  snapshot-logstash-2016.09.14
metadata-logstash-2016.08.12  metadata-logstash-2016.09.03  metadata-logstash-2016.09.25  snapshot-logstash-2016.08.24  snapshot-logstash-2016.09.15
metadata-logstash-2016.08.13  metadata-logstash-2016.09.04  snapshot-logstash-2016.08.03  snapshot-logstash-2016.08.25  snapshot-logstash-2016.09.16
metadata-logstash-2016.08.14  metadata-logstash-2016.09.05  snapshot-logstash-2016.08.04  snapshot-logstash-2016.08.26  snapshot-logstash-2016.09.17
metadata-logstash-2016.08.15  metadata-logstash-2016.09.06  snapshot-logstash-2016.08.05  snapshot-logstash-2016.08.27  snapshot-logstash-2016.09.18
metadata-logstash-2016.08.16  metadata-logstash-2016.09.07  snapshot-logstash-2016.08.06  snapshot-logstash-2016.08.28  snapshot-logstash-2016.09.19
metadata-logstash-2016.08.17  metadata-logstash-2016.09.08  snapshot-logstash-2016.08.07  snapshot-logstash-2016.08.29  snapshot-logstash-2016.09.20
metadata-logstash-2016.08.18  metadata-logstash-2016.09.09  snapshot-logstash-2016.08.08  snapshot-logstash-2016.08.30  snapshot-logstash-2016.09.21
metadata-logstash-2016.08.19  metadata-logstash-2016.09.10  snapshot-logstash-2016.08.09  snapshot-logstash-2016.08.31  snapshot-logstash-2016.09.22
metadata-logstash-2016.08.20  metadata-logstash-2016.09.11  snapshot-logstash-2016.08.10  snapshot-logstash-2016.09.01  snapshot-logstash-2016.09.23
metadata-logstash-2016.08.21  metadata-logstash-2016.09.12  snapshot-logstash-2016.08.11  snapshot-logstash-2016.09.02  snapshot-logstash-2016.09.24
metadata-logstash-2016.08.22  metadata-logstash-2016.09.13  snapshot-logstash-2016.08.12  snapshot-logstash-2016.09.03  snapshot-logstash-2016.09.25

Code: Select all

[root@ indices]# pwd
/nfs/logdata/80e9022e-f73f-429e-8927-f23d0d88dfd2/nodes/0/indices
[root@ indices]# ls
kibana-int           logstash-2016.09.04  logstash-2016.09.09  logstash-2016.09.14  logstash-2016.09.19  logstash-2016.09.24  nagioslogserver_log
logstash-2016.08.24  logstash-2016.09.05  logstash-2016.09.10  logstash-2016.09.15  logstash-2016.09.20  logstash-2016.09.25
logstash-2016.09.01  logstash-2016.09.06  logstash-2016.09.11  logstash-2016.09.16  logstash-2016.09.21  logstash-2016.09.26
logstash-2016.09.02  logstash-2016.09.07  logstash-2016.09.12  logstash-2016.09.17  logstash-2016.09.22  logstash-2016.09.27
logstash-2016.09.03  logstash-2016.09.08  logstash-2016.09.13  logstash-2016.09.18  logstash-2016.09.23  nagioslogserver
[root@ indices]# ls | wc -l
31

Re: Updated maintenance settings but indices still not delet

Posted: Tue Sep 27, 2016 1:34 pm
by mcapra
Can you run the following from the CLI and share the output:

Code: Select all

curator --debug --dry-run delete snapshots --older-than 45 --time-unit days --timestring %Y%m%d --repository Repository

Re: Updated maintenance settings but indices still not delet

Posted: Tue Sep 27, 2016 6:32 pm
by CFT6Server
Can you provide an updated command? thanks.

Code: Select all

curator: error: unrecognized arguments: snapshots --repository Repository

Re: Updated maintenance settings but indices still not delet

Posted: Tue Sep 27, 2016 9:13 pm
by Box293
Can you confirm the command you are typing, there might be a spelling mistake.

NOTE: --repository Repository will need to be replaced with the name of your repository.

For example:

Code: Select all

curator --debug --dry-run delete snapshots --older-than 45 --time-unit days --timestring %Y%m%d --repository test_backups
Which should output something like:

Code: Select all

2016-09-28 12:08:25,334 DEBUG         curator.api.filter         get_date_regex:158  regex = \d{4}\d{2}\d{2}
2016-09-28 12:08:25,334 DEBUG          curator.cli.utils        filter_callback:189  REGEX = (?P<date>\d{4}\d{2}\d{2})
2016-09-28 12:08:25,334 DEBUG          curator.cli.utils        filter_callback:192  Added filter: {'pattern': '(?P<date>\\d{4}\\d{2}\\d{2})', 'value': 45, 'groupname': 'date', 'time_unit': 'days', 'timestring': u'%Y%m%d', 'method': 'older_than'}
2016-09-28 12:08:25,334 DEBUG          curator.cli.utils        filter_callback:193  New list of filters: [{'pattern': '(?P<date>\\d{4}\\d{2}\\d{2})', 'value': 45, 'groupname': 'date', 'time_unit': 'days', 'timestring': u'%Y%m%d', 'method': 'older_than'}]
2016-09-28 12:08:25,334 INFO      curator.cli.snapshot_selection              snapshots:51   Job starting: delete snapshots
2016-09-28 12:08:25,334 INFO                        root              snapshots:55   DRY RUN MODE.  No changes will be made.
2016-09-28 12:08:25,334 DEBUG          curator.cli.utils             get_client:112  kwargs = {'url_prefix': u'', 'http_auth': None, 'certificate': None, 'dry_run': True, 'loglevel': u'INFO', 'quiet': False, 'debug': True, 'logformat': u'default', 'timeout': 30, 'host': u'localhost', 'use_ssl': False, 'logfile': None, 'master_only': False, 'port': 9200, 'ssl_no_validate': False}
2016-09-28 12:08:25,334 DEBUG         urllib3.util.retry               from_int:180  Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-09-28 12:08:25,334 DEBUG     urllib3.connectionpool              _new_conn:212  Starting new HTTP connection (1): localhost
2016-09-28 12:08:25,339 DEBUG     urllib3.connectionpool          _make_request:400  http://localhost:9200 "GET / HTTP/1.1" 200 387
2016-09-28 12:08:25,339 INFO               elasticsearch    log_request_success:63   GET http://localhost:9200/ [status:200 request:0.005s]
2016-09-28 12:08:25,339 DEBUG              elasticsearch    log_request_success:65   > None
2016-09-28 12:08:25,339 DEBUG              elasticsearch    log_request_success:66   < {
  "status" : 200,
  "name" : "faec8524-5c8b-46a9-a3e0-ca8253398062",
  "cluster_name" : "83204311-5d3e-453b-9c6c-e72d92ff5eab",
  "version" : {
    "number" : "1.6.0",
    "build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0",
    "build_timestamp" : "2015-06-09T13:36:34Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

2016-09-28 12:08:25,340 DEBUG          curator.cli.utils          check_version:90   Detected Elasticsearch version 1.6.0
2016-09-28 12:08:25,340 DEBUG         urllib3.util.retry               from_int:180  Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-09-28 12:08:25,341 DEBUG     urllib3.connectionpool          _make_request:400  http://localhost:9200 "GET /_snapshot/test_backups/_all HTTP/1.1" 200 1158
2016-09-28 12:08:25,342 INFO               elasticsearch    log_request_success:63   GET http://localhost:9200/_snapshot/test_backups/_all [status:200 request:0.002s]
2016-09-28 12:08:25,342 DEBUG              elasticsearch    log_request_success:65   > None
2016-09-28 12:08:25,342 DEBUG              elasticsearch    log_request_success:66   < {"snapshots":[{"snapshot":"curator-20160928020606","indices":["kibana-int","logstash-2016.09.28","nagioslogserver","nagioslogserver_log"],"state":"SUCCESS","start_time":"2016-09-28T02:06:06.587Z","start_time_in_millis":1475028366587,"end_time":"2016-09-28T02:06:07.036Z","end_time_in_millis":1475028367036,"duration_in_millis":449,"failures":[],"shards":{"total":16,"failed":0,"successful":16}},{"snapshot":"curator-20160928020616","indices":["kibana-int","logstash-2016.09.28","nagioslogserver","nagioslogserver_log"],"state":"SUCCESS","start_time":"2016-09-28T02:06:16.874Z","start_time_in_millis":1475028376874,"end_time":"2016-09-28T02:06:17.074Z","end_time_in_millis":1475028377074,"duration_in_millis":200,"failures":[],"shards":{"total":16,"failed":0,"successful":16}},{"snapshot":"curator-20160928020621","indices":["kibana-int","logstash-2016.09.28","nagioslogserver","nagioslogserver_log"],"state":"SUCCESS","start_time":"2016-09-28T02:06:21.147Z","start_time_in_millis":1475028381147,"end_time":"2016-09-28T02:06:21.332Z","end_time_in_millis":1475028381332,"duration_in_millis":185,"failures":[],"shards":{"total":16,"failed":0,"successful":16}}]}
2016-09-28 12:08:25,343 DEBUG     curator.cli.snapshot_selection              snapshots:70   All filters: [{'pattern': '(?P<date>\\d{4}\\d{2}\\d{2})', 'value': 45, 'groupname': 'date', 'time_unit': 'days', 'timestring': u'%Y%m%d', 'method': 'older_than'}]
2016-09-28 12:08:25,343 DEBUG     curator.cli.snapshot_selection              snapshots:75   Filter: {'pattern': '(?P<date>\\d{4}\\d{2}\\d{2})', 'value': 45, 'groupname': 'date', 'time_unit': 'days', 'timestring': u'%Y%m%d', 'method': 'older_than'}
2016-09-28 12:08:25,345 DEBUG         curator.api.filter        timestamp_check:301  Timestamp "20160928" is outside the cutoff period (older than 45 days).
2016-09-28 12:08:25,346 DEBUG         curator.api.filter        timestamp_check:301  Timestamp "20160928" is outside the cutoff period (older than 45 days).
2016-09-28 12:08:25,346 DEBUG         curator.api.filter        timestamp_check:301  Timestamp "20160928" is outside the cutoff period (older than 45 days).
2016-09-28 12:08:25,346 WARNING   curator.cli.snapshot_selection              snapshots:100  No snapshots matched provided args.
No snapshots matched provided args.
If you are unsure of the name of your repository then you can execute the following command:

Code: Select all

curl -XGET "localhost:9200/_snapshot?pretty"
Which should output something like:

Code: Select all

{
  "test_backups" : {
    "type" : "fs",
    "settings" : {
      "compress" : "true",
      "location" : "/store/test_backups"
    }
  }
}

Re: Updated maintenance settings but indices still not delet

Posted: Wed Sep 28, 2016 10:33 am
by CFT6Server
Hi There, my Repository is actually named "Repository" and the command is not working.

Code: Select all

[root@ indices]# curator --debug --dry-run delete snapshots --older-than 45 --time-unit days --timestring %Y%m%d --repository Repository
usage: curator [-h] [-v] [--host HOST] [--url_prefix URL_PREFIX] [--port PORT]
               [--ssl] [--auth AUTH] [-t TIMEOUT] [--master-only] [-n] [-D]
               [--loglevel LOG_LEVEL] [-l LOG_FILE] [--logformat LOGFORMAT]
               {show,allocation,alias,snapshot,close,bloom,optimize,delete}
               ...
curator: error: unrecognized arguments: snapshots --repository Repository
[root@ indices]# curl -XGET "localhost:9200/_snapshot?pretty"
{
  "Repository" : {
    "type" : "fs",
    "settings" : {
      "compress" : "true",
      "location" : "/nfs/repository"
    }
  }
}

Re: Updated maintenance settings but indices still not delet

Posted: Wed Sep 28, 2016 10:38 am
by mcapra
What version of NLS are you using?

Can you also share the output of curator --version and curator delete

Re: Updated maintenance settings but indices still not delet

Posted: Wed Sep 28, 2016 11:21 am
by CFT6Server
I am on 1.3 and here's the output.

Code: Select all

[root@ indices]# curator --version
curator 1.2.2
[root@ indices]# curator delete
/usr/bin/curator delete: error: expect one of --older-than or --disk-space

Re: Updated maintenance settings but indices still not delet

Posted: Wed Sep 28, 2016 1:21 pm
by mcapra
Cool, lets use a command that's more appropriate for your curator version:

Code: Select all

curator --debug --dry-run delete --older-than 15

Re: Updated maintenance settings but indices still not delet

Posted: Tue Oct 04, 2016 2:50 pm
by CFT6Server
Here it is.... odd that it did not pick up anything before Sept 15??

Code: Select all

[root@ indices]# curl localhost:9200/_snapshot/Repository/_all?pretty | grep indices
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
1    "indices" : [ "logstash-2016.08.12" ], 0 --:--:-- --:--:-- --:--:--     0
00     "indices" : [ "logstash-2016.08.13" ],
23    "indices" : [ "logstash-2016.08.14" ],
11    "indices" : [ "logstash-2016.08.15" ],
5     "indices" : [ "logstash-2016.08.16" ],
     "indices" : [ "logstash-2016.08.17" ],
10    "indices" : [ "logstash-2016.08.18" ],
0     "indices" : [ "logstash-2016.08.19" ],
23    "indices" : [ "logstash-2016.08.20" ],
11    "indices" : [ "logstash-2016.08.21" ],
5     "indices" : [ "logstash-2016.08.22" ],
      "indices" : [ "logstash-2016.08.23" ],
 0    "indices" : [ "logstash-2016.08.24" ],
      "indices" : [ "logstash-2016.08.25" ],
      "indices" : [ "logstash-2016.08.26" ],
 0    "indices" : [ "logstash-2016.08.27" ],
      "indices" : [ "logstash-2016.08.28" ],
13    "indices" : [ "logstash-2016.08.29" ],
78    "indices" : [ "logstash-2016.08.30" ],
k     "indices" : [ "logstash-2016.08.31" ],
      "indices" : [ "logstash-2016.09.01" ],
      "indices" : [ "logstash-2016.09.02" ],
 0    "indices" : [ "logstash-2016.09.03" ],
 -    "indices" : [ "logstash-2016.09.04" ],
-:    "indices" : [ "logstash-2016.09.05" ],
--    "indices" : [ "logstash-2016.09.06" ],
:-    "indices" : [ "logstash-2016.09.07" ],
-     "indices" : [ "logstash-2016.09.08" ],
      "indices" : [ "logstash-2016.09.09" ],

    "indices" : [ "logstash-2016.09.10" ],
    "indices" : [ "logstash-2016.09.11" ],
    "indices" : [ "logstash-2016.09.12" ],
    "indices" : [ "logstash-2016.09.13" ],
    "indices" : [ "logstash-2016.09.14" ],
    "indices" : [ "logstash-2016.09.15" ],
    "indices" : [ "logstash-2016.09.16" ],
    "indices" : [ "logstash-2016.09.17" ],
    "indices" : [ "logstash-2016.09.18" ],
    "indices" : [ "logstash-2016.09.19" ],
    "indices" : [ "logstash-2016.09.20" ],
    "indices" : [ "logstash-2016.09.21" ],
    "indices" : [ "logstash-2016.09.22" ],
    "indices" : [ "logstash-2016.09.23" ],
    "indices" : [ "logstash-2016.09.24" ],
    "indices" : [ "logstash-2016.09.25" ],
    "indices" : [ "logstash-2016.09.26" ],
    "indices" : [ "logstash-2016.09.27" ],
    "indices" : [ "logstash-2016.09.28" ],
    "indices" : [ "logstash-2016.09.29" ],
    "indices" : [ "logstash-2016.09.30" ],
    "indices" : [ "logstash-2016.10.01" ],
    "indices" : [ "logstash-2016.10.02" ],
    "indices" : [ "logstash-2016.10.03" ],

Re: Updated maintenance settings but indices still not delet

Posted: Tue Oct 04, 2016 4:06 pm
by rkennedy
It did, looks like the dry ran stopped after 19 days, and not 15. Let's debug a little bit as to what's being sent over to NLS.

Please replace your /usr/local/nagioslogserver/scripts/curator.sh script with what I've pasted below. (this will still function as expected, I've just modified to add debugging / output for more verbosity.)

Code: Select all

#!/bin/sh
date=$(date)

curator "$@"
echo --- >> /tmp/curatordebug.txt
echo $date >> /tmp/curatordebug.txt
echo "$@" >> /tmp/curatordebug.txt
curl -s 127.0.0.1:9200/_cluster/state/blocks?pretty >> /tmp/curatordebug.txt
touch /tmp/curatordebug.txt
chmod 777 /tmp/curatordebug.txt
Let this run for a few days (to gain a bit of data), and then post the /tmp/curatordebug.txt file for us to look at. I'm interested to see which variables are passing for your backups.
All times are UTC-05:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited