Updated maintenance settings but indices still not deleting

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

Updated maintenance settings but indices still not deleting

Post by CFT6Server »

We recently reduced the retention period in our Maintenance settings, but I did not see any disk space usage changes. I checked backend and doesn't seem like the new settings took place.

Previous settings: close index after 14 days / Delete after 30 days / Delete backups after 60 days.
New settings: close index after 7 days / Delete after 15 days / Delete backups after 45 days.

Currently I am seeing about 53 indices listed in the backup maintenance page.

Here's the repository as a sample.

Code: Select all

[root@ repository]# ls
index                         metadata-logstash-2016.08.23  metadata-logstash-2016.09.14  snapshot-logstash-2016.08.13  snapshot-logstash-2016.09.04
indices                       metadata-logstash-2016.08.24  metadata-logstash-2016.09.15  snapshot-logstash-2016.08.14  snapshot-logstash-2016.09.05
metadata-logstash-2016.08.03  metadata-logstash-2016.08.25  metadata-logstash-2016.09.16  snapshot-logstash-2016.08.15  snapshot-logstash-2016.09.06
metadata-logstash-2016.08.04  metadata-logstash-2016.08.26  metadata-logstash-2016.09.17  snapshot-logstash-2016.08.16  snapshot-logstash-2016.09.07
metadata-logstash-2016.08.05  metadata-logstash-2016.08.27  metadata-logstash-2016.09.18  snapshot-logstash-2016.08.17  snapshot-logstash-2016.09.08
metadata-logstash-2016.08.06  metadata-logstash-2016.08.28  metadata-logstash-2016.09.19  snapshot-logstash-2016.08.18  snapshot-logstash-2016.09.09
metadata-logstash-2016.08.07  metadata-logstash-2016.08.29  metadata-logstash-2016.09.20  snapshot-logstash-2016.08.19  snapshot-logstash-2016.09.10
metadata-logstash-2016.08.08  metadata-logstash-2016.08.30  metadata-logstash-2016.09.21  snapshot-logstash-2016.08.20  snapshot-logstash-2016.09.11
metadata-logstash-2016.08.09  metadata-logstash-2016.08.31  metadata-logstash-2016.09.22  snapshot-logstash-2016.08.21  snapshot-logstash-2016.09.12
metadata-logstash-2016.08.10  metadata-logstash-2016.09.01  metadata-logstash-2016.09.23  snapshot-logstash-2016.08.22  snapshot-logstash-2016.09.13
metadata-logstash-2016.08.11  metadata-logstash-2016.09.02  metadata-logstash-2016.09.24  snapshot-logstash-2016.08.23  snapshot-logstash-2016.09.14
metadata-logstash-2016.08.12  metadata-logstash-2016.09.03  metadata-logstash-2016.09.25  snapshot-logstash-2016.08.24  snapshot-logstash-2016.09.15
metadata-logstash-2016.08.13  metadata-logstash-2016.09.04  snapshot-logstash-2016.08.03  snapshot-logstash-2016.08.25  snapshot-logstash-2016.09.16
metadata-logstash-2016.08.14  metadata-logstash-2016.09.05  snapshot-logstash-2016.08.04  snapshot-logstash-2016.08.26  snapshot-logstash-2016.09.17
metadata-logstash-2016.08.15  metadata-logstash-2016.09.06  snapshot-logstash-2016.08.05  snapshot-logstash-2016.08.27  snapshot-logstash-2016.09.18
metadata-logstash-2016.08.16  metadata-logstash-2016.09.07  snapshot-logstash-2016.08.06  snapshot-logstash-2016.08.28  snapshot-logstash-2016.09.19
metadata-logstash-2016.08.17  metadata-logstash-2016.09.08  snapshot-logstash-2016.08.07  snapshot-logstash-2016.08.29  snapshot-logstash-2016.09.20
metadata-logstash-2016.08.18  metadata-logstash-2016.09.09  snapshot-logstash-2016.08.08  snapshot-logstash-2016.08.30  snapshot-logstash-2016.09.21
metadata-logstash-2016.08.19  metadata-logstash-2016.09.10  snapshot-logstash-2016.08.09  snapshot-logstash-2016.08.31  snapshot-logstash-2016.09.22
metadata-logstash-2016.08.20  metadata-logstash-2016.09.11  snapshot-logstash-2016.08.10  snapshot-logstash-2016.09.01  snapshot-logstash-2016.09.23
metadata-logstash-2016.08.21  metadata-logstash-2016.09.12  snapshot-logstash-2016.08.11  snapshot-logstash-2016.09.02  snapshot-logstash-2016.09.24
metadata-logstash-2016.08.22  metadata-logstash-2016.09.13  snapshot-logstash-2016.08.12  snapshot-logstash-2016.09.03  snapshot-logstash-2016.09.25

Code: Select all

[root@ indices]# pwd
/nfs/logdata/80e9022e-f73f-429e-8927-f23d0d88dfd2/nodes/0/indices
[root@ indices]# ls
kibana-int           logstash-2016.09.04  logstash-2016.09.09  logstash-2016.09.14  logstash-2016.09.19  logstash-2016.09.24  nagioslogserver_log
logstash-2016.08.24  logstash-2016.09.05  logstash-2016.09.10  logstash-2016.09.15  logstash-2016.09.20  logstash-2016.09.25
logstash-2016.09.01  logstash-2016.09.06  logstash-2016.09.11  logstash-2016.09.16  logstash-2016.09.21  logstash-2016.09.26
logstash-2016.09.02  logstash-2016.09.07  logstash-2016.09.12  logstash-2016.09.17  logstash-2016.09.22  logstash-2016.09.27
logstash-2016.09.03  logstash-2016.09.08  logstash-2016.09.13  logstash-2016.09.18  logstash-2016.09.23  nagioslogserver
[root@ indices]# ls | wc -l
31
You do not have the required permissions to view the files attached to this post.
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Updated maintenance settings but indices still not delet

Post by mcapra »

Can you run the following from the CLI and share the output:

Code: Select all

curator --debug --dry-run delete snapshots --older-than 45 --time-unit days --timestring %Y%m%d --repository Repository
Former Nagios employee
https://www.mcapra.com/
Top
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

Re: Updated maintenance settings but indices still not delet

Post by CFT6Server »

Can you provide an updated command? thanks.

Code: Select all

curator: error: unrecognized arguments: snapshots --repository Repository
Top
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:
Contact Box293

Re: Updated maintenance settings but indices still not delet

Post by Box293 »

Can you confirm the command you are typing, there might be a spelling mistake.

NOTE: --repository Repository will need to be replaced with the name of your repository.

For example:

Code: Select all

curator --debug --dry-run delete snapshots --older-than 45 --time-unit days --timestring %Y%m%d --repository test_backups
Which should output something like:

Code: Select all

2016-09-28 12:08:25,334 DEBUG         curator.api.filter         get_date_regex:158  regex = \d{4}\d{2}\d{2}
2016-09-28 12:08:25,334 DEBUG          curator.cli.utils        filter_callback:189  REGEX = (?P<date>\d{4}\d{2}\d{2})
2016-09-28 12:08:25,334 DEBUG          curator.cli.utils        filter_callback:192  Added filter: {'pattern': '(?P<date>\\d{4}\\d{2}\\d{2})', 'value': 45, 'groupname': 'date', 'time_unit': 'days', 'timestring': u'%Y%m%d', 'method': 'older_than'}
2016-09-28 12:08:25,334 DEBUG          curator.cli.utils        filter_callback:193  New list of filters: [{'pattern': '(?P<date>\\d{4}\\d{2}\\d{2})', 'value': 45, 'groupname': 'date', 'time_unit': 'days', 'timestring': u'%Y%m%d', 'method': 'older_than'}]
2016-09-28 12:08:25,334 INFO      curator.cli.snapshot_selection              snapshots:51   Job starting: delete snapshots
2016-09-28 12:08:25,334 INFO                        root              snapshots:55   DRY RUN MODE.  No changes will be made.
2016-09-28 12:08:25,334 DEBUG          curator.cli.utils             get_client:112  kwargs = {'url_prefix': u'', 'http_auth': None, 'certificate': None, 'dry_run': True, 'loglevel': u'INFO', 'quiet': False, 'debug': True, 'logformat': u'default', 'timeout': 30, 'host': u'localhost', 'use_ssl': False, 'logfile': None, 'master_only': False, 'port': 9200, 'ssl_no_validate': False}
2016-09-28 12:08:25,334 DEBUG         urllib3.util.retry               from_int:180  Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-09-28 12:08:25,334 DEBUG     urllib3.connectionpool              _new_conn:212  Starting new HTTP connection (1): localhost
2016-09-28 12:08:25,339 DEBUG     urllib3.connectionpool          _make_request:400  http://localhost:9200 "GET / HTTP/1.1" 200 387
2016-09-28 12:08:25,339 INFO               elasticsearch    log_request_success:63   GET http://localhost:9200/ [status:200 request:0.005s]
2016-09-28 12:08:25,339 DEBUG              elasticsearch    log_request_success:65   > None
2016-09-28 12:08:25,339 DEBUG              elasticsearch    log_request_success:66   < {
  "status" : 200,
  "name" : "faec8524-5c8b-46a9-a3e0-ca8253398062",
  "cluster_name" : "83204311-5d3e-453b-9c6c-e72d92ff5eab",
  "version" : {
    "number" : "1.6.0",
    "build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0",
    "build_timestamp" : "2015-06-09T13:36:34Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

2016-09-28 12:08:25,340 DEBUG          curator.cli.utils          check_version:90   Detected Elasticsearch version 1.6.0
2016-09-28 12:08:25,340 DEBUG         urllib3.util.retry               from_int:180  Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-09-28 12:08:25,341 DEBUG     urllib3.connectionpool          _make_request:400  http://localhost:9200 "GET /_snapshot/test_backups/_all HTTP/1.1" 200 1158
2016-09-28 12:08:25,342 INFO               elasticsearch    log_request_success:63   GET http://localhost:9200/_snapshot/test_backups/_all [status:200 request:0.002s]
2016-09-28 12:08:25,342 DEBUG              elasticsearch    log_request_success:65   > None
2016-09-28 12:08:25,342 DEBUG              elasticsearch    log_request_success:66   < {"snapshots":[{"snapshot":"curator-20160928020606","indices":["kibana-int","logstash-2016.09.28","nagioslogserver","nagioslogserver_log"],"state":"SUCCESS","start_time":"2016-09-28T02:06:06.587Z","start_time_in_millis":1475028366587,"end_time":"2016-09-28T02:06:07.036Z","end_time_in_millis":1475028367036,"duration_in_millis":449,"failures":[],"shards":{"total":16,"failed":0,"successful":16}},{"snapshot":"curator-20160928020616","indices":["kibana-int","logstash-2016.09.28","nagioslogserver","nagioslogserver_log"],"state":"SUCCESS","start_time":"2016-09-28T02:06:16.874Z","start_time_in_millis":1475028376874,"end_time":"2016-09-28T02:06:17.074Z","end_time_in_millis":1475028377074,"duration_in_millis":200,"failures":[],"shards":{"total":16,"failed":0,"successful":16}},{"snapshot":"curator-20160928020621","indices":["kibana-int","logstash-2016.09.28","nagioslogserver","nagioslogserver_log"],"state":"SUCCESS","start_time":"2016-09-28T02:06:21.147Z","start_time_in_millis":1475028381147,"end_time":"2016-09-28T02:06:21.332Z","end_time_in_millis":1475028381332,"duration_in_millis":185,"failures":[],"shards":{"total":16,"failed":0,"successful":16}}]}
2016-09-28 12:08:25,343 DEBUG     curator.cli.snapshot_selection              snapshots:70   All filters: [{'pattern': '(?P<date>\\d{4}\\d{2}\\d{2})', 'value': 45, 'groupname': 'date', 'time_unit': 'days', 'timestring': u'%Y%m%d', 'method': 'older_than'}]
2016-09-28 12:08:25,343 DEBUG     curator.cli.snapshot_selection              snapshots:75   Filter: {'pattern': '(?P<date>\\d{4}\\d{2}\\d{2})', 'value': 45, 'groupname': 'date', 'time_unit': 'days', 'timestring': u'%Y%m%d', 'method': 'older_than'}
2016-09-28 12:08:25,345 DEBUG         curator.api.filter        timestamp_check:301  Timestamp "20160928" is outside the cutoff period (older than 45 days).
2016-09-28 12:08:25,346 DEBUG         curator.api.filter        timestamp_check:301  Timestamp "20160928" is outside the cutoff period (older than 45 days).
2016-09-28 12:08:25,346 DEBUG         curator.api.filter        timestamp_check:301  Timestamp "20160928" is outside the cutoff period (older than 45 days).
2016-09-28 12:08:25,346 WARNING   curator.cli.snapshot_selection              snapshots:100  No snapshots matched provided args.
No snapshots matched provided args.
If you are unsure of the name of your repository then you can execute the following command:

Code: Select all

curl -XGET "localhost:9200/_snapshot?pretty"
Which should output something like:

Code: Select all

{
  "test_backups" : {
    "type" : "fs",
    "settings" : {
      "compress" : "true",
      "location" : "/store/test_backups"
    }
  }
}
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Top
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

Re: Updated maintenance settings but indices still not delet

Post by CFT6Server »

Hi There, my Repository is actually named "Repository" and the command is not working.

Code: Select all

[root@ indices]# curator --debug --dry-run delete snapshots --older-than 45 --time-unit days --timestring %Y%m%d --repository Repository
usage: curator [-h] [-v] [--host HOST] [--url_prefix URL_PREFIX] [--port PORT]
               [--ssl] [--auth AUTH] [-t TIMEOUT] [--master-only] [-n] [-D]
               [--loglevel LOG_LEVEL] [-l LOG_FILE] [--logformat LOGFORMAT]
               {show,allocation,alias,snapshot,close,bloom,optimize,delete}
               ...
curator: error: unrecognized arguments: snapshots --repository Repository
[root@ indices]# curl -XGET "localhost:9200/_snapshot?pretty"
{
  "Repository" : {
    "type" : "fs",
    "settings" : {
      "compress" : "true",
      "location" : "/nfs/repository"
    }
  }
}
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Updated maintenance settings but indices still not delet

Post by mcapra »

What version of NLS are you using?

Can you also share the output of curator --version and curator delete
Former Nagios employee
https://www.mcapra.com/
Top
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

Re: Updated maintenance settings but indices still not delet

Post by CFT6Server »

I am on 1.3 and here's the output.

Code: Select all

[root@ indices]# curator --version
curator 1.2.2
[root@ indices]# curator delete
/usr/bin/curator delete: error: expect one of --older-than or --disk-space
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Updated maintenance settings but indices still not delet

Post by mcapra »

Cool, lets use a command that's more appropriate for your curator version:

Code: Select all

curator --debug --dry-run delete --older-than 15
Former Nagios employee
https://www.mcapra.com/
Top
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

Re: Updated maintenance settings but indices still not delet

Post by CFT6Server »

Here it is.... odd that it did not pick up anything before Sept 15??

Code: Select all

[root@ indices]# curl localhost:9200/_snapshot/Repository/_all?pretty | grep indices
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
1    "indices" : [ "logstash-2016.08.12" ], 0 --:--:-- --:--:-- --:--:--     0
00     "indices" : [ "logstash-2016.08.13" ],
23    "indices" : [ "logstash-2016.08.14" ],
11    "indices" : [ "logstash-2016.08.15" ],
5     "indices" : [ "logstash-2016.08.16" ],
     "indices" : [ "logstash-2016.08.17" ],
10    "indices" : [ "logstash-2016.08.18" ],
0     "indices" : [ "logstash-2016.08.19" ],
23    "indices" : [ "logstash-2016.08.20" ],
11    "indices" : [ "logstash-2016.08.21" ],
5     "indices" : [ "logstash-2016.08.22" ],
      "indices" : [ "logstash-2016.08.23" ],
 0    "indices" : [ "logstash-2016.08.24" ],
      "indices" : [ "logstash-2016.08.25" ],
      "indices" : [ "logstash-2016.08.26" ],
 0    "indices" : [ "logstash-2016.08.27" ],
      "indices" : [ "logstash-2016.08.28" ],
13    "indices" : [ "logstash-2016.08.29" ],
78    "indices" : [ "logstash-2016.08.30" ],
k     "indices" : [ "logstash-2016.08.31" ],
      "indices" : [ "logstash-2016.09.01" ],
      "indices" : [ "logstash-2016.09.02" ],
 0    "indices" : [ "logstash-2016.09.03" ],
 -    "indices" : [ "logstash-2016.09.04" ],
-:    "indices" : [ "logstash-2016.09.05" ],
--    "indices" : [ "logstash-2016.09.06" ],
:-    "indices" : [ "logstash-2016.09.07" ],
-     "indices" : [ "logstash-2016.09.08" ],
      "indices" : [ "logstash-2016.09.09" ],

    "indices" : [ "logstash-2016.09.10" ],
    "indices" : [ "logstash-2016.09.11" ],
    "indices" : [ "logstash-2016.09.12" ],
    "indices" : [ "logstash-2016.09.13" ],
    "indices" : [ "logstash-2016.09.14" ],
    "indices" : [ "logstash-2016.09.15" ],
    "indices" : [ "logstash-2016.09.16" ],
    "indices" : [ "logstash-2016.09.17" ],
    "indices" : [ "logstash-2016.09.18" ],
    "indices" : [ "logstash-2016.09.19" ],
    "indices" : [ "logstash-2016.09.20" ],
    "indices" : [ "logstash-2016.09.21" ],
    "indices" : [ "logstash-2016.09.22" ],
    "indices" : [ "logstash-2016.09.23" ],
    "indices" : [ "logstash-2016.09.24" ],
    "indices" : [ "logstash-2016.09.25" ],
    "indices" : [ "logstash-2016.09.26" ],
    "indices" : [ "logstash-2016.09.27" ],
    "indices" : [ "logstash-2016.09.28" ],
    "indices" : [ "logstash-2016.09.29" ],
    "indices" : [ "logstash-2016.09.30" ],
    "indices" : [ "logstash-2016.10.01" ],
    "indices" : [ "logstash-2016.10.02" ],
    "indices" : [ "logstash-2016.10.03" ],
You do not have the required permissions to view the files attached to this post.
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Updated maintenance settings but indices still not delet

Post by rkennedy »

It did, looks like the dry ran stopped after 19 days, and not 15. Let's debug a little bit as to what's being sent over to NLS.

Please replace your /usr/local/nagioslogserver/scripts/curator.sh script with what I've pasted below. (this will still function as expected, I've just modified to add debugging / output for more verbosity.)

Code: Select all

#!/bin/sh
date=$(date)

curator "$@"
echo --- >> /tmp/curatordebug.txt
echo $date >> /tmp/curatordebug.txt
echo "$@" >> /tmp/curatordebug.txt
curl -s 127.0.0.1:9200/_cluster/state/blocks?pretty >> /tmp/curatordebug.txt
touch /tmp/curatordebug.txt
chmod 777 /tmp/curatordebug.txt
Let this run for a few days (to gain a bit of data), and then post the /tmp/curatordebug.txt file for us to look at. I'm interested to see which variables are passing for your backups.
Former Nagios Employee
Top
Locked

Return to “Nagios Log Server”