How to stop a currently running snapshot?

[cdienger]

This should clear out the snapshot that seems to be stuck in progress:

Code: Select all

curl -XDELETE 'http://localhost:9200/_snapshot/NLSREPCC/curator-20191022053043'

[rferebee]

Ok, that worked. Let's leave this open until tomorrow morning. I have a snapshot scheduled to run this evening and as long as that completes without an issue I think we'll be okay.

[mbellerue]

Alright, sounds good. We'll wait to hear back from you.

[rferebee]

The snapshot started, but has not finished yet...

I'm a bit concerned because our snapshots are set to run in the evening at 22:30 and for some reason this snapshot didn't start until 05:31 this morning. The only thing I can think of is that because neither of the previous two snapshots finished and we were forced to delete them, this current snapshot is having to snap almost 20 days worth of indices. I would imagine that takes a little extra time?

Code: Select all

{
    "snapshot" : "curator-20191029053112",
    "version_id" : 1070699,
    "version" : "1.7.6",
    "indices" : [ "logstash-2019.10.08", "logstash-2019.10.09", "logstash-2019.10.10", "logstash-2019.10.11", "logstash-2019.10.12", "logstash-2019.10.13", "logstash-2019.10.14", "logstash-2019.10.15", "logstash-2019.10.16", "logstash-2019.10.17", "logstash-2019.10.18", "logstash-2019.10.19", "logstash-2019.10.20", "logstash-2019.10.21", "logstash-2019.10.22", "logstash-2019.10.23", "logstash-2019.10.24", "logstash-2019.10.25", "logstash-2019.10.26", "logstash-2019.10.27", "logstash-2019.10.28" ],
    "state" : "IN_PROGRESS",
    "start_time" : "2019-10-29T05:31:12.493Z",
    "start_time_in_millis" : 1572327072493,
    "failures" : [ ],
    "shards" : {
      "total" : 0,
      "failed" : 0,
      "successful" : 0
    }
  } ]
}

[rferebee]

I'm sure you folks are busy, but I need answers please. I don't know why these snapshots aren't working and I need your help to get them going again. Here's what's in the audit log (there's nothing else for MAINTENANCE after the top line):

Code: Select all

System MAINTENANCE Tue, 29 Oct 2019 04:31:13 -0700 Deleting snapshots more than 731 day(s) old from NLSREPCC  Nagios Log Server  
System MAINTENANCE Tue, 29 Oct 2019 03:16:01 -0700 Deleting snapshots more than 731 day(s) old from NLSREPCC fc00210a-5231-46f3-84f2-6e3c61c7ac0e Nagios Log Server  
System MAINTENANCE Tue, 29 Oct 2019 03:15:43 -0700 Deleting snapshots more than 731 day(s) old from NLSREPCC 4deb767e-abbd-43f0-8839-049760687e98 Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:31:14 -0700 Deleting snapshots more than 731 day(s) old from NLSREPCC 29dbb5cc-f936-4f0e-8a41-26b2277c7083 Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:31:02 -0700 Deleting indexes more than 22 day(s) old. fc00210a-5231-46f3-84f2-6e3c61c7ac0e Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:31:02 -0700 Closing indexes 22 day(s) old. fc00210a-5231-46f3-84f2-6e3c61c7ac0e Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:30:46 -0700 Deleting indexes more than 22 day(s) old. 29dbb5cc-f936-4f0e-8a41-26b2277c7083 Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:30:46 -0700 Deleting indexes more than 22 day(s) old.  Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:30:43 -0700 Deleting indexes more than 22 day(s) old. 4deb767e-abbd-43f0-8839-049760687e98 Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:30:43 -0700 Closing indexes 22 day(s) old. 4deb767e-abbd-43f0-8839-049760687e98 Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:30:15 -0700 Closing indexes 22 day(s) old. 29dbb5cc-f936-4f0e-8a41-26b2277c7083 Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:30:15 -0700 Closing indexes 22 day(s) old.  Nagios Log Server 

Going on 4 hours and the snapshot still says it's running.

[cdienger]

It may be requiring a bit more time since some were missed previously. If you provide a profile we can check to see if the curator script is still running.

[rferebee]

It may be requiring a bit more time since some were missed previously. If you provide a profile we can check to see if the curator script is still running.

PM sent with current system profile. Thank you.

[cdienger]

Curator doesn't appear to be running. Tail the jobs.log again and force another maintenance job to see if there is a snapshot that may be hung and terminated.

[rferebee]

Here's what I got:

Code: Select all

root@nagioslscc2:/root> tail -f /usr/local/nagioslogserver/var/jobs.log
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command do_maintenance with args ' ' for job id: snapshots_maintenance
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh close indices --older-than 22 --time-unit days --timestring %Y.%m.%d
Return: 0
2019-10-29 12:52:02,586 INFO      Job starting: close indices
2019-10-29 12:52:02,613 INFO      Action close will be performed on the following indices: [u'logstash-2019.08.23']
2019-10-29 12:52:03,937 INFO      Provided indices successfully sealed. (Shown with --debug flag enabled.)
2019-10-29 12:52:04,140 INFO      Job completed successfully.
-----
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete indices --older-than 22 --time-unit days --timestring %Y.%m.%d
Return: 0
2019-10-29 12:52:04,308 INFO      Job starting: delete indices
2019-10-29 12:52:04,402 INFO      Pruning Kibana-related indices to prevent accidental deletion.
2019-10-29 12:52:04,402 INFO      Action delete will be performed on the following indices: [u'logstash-2019.08.23']
2019-10-29 12:52:04,403 INFO      Deleting indices as a batch operation:
2019-10-29 12:52:04,404 INFO      ---deleting index logstash-2019.08.23
2019-10-29 12:52:04,497 INFO      Job completed successfully.
-----
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "NLSREPCC" --ignore_unavailable indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Return: 1
2019-10-29 12:52:05,647 INFO      Job starting: snapshot indices
2019-10-29 12:52:05,647 WARNING   Overriding default connection timeout.  New timeout: 21600
2019-10-29 12:52:05,676 INFO      Action snapshot will be performed on the following indices: [u'logstash-2019.10.08', u'logstash-2019.10.09', u'logstash-2019.10.10', u'logstash-2019.10.11', u'logstash-2019.10.12', u'logstash-2019.10.13', u'logstash-2019.10.14', u'logstash-2019.10.15', u'logstash-2019.10.16', u'logstash-2019.10.17', u'logstash-2019.10.18', u'logstash-2019.10.19', u'logstash-2019.10.20', u'logstash-2019.10.21', u'logstash-2019.10.22', u'logstash-2019.10.23', u'logstash-2019.10.24', u'logstash-2019.10.25', u'logstash-2019.10.26', u'logstash-2019.10.27', u'logstash-2019.10.28']
2019-10-29 12:52:05,719 ERROR     Snapshot already in progress: [{u'stats': {u'number_of_files': 17165, u'total_size_in_bytes': 4852136037890, u'time_in_millis': 2819798, u'processed_files': 16284, u'processed_size_in_bytes': 4610107613032, u'start_time_in_millis': 1572369101668}, u'repository': u'NLSREPCC', u'shards_stats': {u'started': 3, u'failed': 5, u'done': 95, u'finalizing': 0, u'initializing': 2, u'total': 105}, u'state': u'STARTED', u'snapshot': u'curator-20191029053112', u'indices': {u'logstash-2019.10.08': {u'stats': {u'number_of_files': 497, u'total_size_in_bytes': 164779602960, u'time_in_millis': 26595132, u'processed_files': 497, u'processed_size_in_bytes': 164779602960, u'start_time_in_millis': 1572344264601}, u'shards': {u'1': {u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 54935453153, u'time_in_millis': 4081309, u'processed_files': 181, u'processed_size_in_bytes': 54935453153, u'start_time_in_millis': 1572344264601}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 1, u'total_size_in_bytes': 1609, u'time_in_millis': 853, u'processed_files': 1, u'processed_size_in_bytes': 1609, u'start_time_in_millis': 1572366745336}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 1, u'total_size_in_bytes': 2068, u'time_in_millis': 840, u'processed_files': 1, u'processed_size_in_bytes': 2068, u'start_time_in_millis': 1572359652742}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 151, u'total_size_in_bytes': 54916756936, u'time_in_millis': 4108905, u'processed_files': 151, u'processed_size_in_bytes': 54916756936, u'start_time_in_millis': 1572366750828}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 163, u'total_size_in_bytes': 54927389194, u'time_in_millis': 4107799, u'processed_files': 163, u'processed_size_in_bytes': 54927389194, u'start_time_in_millis': 1572362637342}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.09': {u'stats': {u'number_of_files': 504, u'total_size_in_bytes': 161754179126, u'time_in_millis': 15838891, u'processed_files': 504, u'processed_size_in_bytes': 161754179126, u'start_time_in_millis': 1572361794998}, u'shards': {u'1': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 54048475508, u'time_in_millis': 4256707, u'processed_files': 178, u'processed_size_in_bytes': 54048475508, u'start_time_in_millis': 1572364844833}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 142, u'total_size_in_bytes': 53856516527, u'time_in_millis': 4199456, u'processed_files': 142, u'processed_size_in_bytes': 53856516527, u'start_time_in_millis': 1572361794998}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 1, u'total_size_in_bytes': 2251, u'time_in_millis': 813, u'processed_files': 1, u'processed_size_in_bytes': 2251, u'start_time_in_millis': 1572377633076}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 1, u'total_size_in_bytes': 1733, u'time_in_millis': 1142, u'processed_files': 1, u'processed_size_in_bytes': 1733, u'start_time_in_millis': 1572377631769}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 182, u'total_size_in_bytes': 53849183107, u'time_in_millis': 4020042, u'processed_files': 182, u'processed_size_in_bytes': 53849183107, u'start_time_in_millis': 1572373191458}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.19': {u'stats': {u'number_of_files': 789, u'total_size_in_bytes': 219022792239, u'time_in_millis': 28026275, u'processed_files': 789, u'processed_size_in_bytes': 219022792239, u'start_time_in_millis': 1572343824085}, u'shards': {u'1': {u'stats': {u'number_of_files': 159, u'total_size_in_bytes': 43831674150, u'time_in_millis': 3215483, u'processed_files': 159, u'processed_size_in_bytes': 43831674150, u'start_time_in_millis': 1572349813519}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 162, u'total_size_in_bytes': 43787432086, u'time_in_millis': 3451721, u'processed_files': 162, u'processed_size_in_bytes': 43787432086, u'start_time_in_millis': 1572365564363}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 156, u'total_size_in_bytes': 43801320726, u'time_in_millis': 3193904, u'processed_files': 156, u'processed_size_in_bytes': 43801320726, u'start_time_in_millis': 1572343824085}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 162, u'total_size_in_bytes': 43853482463, u'time_in_millis': 2833728, u'processed_files': 162, u'processed_size_in_bytes': 43853482463, u'start_time_in_millis': 1572369016632}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 150, u'total_size_in_bytes': 43748882814, u'time_in_millis': 3190052, u'processed_files': 150, u'processed_size_in_bytes': 43748882814, u'start_time_in_millis': 1572346623282}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.18': {u'stats': {u'number_of_files': 818, u'total_size_in_bytes': 280765886240, u'time_in_millis': 19336935, u'processed_files': 818, u'processed_size_in_bytes': 280765886240, u'start_time_in_millis': 1572331050292}, u'shards': {u'1': {u'stats': {u'number_of_files': 163, u'total_size_in_bytes': 56227667368, u'time_in_millis': 4112182, u'processed_files': 163, u'processed_size_in_bytes': 56227667368, u'start_time_in_millis': 1572335123369}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 172, u'total_size_in_bytes': 56136510688, u'time_in_millis': 4184222, u'processed_files': 172, u'processed_size_in_bytes': 56136510688, u'start_time_in_millis': 1572346203005}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 160, u'total_size_in_bytes': 56105884207, u'time_in_millis': 4073025, u'processed_files': 160, u'processed_size_in_bytes': 56105884207, u'start_time_in_millis': 1572331050292}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 148, u'total_size_in_bytes': 56249798351, u'time_in_millis': 4159919, u'processed_files': 148, u'processed_size_in_bytes': 56249798351, u'start_time_in_millis': 1572335075808}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 175, u'total_size_in_bytes': 56046025626, u'time_in_millis': 4195238, u'processed_files': 175, u'processed_size_in_bytes': 56046025626, u'start_time_in_millis': 1572339235767}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.13': {u'stats': {u'number_of_files': 799, u'total_size_in_bytes': 208577990007, u'time_in_millis': 30082128, u'processed_files': 648, u'processed_size_in_bytes': 168877875444, u'start_time_in_millis': 1572347549510}, u'shards': {u'1': {u'stats': {u'number_of_files': 164, u'total_size_in_bytes': 41711368614, u'time_in_millis': 3115322, u'processed_files': 164, u'processed_size_in_bytes': 41711368614, u'start_time_in_millis': 1572347549510}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 161, u'total_size_in_bytes': 41713031545, u'time_in_millis': 3116389, u'processed_files': 161, u'processed_size_in_bytes': 41713031545, u'start_time_in_millis': 1572374515249}, u'stage': u'DONE'}, u'3': {u'reason': u'IndexShardSnapshotFailedException[[logstash-2019.10.13][3] Failed to perform snapshot (index files)]; nested: IOException[Input/output error]; ', u'stats': {u'number_of_files': 158, u'total_size_in_bytes': 41725113766, u'time_in_millis': 255276, u'processed_files': 7, u'processed_size_in_bytes': 2024999203, u'start_time_in_millis': 1572371134950}, u'stage': u'FAILURE'}, u'2': {u'stats': {u'number_of_files': 167, u'total_size_in_bytes': 41706369387, u'time_in_millis': 3116973, u'processed_files': 167, u'processed_size_in_bytes': 41706369387, u'start_time_in_millis': 1572350387320}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 149, u'total_size_in_bytes': 41722106695, u'time_in_millis': 3124839, u'processed_files': 149, u'processed_size_in_bytes': 41722106695, u'start_time_in_millis': 1572371390333}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 1, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.12': {u'stats': {u'number_of_files': 749, u'total_size_in_bytes': 214665145963, u'time_in_millis': 46700695, u'processed_files': 656, u'processed_size_in_bytes': 185048279825, u'start_time_in_millis': 1572330933264}, u'shards': {u'1': {u'stats': {u'number_of_files': 129, u'total_size_in_bytes': 42916466512, u'time_in_millis': 3200154, u'processed_files': 129, u'processed_size_in_bytes': 42916466512, u'start_time_in_millis': 1572358092510}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 147, u'total_size_in_bytes': 42894795954, u'time_in_millis': 3152232, u'processed_files': 147, u'processed_size_in_bytes': 42894795954, u'start_time_in_millis': 1572355360484}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 182, u'total_size_in_bytes': 43015783758, u'time_in_millis': 3192206, u'processed_files': 182, u'processed_size_in_bytes': 43015783758, u'start_time_in_millis': 1572334116596}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 150, u'total_size_in_bytes': 42920383705, u'time_in_millis': 3183277, u'processed_files': 150, u'processed_size_in_bytes': 42920383705, u'start_time_in_millis': 1572330933264}, u'stage': u'DONE'}, u'4': {u'node': u'pcqYieAcSAK7http3p3yzQ', u'stats': {u'number_of_files': 141, u'total_size_in_bytes': 42917716034, u'time_in_millis': 0, u'processed_files': 48, u'processed_size_in_bytes': 13300849896, u'start_time_in_millis': 1572377633959}, u'stage': u'STARTED'}}, u'shards_stats': {u'started': 1, u'failed': 0, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.11': {u'stats': {u'number_of_files': 887, u'total_size_in_bytes': 261491516465, u'time_in_millis': 31910673, u'processed_files': 769, u'processed_size_in_bytes': 218363307508, u'start_time_in_millis': 1572339235671}, u'shards': {u'1': {u'stats': {u'number_of_files': 184, u'total_size_in_bytes': 52372248459, u'time_in_millis': 3925962, u'processed_files': 184, u'processed_size_in_bytes': 52372248459, u'start_time_in_millis': 1572352392189}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 166, u'total_size_in_bytes': 52328962500, u'time_in_millis': 3836446, u'processed_files': 166, u'processed_size_in_bytes': 52328962500, u'start_time_in_millis': 1572339235671}, u'stage': u'DONE'}, u'3': {u'reason': u'IndexShardSnapshotFailedException[[logstash-2019.10.11][3] Failed to perform snapshot (index files)]; nested: FileSystemException[/nlsrepcc/indices/logstash-2019.10.11/3/__4c: Resource temporarily unavailable]; ', u'stats': {u'number_of_files': 187, u'total_size_in_bytes': 52254623914, u'time_in_millis': 2909600, u'processed_files': 157, u'processed_size_in_bytes': 35939162106, u'start_time_in_millis': 1572368223987}, u'stage': u'FAILURE'}, u'2': {u'stats': {u'number_of_files': 169, u'total_size_in_bytes': 52264966423, u'time_in_millis': 3832091, u'processed_files': 169, u'processed_size_in_bytes': 52264966423, u'start_time_in_millis': 1572341665937}, u'stage': u'DONE'}, u'4': {u'reason': u'IndexShardSnapshotFailedException[[logstash-2019.10.11][4] Failed to perform snapshot (index files)]; nested: IOException[Input/output error]; ', u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 52270715169, u'time_in_millis': 2050210, u'processed_files': 93, u'processed_size_in_bytes': 25457968020, u'start_time_in_millis': 1572369096134}, u'stage': u'FAILURE'}}, u'shards_stats': {u'started': 0, u'failed': 2, u'done': 3, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.10': {u'stats': {u'number_of_files': 872, u'total_size_in_bytes': 275976574379, u'time_in_millis': 31033802, u'processed_files': 801, u'processed_size_in_bytes': 251893660003, u'start_time_in_millis': 1572327058643}, u'shards': {u'1': {u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 55281974802, u'time_in_millis': 4041932, u'processed_files': 181, u'processed_size_in_bytes': 55281974802, u'start_time_in_millis': 1572327058643}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 172, u'total_size_in_bytes': 55114163661, u'time_in_millis': 4081469, u'processed_files': 172, u'processed_size_in_bytes': 55114163661, u'start_time_in_millis': 1572330138507}, u'stage': u'DONE'}, u'3': {u'reason': u'IndexShardSnapshotFailedException[[logstash-2019.10.10][3] Failed to perform snapshot (index files)]; nested: IOException[Input/output error]; ', u'stats': {u'number_of_files': 172, u'total_size_in_bytes': 55108893362, u'time_in_millis': 2408430, u'processed_files': 101, u'processed_size_in_bytes': 31025978986, u'start_time_in_millis': 1572353029167}, u'stage': u'FAILURE'}, u'2': {u'stats': {u'number_of_files': 169, u'total_size_in_bytes': 55237337185, u'time_in_millis': 4033960, u'processed_files': 169, u'processed_size_in_bytes': 55237337185, u'start_time_in_millis': 1572350015230}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 55234205369, u'time_in_millis': 4043166, u'processed_files': 178, u'processed_size_in_bytes': 55234205369, u'start_time_in_millis': 1572354049279}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 1, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.17': {u'stats': {u'number_of_files': 893, u'total_size_in_bytes': 279019754424, u'time_in_millis': 21160285, u'processed_files': 761, u'processed_size_in_bytes': 243499524871, u'start_time_in_millis': 1572357154324}, u'shards': {u'1': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 55809460912, u'time_in_millis': 4169636, u'processed_files': 178, u'processed_size_in_bytes': 55809460912, u'start_time_in_millis': 1572374144973}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 184, u'total_size_in_bytes': 55822917265, u'time_in_millis': 4401125, u'processed_files': 184, u'processed_size_in_bytes': 55822917265, u'start_time_in_millis': 1572365994769}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 166, u'total_size_in_bytes': 55781285152, u'time_in_millis': 4245499, u'processed_files': 166, u'processed_size_in_bytes': 55781285152, u'start_time_in_millis': 1572357549372}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 55775906733, u'time_in_millis': 4231159, u'processed_files': 181, u'processed_size_in_bytes': 55775906733, u'start_time_in_millis': 1572357154324}, u'stage': u'DONE'}, u'4': {u'node': u'pcqYieAcSAK7http3p3yzQ', u'stats': {u'number_of_files': 184, u'total_size_in_bytes': 55830184362, u'time_in_millis': 0, u'processed_files': 52, u'processed_size_in_bytes': 20309954809, u'start_time_in_millis': 1572377211590}, u'stage': u'STARTED'}}, u'shards_stats': {u'started': 1, u'failed': 0, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.16': {u'stats': {u'number_of_files': 979, u'total_size_in_bytes': 286245005801, u'time_in_millis': 30627354, u'processed_files': 979, u'processed_size_in_bytes': 286245005801, u'start_time_in_millis': 1572337596367}, u'shards': {u'1': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 57138927015, u'time_in_millis': 4288141, u'processed_files': 178, u'processed_size_in_bytes': 57138927015, u'start_time_in_millis': 1572363935580}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 186, u'total_size_in_bytes': 57372637481, u'time_in_millis': 4280832, u'processed_files': 186, u'processed_size_in_bytes': 57372637481, u'start_time_in_millis': 1572359653722}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 253, u'total_size_in_bytes': 57481970817, u'time_in_millis': 4314117, u'processed_files': 253, u'processed_size_in_bytes': 57481970817, u'start_time_in_millis': 1572341888785}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 175, u'total_size_in_bytes': 57154755843, u'time_in_millis': 4292173, u'processed_files': 175, u'processed_size_in_bytes': 57154755843, u'start_time_in_millis': 1572337596367}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 187, u'total_size_in_bytes': 57096714645, u'time_in_millis': 4214871, u'processed_files': 187, u'processed_size_in_bytes': 57096714645, u'start_time_in_millis': 1572355437688}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.15': {u'stats': {u'number_of_files': 1022, u'total_size_in_bytes': 269604822100, u'time_in_millis': 25671669, u'processed_files': 1022, u'processed_size_in_bytes': 269604822100, u'start_time_in_millis': 1572339892580}, u'shards': {u'1': {u'stats': {u'number_of_files': 154, u'total_size_in_bytes': 53800509094, u'time_in_millis': 3937401, u'processed_files': 154, u'processed_size_in_bytes': 53800509094, u'start_time_in_millis': 1572349472191}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 298, u'total_size_in_bytes': 54109857026, u'time_in_millis': 3973928, u'processed_files': 298, u'processed_size_in_bytes': 54109857026, u'start_time_in_millis': 1572345498155}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 259, u'total_size_in_bytes': 54074938405, u'time_in_millis': 4045734, u'processed_files': 259, u'processed_size_in_bytes': 54074938405, u'start_time_in_millis': 1572348346161}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 169, u'total_size_in_bytes': 53832418563, u'time_in_millis': 4178563, u'processed_files': 169, u'processed_size_in_bytes': 53832418563, u'start_time_in_millis': 1572361385686}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 142, u'total_size_in_bytes': 53787099012, u'time_in_millis': 3930893, u'processed_files': 142, u'processed_size_in_bytes': 53787099012, u'start_time_in_millis': 1572339892580}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.14': {u'stats': {u'number_of_files': 1013, u'total_size_in_bytes': 259968476287, u'time_in_millis': 11590897, u'processed_files': 1013, u'processed_size_in_bytes': 259968476287, u'start_time_in_millis': 1572327058536}, u'shards': {u'1': {u'stats': {u'number_of_files': 172, u'total_size_in_bytes': 51905007310, u'time_in_millis': 3795434, u'processed_files': 172, u'processed_size_in_bytes': 51905007310, u'start_time_in_millis': 1572334853999}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 240, u'total_size_in_bytes': 52086529312, u'time_in_millis': 3817387, u'processed_files': 240, u'processed_size_in_bytes': 52086529312, u'start_time_in_millis': 1572331100692}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 242, u'total_size_in_bytes': 52118148172, u'time_in_millis': 3888976, u'processed_files': 242, u'processed_size_in_bytes': 52118148172, u'start_time_in_millis': 1572327076106}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 51919465710, u'time_in_millis': 3857174, u'processed_files': 178, u'processed_size_in_bytes': 51919465710, u'start_time_in_millis': 1572327076027}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 51939325783, u'time_in_millis': 3808868, u'processed_files': 181, u'processed_size_in_bytes': 51939325783, u'start_time_in_millis': 1572327058536}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.22': {u'stats': {u'number_of_files': 1404, u'total_size_in_bytes': 271721502415, u'time_in_millis': 28301848, u'processed_files': 1229, u'processed_size_in_bytes': 242613387832, u'start_time_in_millis': 1572327058544}, u'shards': {u'1': {u'stats': {u'number_of_files': 206, u'total_size_in_bytes': 54291038837, u'time_in_millis': 3991670, u'processed_files': 206, u'processed_size_in_bytes': 54291038837, u'start_time_in_millis': 1572327058544}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 148, u'total_size_in_bytes': 54164523999, u'time_in_millis': 3964231, u'processed_files': 148, u'processed_size_in_bytes': 54164523999, u'start_time_in_millis': 1572330108758}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 437, u'total_size_in_bytes': 54511225627, u'time_in_millis': 4047507, u'processed_files': 437, u'processed_size_in_bytes': 54511225627, u'start_time_in_millis': 1572340112663}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 206, u'total_size_in_bytes': 54267759862, u'time_in_millis': 3986355, u'processed_files': 206, u'processed_size_in_bytes': 54267759862, u'start_time_in_millis': 1572330867476}, u'stage': u'DONE'}, u'4': {u'reason': u'IndexShardSnapshotFailedException[[logstash-2019.10.22][4] Failed to perform snapshot (index files)]; nested: IOException[Input/output error]; ', u'stats': {u'number_of_files': 407, u'total_size_in_bytes': 54486954090, u'time_in_millis': 1950707, u'processed_files': 232, u'processed_size_in_bytes': 25378839507, u'start_time_in_millis': 1572353409685}, u'stage': u'FAILURE'}}, u'shards_stats': {u'started': 0, u'failed': 1, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.23': {u'stats': {u'number_of_files': 790, u'total_size_in_bytes': 242767025560, u'time_in_millis': 30076610, u'processed_files': 790, u'processed_size_in_bytes': 242767025560, u'start_time_in_millis': 1572327077571}, u'shards': {u'1': {u'stats': {u'number_of_files': 154, u'total_size_in_bytes': 48565747318, u'time_in_millis': 3615300, u'processed_files': 154, u'processed_size_in_bytes': 48565747318, u'start_time_in_millis': 1572350664941}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 151, u'total_size_in_bytes': 48579930068, u'time_in_millis': 3575139, u'processed_files': 151, u'processed_size_in_bytes': 48579930068, u'start_time_in_millis': 1572327077571}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 154, u'total_size_in_bytes': 48513651444, u'time_in_millis': 3550844, u'processed_files': 154, u'processed_size_in_bytes': 48513651444, u'start_time_in_millis': 1572343072389}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 163, u'total_size_in_bytes': 48569545646, u'time_in_millis': 3649705, u'processed_files': 163, u'processed_size_in_bytes': 48569545646, u'start_time_in_millis': 1572353504476}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 168, u'total_size_in_bytes': 48538151084, u'time_in_millis': 3530644, u'processed_files': 168, u'processed_size_in_bytes': 48538151084, u'start_time_in_millis': 1572340679668}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.20': {u'stats': {u'number_of_files': 492, u'total_size_in_bytes': 135893649462, u'time_in_millis': 2819798, u'processed_files': 351, u'processed_size_in_bytes': 95021672774, u'start_time_in_millis': 1572369101668}, u'shards': {u'1': {u'node': u'pcqYieAcSAK7http3p3yzQ', u'stats': {u'number_of_files': 165, u'total_size_in_bytes': 45334605671, u'time_in_millis': 0, u'processed_files': 24, u'processed_size_in_bytes': 4462628983, u'start_time_in_millis': 1572378314712}, u'stage': u'STARTED'}, u'0': {u'stats': {u'number_of_files': 171, u'total_size_in_bytes': 45300601775, u'time_in_millis': 3376276, u'processed_files': 171, u'processed_size_in_bytes': 45300601775, u'start_time_in_millis': 1572334220018}, u'stage': u'DONE'}, u'3': {u'node': u'pcqYieAcSAK7http3p3yzQ', u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'INIT'}, u'2': {u'node': u'pcqYieAcSAK7http3p3yzQ', u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'INIT'}, u'4': {u'stats': {u'number_of_files': 156, u'total_size_in_bytes': 45258442016, u'time_in_millis': 2819798, u'processed_files': 156, u'processed_size_in_bytes': 45258442016, u'start_time_in_millis': 1572369101668}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 1, u'failed': 0, u'done': 2, u'finalizing': 0, u'initializing': 2, u'total': 5}}, u'logstash-2019.10.21': {u'stats': {u'number_of_files': 672, u'total_size_in_bytes': 156990220334, u'time_in_millis': 35273536, u'processed_files': 672, u'processed_size_in_bytes': 156990220334, u'start_time_in_millis': 1572337917815}, u'shards': {u'1': {u'stats': {u'number_of_files': 102, u'total_size_in_bytes': 26894151137, u'time_in_millis': 2023995, u'processed_files': 102, u'processed_size_in_bytes': 26894151137, u'start_time_in_millis': 1572356318259}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 144, u'total_size_in_bytes': 26992848718, u'time_in_millis': 2044866, u'processed_files': 144, u'processed_size_in_bytes': 26992848718, u'start_time_in_millis': 1572371146485}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 146, u'total_size_in_bytes': 26971916112, u'time_in_millis': 1977300, u'processed_files': 146, u'processed_size_in_bytes': 26971916112, u'start_time_in_millis': 1572342158579}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 114, u'total_size_in_bytes': 26881708675, u'time_in_millis': 1974642, u'processed_files': 114, u'processed_size_in_bytes': 26881708675, u'start_time_in_millis': 1572337917815}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 166, u'total_size_in_bytes': 49249595692, u'time_in_millis': 1339826, u'processed_files': 166, u'processed_size_in_bytes': 49249595692, u'start_time_in_millis': 1572344135943}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.26': {u'stats': {u'number_of_files': 766, u'total_size_in_bytes': 204978273916, u'time_in_millis': 17188424, u'processed_files': 766, u'processed_size_in_bytes': 204978273916, u'start_time_in_millis': 1572327076041}, u'shards': {u'1': {u'stats': {u'number_of_files': 155, u'total_size_in_bytes': 40977660545, u'time_in_millis': 3048689, u'processed_files': 155, u'processed_size_in_bytes': 40977660545, u'start_time_in_millis': 1572341215776}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 140, u'total_size_in_bytes': 41003683675, u'time_in_millis': 3011502, u'processed_files': 140, u'processed_size_in_bytes': 41003683675, u'start_time_in_millis': 1572334073027}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 167, u'total_size_in_bytes': 41020972013, u'time_in_millis': 3032103, u'processed_files': 167, u'processed_size_in_bytes': 41020972013, u'start_time_in_millis': 1572327076605}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 158, u'total_size_in_bytes': 40983311677, u'time_in_millis': 2999481, u'processed_files': 158, u'processed_size_in_bytes': 40983311677, u'start_time_in_millis': 1572334918136}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 146, u'total_size_in_bytes': 40992646006, u'time_in_millis': 3062429, u'processed_files': 146, u'processed_size_in_bytes': 40992646006, u'start_time_in_millis': 1572327076041}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.27': {u'stats': {u'number_of_files': 778, u'total_size_in_bytes': 205031912054, u'time_in_millis': 27638650, u'processed_files': 778, u'processed_size_in_bytes': 205031912054, u'start_time_in_millis': 1572333839458}, u'shards': {u'1': {u'stats': {u'number_of_files': 161, u'total_size_in_bytes': 41011213040, u'time_in_millis': 2997016, u'processed_files': 161, u'processed_size_in_bytes': 41011213040, u'start_time_in_millis': 1572347018058}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 161, u'total_size_in_bytes': 40992839647, u'time_in_millis': 3005369, u'processed_files': 161, u'processed_size_in_bytes': 40992839647, u'start_time_in_millis': 1572333839458}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 149, u'total_size_in_bytes': 41006452803, u'time_in_millis': 3027749, u'processed_files': 149, u'processed_size_in_bytes': 41006452803, u'start_time_in_millis': 1572337084573}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 161, u'total_size_in_bytes': 41014987754, u'time_in_millis': 3135799, u'processed_files': 161, u'processed_size_in_bytes': 41014987754, u'start_time_in_millis': 1572358342309}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 146, u'total_size_in_bytes': 41006418810, u'time_in_millis': 3016297, u'processed_files': 146, u'processed_size_in_bytes': 41006418810, u'start_time_in_millis': 1572338649480}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.24': {u'stats': {u'number_of_files': 863, u'total_size_in_bytes': 276544830501, u'time_in_millis': 35560510, u'processed_files': 863, u'processed_size_in_bytes': 276544830501, u'start_time_in_millis': 1572327076717}, u'shards': {u'1': {u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 55218804771, u'time_in_millis': 4052499, u'processed_files': 181, u'processed_size_in_bytes': 55218804771, u'start_time_in_millis': 1572327076717}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 157, u'total_size_in_bytes': 55347330117, u'time_in_millis': 4118011, u'processed_files': 157, u'processed_size_in_bytes': 55347330117, u'start_time_in_millis': 1572343431401}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 169, u'total_size_in_bytes': 55459692927, u'time_in_millis': 4110405, u'processed_files': 169, u'processed_size_in_bytes': 55459692927, u'start_time_in_millis': 1572330965329}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 55256455898, u'time_in_millis': 4124274, u'processed_files': 178, u'processed_size_in_bytes': 55256455898, u'start_time_in_millis': 1572358512953}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 55262546788, u'time_in_millis': 4053202, u'processed_files': 178, u'processed_size_in_bytes': 55262546788, u'start_time_in_millis': 1572331129336}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.25': {u'stats': {u'number_of_files': 734, u'total_size_in_bytes': 216167937453, u'time_in_millis': 43491614, u'processed_files': 734, u'processed_size_in_bytes': 216167937453, u'start_time_in_millis': 1572330652820}, u'shards': {u'1': {u'stats': {u'number_of_files': 135, u'total_size_in_bytes': 43203877507, u'time_in_millis': 3366481, u'processed_files': 135, u'processed_size_in_bytes': 43203877507, u'start_time_in_millis': 1572361478176}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 135, u'total_size_in_bytes': 43195186864, u'time_in_millis': 3166900, u'processed_files': 135, u'processed_size_in_bytes': 43195186864, u'start_time_in_millis': 1572335182588}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 155, u'total_size_in_bytes': 43259855203, u'time_in_millis': 3186584, u'processed_files': 155, u'processed_size_in_bytes': 43259855203, u'start_time_in_millis': 1572330652820}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 150, u'total_size_in_bytes': 43223040996, u'time_in_millis': 3284614, u'processed_files': 150, u'processed_size_in_bytes': 43223040996, u'start_time_in_millis': 1572370859820}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 159, u'total_size_in_bytes': 43285976883, u'time_in_millis': 3268923, u'processed_files': 159, u'processed_size_in_bytes': 43285976883, u'start_time_in_millis': 1572354280394}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.28': {u'stats': {u'number_of_files': 844, u'total_size_in_bytes': 260168940204, u'time_in_millis': 32251135, u'processed_files': 844, u'processed_size_in_bytes': 260168940204, u'start_time_in_millis': 1572336844876}, u'shards': {u'1': {u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 52048167477, u'time_in_millis': 3894403, u'processed_files': 181, u'processed_size_in_bytes': 52048167477, u'start_time_in_millis': 1572361292986}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 142, u'total_size_in_bytes': 51977122819, u'time_in_millis': 3808898, u'processed_files': 142, u'processed_size_in_bytes': 51977122819, u'start_time_in_millis': 1572338349564}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 175, u'total_size_in_bytes': 52047038243, u'time_in_millis': 3834506, u'processed_files': 175, u'processed_size_in_bytes': 52047038243, u'start_time_in_millis': 1572336844876}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 172, u'total_size_in_bytes': 52042546090, u'time_in_millis': 3906881, u'processed_files': 172, u'processed_size_in_bytes': 52042546090, u'start_time_in_millis': 1572337308851}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 174, u'total_size_in_bytes': 52054065575, u'time_in_millis': 3908274, u'processed_files': 174, u'processed_size_in_bytes': 52054065575, u'start_time_in_millis': 1572365187737}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}}}]
2019-10-29 12:52:05,720 WARNING   Job did not complete successfully.
-----
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete snapshots --older-than 731 --time-unit days --timestring %Y%m%d --repository "NLSREPCC"
Return: 0
2019-10-29 12:52:05,902 INFO      Job starting: delete snapshots
2019-10-29 12:52:06,388 WARNING   No snapshots matched provided args.
No snapshots matched provided args.
-----
SUCCESS
Processed 0 node jobs.
Processed 1 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated

[cdienger]

Disable the maintenance job and we'll try to get this caught up manually.

Disable by setting the next run time to somewhere in the distant future and then run this from the command line:

Code: Select all

curl -XDELETE 'http://localhost:9200/_snapshot/NLSREPCC/curator-curator-20191029053112'

/usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "NLSREPCC" --ignore_unavailable indices --older-than 15 --time-unit days --timestring %Y.%m.%d
/usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "NLSREPCC" --ignore_unavailable indices --older-than 10 --time-unit days --timestring %Y.%m.%d
/usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "NLSREPCC" --ignore_unavailable indices --older-than 5 --time-unit days --timestring %Y.%m.%d
/usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "NLSREPCC" --ignore_unavailable indices --older-than 1 --time-unit days --timestring %Y.%m.%d

Each run of the curator script may take a long time to finish, but it should be a bit more successful with this method. Once it is caught up you can set the maintenance job back to run everyday.