Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

How to stop a currently running snapshot?

https://support.nagios.com/forum/viewtopic.php?t=56121

Page 4 of 6

Re: How to stop a currently running snapshot?

Posted: Fri Oct 25, 2019 9:43 am
by cdienger
This should clear out the snapshot that seems to be stuck in progress:

Code: Select all

curl -XDELETE 'http://localhost:9200/_snapshot/NLSREPCC/curator-20191022053043'

Re: How to stop a currently running snapshot?

Posted: Mon Oct 28, 2019 9:42 am
by rferebee
Ok, that worked. Let's leave this open until tomorrow morning. I have a snapshot scheduled to run this evening and as long as that completes without an issue I think we'll be okay.

Re: How to stop a currently running snapshot?

Posted: Mon Oct 28, 2019 3:36 pm
by mbellerue
Alright, sounds good. We'll wait to hear back from you.

Re: How to stop a currently running snapshot?

Posted: Tue Oct 29, 2019 9:30 am
by rferebee
The snapshot started, but has not finished yet...

I'm a bit concerned because our snapshots are set to run in the evening at 22:30 and for some reason this snapshot didn't start until 05:31 this morning. The only thing I can think of is that because neither of the previous two snapshots finished and we were forced to delete them, this current snapshot is having to snap almost 20 days worth of indices. I would imagine that takes a little extra time?

Code: Select all

{
    "snapshot" : "curator-20191029053112",
    "version_id" : 1070699,
    "version" : "1.7.6",
    "indices" : [ "logstash-2019.10.08", "logstash-2019.10.09", "logstash-2019.10.10", "logstash-2019.10.11", "logstash-2019.10.12", "logstash-2019.10.13", "logstash-2019.10.14", "logstash-2019.10.15", "logstash-2019.10.16", "logstash-2019.10.17", "logstash-2019.10.18", "logstash-2019.10.19", "logstash-2019.10.20", "logstash-2019.10.21", "logstash-2019.10.22", "logstash-2019.10.23", "logstash-2019.10.24", "logstash-2019.10.25", "logstash-2019.10.26", "logstash-2019.10.27", "logstash-2019.10.28" ],
    "state" : "IN_PROGRESS",
    "start_time" : "2019-10-29T05:31:12.493Z",
    "start_time_in_millis" : 1572327072493,
    "failures" : [ ],
    "shards" : {
      "total" : 0,
      "failed" : 0,
      "successful" : 0
    }
  } ]
}

Re: How to stop a currently running snapshot?

Posted: Tue Oct 29, 2019 1:25 pm
by rferebee
I'm sure you folks are busy, but I need answers please. I don't know why these snapshots aren't working and I need your help to get them going again. Here's what's in the audit log (there's nothing else for MAINTENANCE after the top line):

Code: Select all

System MAINTENANCE Tue, 29 Oct 2019 04:31:13 -0700 Deleting snapshots more than 731 day(s) old from NLSREPCC  Nagios Log Server  
System MAINTENANCE Tue, 29 Oct 2019 03:16:01 -0700 Deleting snapshots more than 731 day(s) old from NLSREPCC fc00210a-5231-46f3-84f2-6e3c61c7ac0e Nagios Log Server  
System MAINTENANCE Tue, 29 Oct 2019 03:15:43 -0700 Deleting snapshots more than 731 day(s) old from NLSREPCC 4deb767e-abbd-43f0-8839-049760687e98 Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:31:14 -0700 Deleting snapshots more than 731 day(s) old from NLSREPCC 29dbb5cc-f936-4f0e-8a41-26b2277c7083 Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:31:02 -0700 Deleting indexes more than 22 day(s) old. fc00210a-5231-46f3-84f2-6e3c61c7ac0e Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:31:02 -0700 Closing indexes 22 day(s) old. fc00210a-5231-46f3-84f2-6e3c61c7ac0e Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:30:46 -0700 Deleting indexes more than 22 day(s) old. 29dbb5cc-f936-4f0e-8a41-26b2277c7083 Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:30:46 -0700 Deleting indexes more than 22 day(s) old.  Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:30:43 -0700 Deleting indexes more than 22 day(s) old. 4deb767e-abbd-43f0-8839-049760687e98 Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:30:43 -0700 Closing indexes 22 day(s) old. 4deb767e-abbd-43f0-8839-049760687e98 Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:30:15 -0700 Closing indexes 22 day(s) old. 29dbb5cc-f936-4f0e-8a41-26b2277c7083 Nagios Log Server  
System MAINTENANCE Mon, 28 Oct 2019 22:30:15 -0700 Closing indexes 22 day(s) old.  Nagios Log Server
Going on 4 hours and the snapshot still says it's running.

Re: How to stop a currently running snapshot?

Posted: Tue Oct 29, 2019 2:19 pm
by cdienger
It may be requiring a bit more time since some were missed previously. If you provide a profile we can check to see if the curator script is still running.

Re: How to stop a currently running snapshot?

Posted: Tue Oct 29, 2019 2:39 pm
by rferebee
cdienger wrote:It may be requiring a bit more time since some were missed previously. If you provide a profile we can check to see if the curator script is still running.
PM sent with current system profile. Thank you.

Re: How to stop a currently running snapshot?

Posted: Tue Oct 29, 2019 2:44 pm
by cdienger
Curator doesn't appear to be running. Tail the jobs.log again and force another maintenance job to see if there is a snapshot that may be hung and terminated.

Re: How to stop a currently running snapshot?

Posted: Tue Oct 29, 2019 2:54 pm
by rferebee
Here's what I got:

Code: Select all

root@nagioslscc2:/root> tail -f /usr/local/nagioslogserver/var/jobs.log
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command do_maintenance with args ' ' for job id: snapshots_maintenance
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh close indices --older-than 22 --time-unit days --timestring %Y.%m.%d
Return: 0
2019-10-29 12:52:02,586 INFO      Job starting: close indices
2019-10-29 12:52:02,613 INFO      Action close will be performed on the following indices: [u'logstash-2019.08.23']
2019-10-29 12:52:03,937 INFO      Provided indices successfully sealed. (Shown with --debug flag enabled.)
2019-10-29 12:52:04,140 INFO      Job completed successfully.
-----
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete indices --older-than 22 --time-unit days --timestring %Y.%m.%d
Return: 0
2019-10-29 12:52:04,308 INFO      Job starting: delete indices
2019-10-29 12:52:04,402 INFO      Pruning Kibana-related indices to prevent accidental deletion.
2019-10-29 12:52:04,402 INFO      Action delete will be performed on the following indices: [u'logstash-2019.08.23']
2019-10-29 12:52:04,403 INFO      Deleting indices as a batch operation:
2019-10-29 12:52:04,404 INFO      ---deleting index logstash-2019.08.23
2019-10-29 12:52:04,497 INFO      Job completed successfully.
-----
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "NLSREPCC" --ignore_unavailable indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Return: 1
2019-10-29 12:52:05,647 INFO      Job starting: snapshot indices
2019-10-29 12:52:05,647 WARNING   Overriding default connection timeout.  New timeout: 21600
2019-10-29 12:52:05,676 INFO      Action snapshot will be performed on the following indices: [u'logstash-2019.10.08', u'logstash-2019.10.09', u'logstash-2019.10.10', u'logstash-2019.10.11', u'logstash-2019.10.12', u'logstash-2019.10.13', u'logstash-2019.10.14', u'logstash-2019.10.15', u'logstash-2019.10.16', u'logstash-2019.10.17', u'logstash-2019.10.18', u'logstash-2019.10.19', u'logstash-2019.10.20', u'logstash-2019.10.21', u'logstash-2019.10.22', u'logstash-2019.10.23', u'logstash-2019.10.24', u'logstash-2019.10.25', u'logstash-2019.10.26', u'logstash-2019.10.27', u'logstash-2019.10.28']
2019-10-29 12:52:05,719 ERROR     Snapshot already in progress: [{u'stats': {u'number_of_files': 17165, u'total_size_in_bytes': 4852136037890, u'time_in_millis': 2819798, u'processed_files': 16284, u'processed_size_in_bytes': 4610107613032, u'start_time_in_millis': 1572369101668}, u'repository': u'NLSREPCC', u'shards_stats': {u'started': 3, u'failed': 5, u'done': 95, u'finalizing': 0, u'initializing': 2, u'total': 105}, u'state': u'STARTED', u'snapshot': u'curator-20191029053112', u'indices': {u'logstash-2019.10.08': {u'stats': {u'number_of_files': 497, u'total_size_in_bytes': 164779602960, u'time_in_millis': 26595132, u'processed_files': 497, u'processed_size_in_bytes': 164779602960, u'start_time_in_millis': 1572344264601}, u'shards': {u'1': {u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 54935453153, u'time_in_millis': 4081309, u'processed_files': 181, u'processed_size_in_bytes': 54935453153, u'start_time_in_millis': 1572344264601}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 1, u'total_size_in_bytes': 1609, u'time_in_millis': 853, u'processed_files': 1, u'processed_size_in_bytes': 1609, u'start_time_in_millis': 1572366745336}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 1, u'total_size_in_bytes': 2068, u'time_in_millis': 840, u'processed_files': 1, u'processed_size_in_bytes': 2068, u'start_time_in_millis': 1572359652742}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 151, u'total_size_in_bytes': 54916756936, u'time_in_millis': 4108905, u'processed_files': 151, u'processed_size_in_bytes': 54916756936, u'start_time_in_millis': 1572366750828}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 163, u'total_size_in_bytes': 54927389194, u'time_in_millis': 4107799, u'processed_files': 163, u'processed_size_in_bytes': 54927389194, u'start_time_in_millis': 1572362637342}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.09': {u'stats': {u'number_of_files': 504, u'total_size_in_bytes': 161754179126, u'time_in_millis': 15838891, u'processed_files': 504, u'processed_size_in_bytes': 161754179126, u'start_time_in_millis': 1572361794998}, u'shards': {u'1': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 54048475508, u'time_in_millis': 4256707, u'processed_files': 178, u'processed_size_in_bytes': 54048475508, u'start_time_in_millis': 1572364844833}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 142, u'total_size_in_bytes': 53856516527, u'time_in_millis': 4199456, u'processed_files': 142, u'processed_size_in_bytes': 53856516527, u'start_time_in_millis': 1572361794998}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 1, u'total_size_in_bytes': 2251, u'time_in_millis': 813, u'processed_files': 1, u'processed_size_in_bytes': 2251, u'start_time_in_millis': 1572377633076}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 1, u'total_size_in_bytes': 1733, u'time_in_millis': 1142, u'processed_files': 1, u'processed_size_in_bytes': 1733, u'start_time_in_millis': 1572377631769}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 182, u'total_size_in_bytes': 53849183107, u'time_in_millis': 4020042, u'processed_files': 182, u'processed_size_in_bytes': 53849183107, u'start_time_in_millis': 1572373191458}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.19': {u'stats': {u'number_of_files': 789, u'total_size_in_bytes': 219022792239, u'time_in_millis': 28026275, u'processed_files': 789, u'processed_size_in_bytes': 219022792239, u'start_time_in_millis': 1572343824085}, u'shards': {u'1': {u'stats': {u'number_of_files': 159, u'total_size_in_bytes': 43831674150, u'time_in_millis': 3215483, u'processed_files': 159, u'processed_size_in_bytes': 43831674150, u'start_time_in_millis': 1572349813519}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 162, u'total_size_in_bytes': 43787432086, u'time_in_millis': 3451721, u'processed_files': 162, u'processed_size_in_bytes': 43787432086, u'start_time_in_millis': 1572365564363}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 156, u'total_size_in_bytes': 43801320726, u'time_in_millis': 3193904, u'processed_files': 156, u'processed_size_in_bytes': 43801320726, u'start_time_in_millis': 1572343824085}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 162, u'total_size_in_bytes': 43853482463, u'time_in_millis': 2833728, u'processed_files': 162, u'processed_size_in_bytes': 43853482463, u'start_time_in_millis': 1572369016632}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 150, u'total_size_in_bytes': 43748882814, u'time_in_millis': 3190052, u'processed_files': 150, u'processed_size_in_bytes': 43748882814, u'start_time_in_millis': 1572346623282}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.18': {u'stats': {u'number_of_files': 818, u'total_size_in_bytes': 280765886240, u'time_in_millis': 19336935, u'processed_files': 818, u'processed_size_in_bytes': 280765886240, u'start_time_in_millis': 1572331050292}, u'shards': {u'1': {u'stats': {u'number_of_files': 163, u'total_size_in_bytes': 56227667368, u'time_in_millis': 4112182, u'processed_files': 163, u'processed_size_in_bytes': 56227667368, u'start_time_in_millis': 1572335123369}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 172, u'total_size_in_bytes': 56136510688, u'time_in_millis': 4184222, u'processed_files': 172, u'processed_size_in_bytes': 56136510688, u'start_time_in_millis': 1572346203005}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 160, u'total_size_in_bytes': 56105884207, u'time_in_millis': 4073025, u'processed_files': 160, u'processed_size_in_bytes': 56105884207, u'start_time_in_millis': 1572331050292}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 148, u'total_size_in_bytes': 56249798351, u'time_in_millis': 4159919, u'processed_files': 148, u'processed_size_in_bytes': 56249798351, u'start_time_in_millis': 1572335075808}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 175, u'total_size_in_bytes': 56046025626, u'time_in_millis': 4195238, u'processed_files': 175, u'processed_size_in_bytes': 56046025626, u'start_time_in_millis': 1572339235767}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.13': {u'stats': {u'number_of_files': 799, u'total_size_in_bytes': 208577990007, u'time_in_millis': 30082128, u'processed_files': 648, u'processed_size_in_bytes': 168877875444, u'start_time_in_millis': 1572347549510}, u'shards': {u'1': {u'stats': {u'number_of_files': 164, u'total_size_in_bytes': 41711368614, u'time_in_millis': 3115322, u'processed_files': 164, u'processed_size_in_bytes': 41711368614, u'start_time_in_millis': 1572347549510}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 161, u'total_size_in_bytes': 41713031545, u'time_in_millis': 3116389, u'processed_files': 161, u'processed_size_in_bytes': 41713031545, u'start_time_in_millis': 1572374515249}, u'stage': u'DONE'}, u'3': {u'reason': u'IndexShardSnapshotFailedException[[logstash-2019.10.13][3] Failed to perform snapshot (index files)]; nested: IOException[Input/output error]; ', u'stats': {u'number_of_files': 158, u'total_size_in_bytes': 41725113766, u'time_in_millis': 255276, u'processed_files': 7, u'processed_size_in_bytes': 2024999203, u'start_time_in_millis': 1572371134950}, u'stage': u'FAILURE'}, u'2': {u'stats': {u'number_of_files': 167, u'total_size_in_bytes': 41706369387, u'time_in_millis': 3116973, u'processed_files': 167, u'processed_size_in_bytes': 41706369387, u'start_time_in_millis': 1572350387320}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 149, u'total_size_in_bytes': 41722106695, u'time_in_millis': 3124839, u'processed_files': 149, u'processed_size_in_bytes': 41722106695, u'start_time_in_millis': 1572371390333}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 1, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.12': {u'stats': {u'number_of_files': 749, u'total_size_in_bytes': 214665145963, u'time_in_millis': 46700695, u'processed_files': 656, u'processed_size_in_bytes': 185048279825, u'start_time_in_millis': 1572330933264}, u'shards': {u'1': {u'stats': {u'number_of_files': 129, u'total_size_in_bytes': 42916466512, u'time_in_millis': 3200154, u'processed_files': 129, u'processed_size_in_bytes': 42916466512, u'start_time_in_millis': 1572358092510}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 147, u'total_size_in_bytes': 42894795954, u'time_in_millis': 3152232, u'processed_files': 147, u'processed_size_in_bytes': 42894795954, u'start_time_in_millis': 1572355360484}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 182, u'total_size_in_bytes': 43015783758, u'time_in_millis': 3192206, u'processed_files': 182, u'processed_size_in_bytes': 43015783758, u'start_time_in_millis': 1572334116596}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 150, u'total_size_in_bytes': 42920383705, u'time_in_millis': 3183277, u'processed_files': 150, u'processed_size_in_bytes': 42920383705, u'start_time_in_millis': 1572330933264}, u'stage': u'DONE'}, u'4': {u'node': u'pcqYieAcSAK7http3p3yzQ', u'stats': {u'number_of_files': 141, u'total_size_in_bytes': 42917716034, u'time_in_millis': 0, u'processed_files': 48, u'processed_size_in_bytes': 13300849896, u'start_time_in_millis': 1572377633959}, u'stage': u'STARTED'}}, u'shards_stats': {u'started': 1, u'failed': 0, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.11': {u'stats': {u'number_of_files': 887, u'total_size_in_bytes': 261491516465, u'time_in_millis': 31910673, u'processed_files': 769, u'processed_size_in_bytes': 218363307508, u'start_time_in_millis': 1572339235671}, u'shards': {u'1': {u'stats': {u'number_of_files': 184, u'total_size_in_bytes': 52372248459, u'time_in_millis': 3925962, u'processed_files': 184, u'processed_size_in_bytes': 52372248459, u'start_time_in_millis': 1572352392189}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 166, u'total_size_in_bytes': 52328962500, u'time_in_millis': 3836446, u'processed_files': 166, u'processed_size_in_bytes': 52328962500, u'start_time_in_millis': 1572339235671}, u'stage': u'DONE'}, u'3': {u'reason': u'IndexShardSnapshotFailedException[[logstash-2019.10.11][3] Failed to perform snapshot (index files)]; nested: FileSystemException[/nlsrepcc/indices/logstash-2019.10.11/3/__4c: Resource temporarily unavailable]; ', u'stats': {u'number_of_files': 187, u'total_size_in_bytes': 52254623914, u'time_in_millis': 2909600, u'processed_files': 157, u'processed_size_in_bytes': 35939162106, u'start_time_in_millis': 1572368223987}, u'stage': u'FAILURE'}, u'2': {u'stats': {u'number_of_files': 169, u'total_size_in_bytes': 52264966423, u'time_in_millis': 3832091, u'processed_files': 169, u'processed_size_in_bytes': 52264966423, u'start_time_in_millis': 1572341665937}, u'stage': u'DONE'}, u'4': {u'reason': u'IndexShardSnapshotFailedException[[logstash-2019.10.11][4] Failed to perform snapshot (index files)]; nested: IOException[Input/output error]; ', u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 52270715169, u'time_in_millis': 2050210, u'processed_files': 93, u'processed_size_in_bytes': 25457968020, u'start_time_in_millis': 1572369096134}, u'stage': u'FAILURE'}}, u'shards_stats': {u'started': 0, u'failed': 2, u'done': 3, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.10': {u'stats': {u'number_of_files': 872, u'total_size_in_bytes': 275976574379, u'time_in_millis': 31033802, u'processed_files': 801, u'processed_size_in_bytes': 251893660003, u'start_time_in_millis': 1572327058643}, u'shards': {u'1': {u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 55281974802, u'time_in_millis': 4041932, u'processed_files': 181, u'processed_size_in_bytes': 55281974802, u'start_time_in_millis': 1572327058643}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 172, u'total_size_in_bytes': 55114163661, u'time_in_millis': 4081469, u'processed_files': 172, u'processed_size_in_bytes': 55114163661, u'start_time_in_millis': 1572330138507}, u'stage': u'DONE'}, u'3': {u'reason': u'IndexShardSnapshotFailedException[[logstash-2019.10.10][3] Failed to perform snapshot (index files)]; nested: IOException[Input/output error]; ', u'stats': {u'number_of_files': 172, u'total_size_in_bytes': 55108893362, u'time_in_millis': 2408430, u'processed_files': 101, u'processed_size_in_bytes': 31025978986, u'start_time_in_millis': 1572353029167}, u'stage': u'FAILURE'}, u'2': {u'stats': {u'number_of_files': 169, u'total_size_in_bytes': 55237337185, u'time_in_millis': 4033960, u'processed_files': 169, u'processed_size_in_bytes': 55237337185, u'start_time_in_millis': 1572350015230}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 55234205369, u'time_in_millis': 4043166, u'processed_files': 178, u'processed_size_in_bytes': 55234205369, u'start_time_in_millis': 1572354049279}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 1, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.17': {u'stats': {u'number_of_files': 893, u'total_size_in_bytes': 279019754424, u'time_in_millis': 21160285, u'processed_files': 761, u'processed_size_in_bytes': 243499524871, u'start_time_in_millis': 1572357154324}, u'shards': {u'1': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 55809460912, u'time_in_millis': 4169636, u'processed_files': 178, u'processed_size_in_bytes': 55809460912, u'start_time_in_millis': 1572374144973}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 184, u'total_size_in_bytes': 55822917265, u'time_in_millis': 4401125, u'processed_files': 184, u'processed_size_in_bytes': 55822917265, u'start_time_in_millis': 1572365994769}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 166, u'total_size_in_bytes': 55781285152, u'time_in_millis': 4245499, u'processed_files': 166, u'processed_size_in_bytes': 55781285152, u'start_time_in_millis': 1572357549372}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 55775906733, u'time_in_millis': 4231159, u'processed_files': 181, u'processed_size_in_bytes': 55775906733, u'start_time_in_millis': 1572357154324}, u'stage': u'DONE'}, u'4': {u'node': u'pcqYieAcSAK7http3p3yzQ', u'stats': {u'number_of_files': 184, u'total_size_in_bytes': 55830184362, u'time_in_millis': 0, u'processed_files': 52, u'processed_size_in_bytes': 20309954809, u'start_time_in_millis': 1572377211590}, u'stage': u'STARTED'}}, u'shards_stats': {u'started': 1, u'failed': 0, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.16': {u'stats': {u'number_of_files': 979, u'total_size_in_bytes': 286245005801, u'time_in_millis': 30627354, u'processed_files': 979, u'processed_size_in_bytes': 286245005801, u'start_time_in_millis': 1572337596367}, u'shards': {u'1': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 57138927015, u'time_in_millis': 4288141, u'processed_files': 178, u'processed_size_in_bytes': 57138927015, u'start_time_in_millis': 1572363935580}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 186, u'total_size_in_bytes': 57372637481, u'time_in_millis': 4280832, u'processed_files': 186, u'processed_size_in_bytes': 57372637481, u'start_time_in_millis': 1572359653722}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 253, u'total_size_in_bytes': 57481970817, u'time_in_millis': 4314117, u'processed_files': 253, u'processed_size_in_bytes': 57481970817, u'start_time_in_millis': 1572341888785}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 175, u'total_size_in_bytes': 57154755843, u'time_in_millis': 4292173, u'processed_files': 175, u'processed_size_in_bytes': 57154755843, u'start_time_in_millis': 1572337596367}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 187, u'total_size_in_bytes': 57096714645, u'time_in_millis': 4214871, u'processed_files': 187, u'processed_size_in_bytes': 57096714645, u'start_time_in_millis': 1572355437688}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.15': {u'stats': {u'number_of_files': 1022, u'total_size_in_bytes': 269604822100, u'time_in_millis': 25671669, u'processed_files': 1022, u'processed_size_in_bytes': 269604822100, u'start_time_in_millis': 1572339892580}, u'shards': {u'1': {u'stats': {u'number_of_files': 154, u'total_size_in_bytes': 53800509094, u'time_in_millis': 3937401, u'processed_files': 154, u'processed_size_in_bytes': 53800509094, u'start_time_in_millis': 1572349472191}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 298, u'total_size_in_bytes': 54109857026, u'time_in_millis': 3973928, u'processed_files': 298, u'processed_size_in_bytes': 54109857026, u'start_time_in_millis': 1572345498155}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 259, u'total_size_in_bytes': 54074938405, u'time_in_millis': 4045734, u'processed_files': 259, u'processed_size_in_bytes': 54074938405, u'start_time_in_millis': 1572348346161}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 169, u'total_size_in_bytes': 53832418563, u'time_in_millis': 4178563, u'processed_files': 169, u'processed_size_in_bytes': 53832418563, u'start_time_in_millis': 1572361385686}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 142, u'total_size_in_bytes': 53787099012, u'time_in_millis': 3930893, u'processed_files': 142, u'processed_size_in_bytes': 53787099012, u'start_time_in_millis': 1572339892580}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.14': {u'stats': {u'number_of_files': 1013, u'total_size_in_bytes': 259968476287, u'time_in_millis': 11590897, u'processed_files': 1013, u'processed_size_in_bytes': 259968476287, u'start_time_in_millis': 1572327058536}, u'shards': {u'1': {u'stats': {u'number_of_files': 172, u'total_size_in_bytes': 51905007310, u'time_in_millis': 3795434, u'processed_files': 172, u'processed_size_in_bytes': 51905007310, u'start_time_in_millis': 1572334853999}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 240, u'total_size_in_bytes': 52086529312, u'time_in_millis': 3817387, u'processed_files': 240, u'processed_size_in_bytes': 52086529312, u'start_time_in_millis': 1572331100692}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 242, u'total_size_in_bytes': 52118148172, u'time_in_millis': 3888976, u'processed_files': 242, u'processed_size_in_bytes': 52118148172, u'start_time_in_millis': 1572327076106}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 51919465710, u'time_in_millis': 3857174, u'processed_files': 178, u'processed_size_in_bytes': 51919465710, u'start_time_in_millis': 1572327076027}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 51939325783, u'time_in_millis': 3808868, u'processed_files': 181, u'processed_size_in_bytes': 51939325783, u'start_time_in_millis': 1572327058536}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.22': {u'stats': {u'number_of_files': 1404, u'total_size_in_bytes': 271721502415, u'time_in_millis': 28301848, u'processed_files': 1229, u'processed_size_in_bytes': 242613387832, u'start_time_in_millis': 1572327058544}, u'shards': {u'1': {u'stats': {u'number_of_files': 206, u'total_size_in_bytes': 54291038837, u'time_in_millis': 3991670, u'processed_files': 206, u'processed_size_in_bytes': 54291038837, u'start_time_in_millis': 1572327058544}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 148, u'total_size_in_bytes': 54164523999, u'time_in_millis': 3964231, u'processed_files': 148, u'processed_size_in_bytes': 54164523999, u'start_time_in_millis': 1572330108758}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 437, u'total_size_in_bytes': 54511225627, u'time_in_millis': 4047507, u'processed_files': 437, u'processed_size_in_bytes': 54511225627, u'start_time_in_millis': 1572340112663}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 206, u'total_size_in_bytes': 54267759862, u'time_in_millis': 3986355, u'processed_files': 206, u'processed_size_in_bytes': 54267759862, u'start_time_in_millis': 1572330867476}, u'stage': u'DONE'}, u'4': {u'reason': u'IndexShardSnapshotFailedException[[logstash-2019.10.22][4] Failed to perform snapshot (index files)]; nested: IOException[Input/output error]; ', u'stats': {u'number_of_files': 407, u'total_size_in_bytes': 54486954090, u'time_in_millis': 1950707, u'processed_files': 232, u'processed_size_in_bytes': 25378839507, u'start_time_in_millis': 1572353409685}, u'stage': u'FAILURE'}}, u'shards_stats': {u'started': 0, u'failed': 1, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.23': {u'stats': {u'number_of_files': 790, u'total_size_in_bytes': 242767025560, u'time_in_millis': 30076610, u'processed_files': 790, u'processed_size_in_bytes': 242767025560, u'start_time_in_millis': 1572327077571}, u'shards': {u'1': {u'stats': {u'number_of_files': 154, u'total_size_in_bytes': 48565747318, u'time_in_millis': 3615300, u'processed_files': 154, u'processed_size_in_bytes': 48565747318, u'start_time_in_millis': 1572350664941}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 151, u'total_size_in_bytes': 48579930068, u'time_in_millis': 3575139, u'processed_files': 151, u'processed_size_in_bytes': 48579930068, u'start_time_in_millis': 1572327077571}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 154, u'total_size_in_bytes': 48513651444, u'time_in_millis': 3550844, u'processed_files': 154, u'processed_size_in_bytes': 48513651444, u'start_time_in_millis': 1572343072389}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 163, u'total_size_in_bytes': 48569545646, u'time_in_millis': 3649705, u'processed_files': 163, u'processed_size_in_bytes': 48569545646, u'start_time_in_millis': 1572353504476}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 168, u'total_size_in_bytes': 48538151084, u'time_in_millis': 3530644, u'processed_files': 168, u'processed_size_in_bytes': 48538151084, u'start_time_in_millis': 1572340679668}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.20': {u'stats': {u'number_of_files': 492, u'total_size_in_bytes': 135893649462, u'time_in_millis': 2819798, u'processed_files': 351, u'processed_size_in_bytes': 95021672774, u'start_time_in_millis': 1572369101668}, u'shards': {u'1': {u'node': u'pcqYieAcSAK7http3p3yzQ', u'stats': {u'number_of_files': 165, u'total_size_in_bytes': 45334605671, u'time_in_millis': 0, u'processed_files': 24, u'processed_size_in_bytes': 4462628983, u'start_time_in_millis': 1572378314712}, u'stage': u'STARTED'}, u'0': {u'stats': {u'number_of_files': 171, u'total_size_in_bytes': 45300601775, u'time_in_millis': 3376276, u'processed_files': 171, u'processed_size_in_bytes': 45300601775, u'start_time_in_millis': 1572334220018}, u'stage': u'DONE'}, u'3': {u'node': u'pcqYieAcSAK7http3p3yzQ', u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'INIT'}, u'2': {u'node': u'pcqYieAcSAK7http3p3yzQ', u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'INIT'}, u'4': {u'stats': {u'number_of_files': 156, u'total_size_in_bytes': 45258442016, u'time_in_millis': 2819798, u'processed_files': 156, u'processed_size_in_bytes': 45258442016, u'start_time_in_millis': 1572369101668}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 1, u'failed': 0, u'done': 2, u'finalizing': 0, u'initializing': 2, u'total': 5}}, u'logstash-2019.10.21': {u'stats': {u'number_of_files': 672, u'total_size_in_bytes': 156990220334, u'time_in_millis': 35273536, u'processed_files': 672, u'processed_size_in_bytes': 156990220334, u'start_time_in_millis': 1572337917815}, u'shards': {u'1': {u'stats': {u'number_of_files': 102, u'total_size_in_bytes': 26894151137, u'time_in_millis': 2023995, u'processed_files': 102, u'processed_size_in_bytes': 26894151137, u'start_time_in_millis': 1572356318259}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 144, u'total_size_in_bytes': 26992848718, u'time_in_millis': 2044866, u'processed_files': 144, u'processed_size_in_bytes': 26992848718, u'start_time_in_millis': 1572371146485}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 146, u'total_size_in_bytes': 26971916112, u'time_in_millis': 1977300, u'processed_files': 146, u'processed_size_in_bytes': 26971916112, u'start_time_in_millis': 1572342158579}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 114, u'total_size_in_bytes': 26881708675, u'time_in_millis': 1974642, u'processed_files': 114, u'processed_size_in_bytes': 26881708675, u'start_time_in_millis': 1572337917815}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 166, u'total_size_in_bytes': 49249595692, u'time_in_millis': 1339826, u'processed_files': 166, u'processed_size_in_bytes': 49249595692, u'start_time_in_millis': 1572344135943}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.26': {u'stats': {u'number_of_files': 766, u'total_size_in_bytes': 204978273916, u'time_in_millis': 17188424, u'processed_files': 766, u'processed_size_in_bytes': 204978273916, u'start_time_in_millis': 1572327076041}, u'shards': {u'1': {u'stats': {u'number_of_files': 155, u'total_size_in_bytes': 40977660545, u'time_in_millis': 3048689, u'processed_files': 155, u'processed_size_in_bytes': 40977660545, u'start_time_in_millis': 1572341215776}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 140, u'total_size_in_bytes': 41003683675, u'time_in_millis': 3011502, u'processed_files': 140, u'processed_size_in_bytes': 41003683675, u'start_time_in_millis': 1572334073027}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 167, u'total_size_in_bytes': 41020972013, u'time_in_millis': 3032103, u'processed_files': 167, u'processed_size_in_bytes': 41020972013, u'start_time_in_millis': 1572327076605}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 158, u'total_size_in_bytes': 40983311677, u'time_in_millis': 2999481, u'processed_files': 158, u'processed_size_in_bytes': 40983311677, u'start_time_in_millis': 1572334918136}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 146, u'total_size_in_bytes': 40992646006, u'time_in_millis': 3062429, u'processed_files': 146, u'processed_size_in_bytes': 40992646006, u'start_time_in_millis': 1572327076041}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.27': {u'stats': {u'number_of_files': 778, u'total_size_in_bytes': 205031912054, u'time_in_millis': 27638650, u'processed_files': 778, u'processed_size_in_bytes': 205031912054, u'start_time_in_millis': 1572333839458}, u'shards': {u'1': {u'stats': {u'number_of_files': 161, u'total_size_in_bytes': 41011213040, u'time_in_millis': 2997016, u'processed_files': 161, u'processed_size_in_bytes': 41011213040, u'start_time_in_millis': 1572347018058}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 161, u'total_size_in_bytes': 40992839647, u'time_in_millis': 3005369, u'processed_files': 161, u'processed_size_in_bytes': 40992839647, u'start_time_in_millis': 1572333839458}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 149, u'total_size_in_bytes': 41006452803, u'time_in_millis': 3027749, u'processed_files': 149, u'processed_size_in_bytes': 41006452803, u'start_time_in_millis': 1572337084573}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 161, u'total_size_in_bytes': 41014987754, u'time_in_millis': 3135799, u'processed_files': 161, u'processed_size_in_bytes': 41014987754, u'start_time_in_millis': 1572358342309}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 146, u'total_size_in_bytes': 41006418810, u'time_in_millis': 3016297, u'processed_files': 146, u'processed_size_in_bytes': 41006418810, u'start_time_in_millis': 1572338649480}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.24': {u'stats': {u'number_of_files': 863, u'total_size_in_bytes': 276544830501, u'time_in_millis': 35560510, u'processed_files': 863, u'processed_size_in_bytes': 276544830501, u'start_time_in_millis': 1572327076717}, u'shards': {u'1': {u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 55218804771, u'time_in_millis': 4052499, u'processed_files': 181, u'processed_size_in_bytes': 55218804771, u'start_time_in_millis': 1572327076717}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 157, u'total_size_in_bytes': 55347330117, u'time_in_millis': 4118011, u'processed_files': 157, u'processed_size_in_bytes': 55347330117, u'start_time_in_millis': 1572343431401}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 169, u'total_size_in_bytes': 55459692927, u'time_in_millis': 4110405, u'processed_files': 169, u'processed_size_in_bytes': 55459692927, u'start_time_in_millis': 1572330965329}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 55256455898, u'time_in_millis': 4124274, u'processed_files': 178, u'processed_size_in_bytes': 55256455898, u'start_time_in_millis': 1572358512953}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 178, u'total_size_in_bytes': 55262546788, u'time_in_millis': 4053202, u'processed_files': 178, u'processed_size_in_bytes': 55262546788, u'start_time_in_millis': 1572331129336}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.25': {u'stats': {u'number_of_files': 734, u'total_size_in_bytes': 216167937453, u'time_in_millis': 43491614, u'processed_files': 734, u'processed_size_in_bytes': 216167937453, u'start_time_in_millis': 1572330652820}, u'shards': {u'1': {u'stats': {u'number_of_files': 135, u'total_size_in_bytes': 43203877507, u'time_in_millis': 3366481, u'processed_files': 135, u'processed_size_in_bytes': 43203877507, u'start_time_in_millis': 1572361478176}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 135, u'total_size_in_bytes': 43195186864, u'time_in_millis': 3166900, u'processed_files': 135, u'processed_size_in_bytes': 43195186864, u'start_time_in_millis': 1572335182588}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 155, u'total_size_in_bytes': 43259855203, u'time_in_millis': 3186584, u'processed_files': 155, u'processed_size_in_bytes': 43259855203, u'start_time_in_millis': 1572330652820}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 150, u'total_size_in_bytes': 43223040996, u'time_in_millis': 3284614, u'processed_files': 150, u'processed_size_in_bytes': 43223040996, u'start_time_in_millis': 1572370859820}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 159, u'total_size_in_bytes': 43285976883, u'time_in_millis': 3268923, u'processed_files': 159, u'processed_size_in_bytes': 43285976883, u'start_time_in_millis': 1572354280394}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.28': {u'stats': {u'number_of_files': 844, u'total_size_in_bytes': 260168940204, u'time_in_millis': 32251135, u'processed_files': 844, u'processed_size_in_bytes': 260168940204, u'start_time_in_millis': 1572336844876}, u'shards': {u'1': {u'stats': {u'number_of_files': 181, u'total_size_in_bytes': 52048167477, u'time_in_millis': 3894403, u'processed_files': 181, u'processed_size_in_bytes': 52048167477, u'start_time_in_millis': 1572361292986}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 142, u'total_size_in_bytes': 51977122819, u'time_in_millis': 3808898, u'processed_files': 142, u'processed_size_in_bytes': 51977122819, u'start_time_in_millis': 1572338349564}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 175, u'total_size_in_bytes': 52047038243, u'time_in_millis': 3834506, u'processed_files': 175, u'processed_size_in_bytes': 52047038243, u'start_time_in_millis': 1572336844876}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 172, u'total_size_in_bytes': 52042546090, u'time_in_millis': 3906881, u'processed_files': 172, u'processed_size_in_bytes': 52042546090, u'start_time_in_millis': 1572337308851}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 174, u'total_size_in_bytes': 52054065575, u'time_in_millis': 3908274, u'processed_files': 174, u'processed_size_in_bytes': 52054065575, u'start_time_in_millis': 1572365187737}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}}}]
2019-10-29 12:52:05,720 WARNING   Job did not complete successfully.
-----
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete snapshots --older-than 731 --time-unit days --timestring %Y%m%d --repository "NLSREPCC"
Return: 0
2019-10-29 12:52:05,902 INFO      Job starting: delete snapshots
2019-10-29 12:52:06,388 WARNING   No snapshots matched provided args.
No snapshots matched provided args.
-----
SUCCESS
Processed 0 node jobs.
Processed 1 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated

Re: How to stop a currently running snapshot?

Posted: Tue Oct 29, 2019 3:26 pm
by cdienger
Disable the maintenance job and we'll try to get this caught up manually.

Disable by setting the next run time to somewhere in the distant future and then run this from the command line:

Code: Select all

curl -XDELETE 'http://localhost:9200/_snapshot/NLSREPCC/curator-curator-20191029053112'

/usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "NLSREPCC" --ignore_unavailable indices --older-than 15 --time-unit days --timestring %Y.%m.%d
/usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "NLSREPCC" --ignore_unavailable indices --older-than 10 --time-unit days --timestring %Y.%m.%d
/usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "NLSREPCC" --ignore_unavailable indices --older-than 5 --time-unit days --timestring %Y.%m.%d
/usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "NLSREPCC" --ignore_unavailable indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Each run of the curator script may take a long time to finish, but it should be a bit more successful with this method. Once it is caught up you can set the maintenance job back to run everyday.
All times are UTC-05:00
Page 4 of 6

Powered by phpBB® Forum Software © phpBB Limited