How to stop a currently running snapshot?

[rferebee]

Everything is green. Has been for a few days.

[cdienger]

If all is green now then just continue to monitor and let us know if anything pops up again tomorrow.

[rferebee]

I received similar errors during last night's snapshot. See below:

Code: Select all

{
    "snapshot" : "curator-20191101053030",
    "version_id" : 1070699,
    "version" : "1.7.6",
    "indices" : [ "logstash-2019.10.11", "logstash-2019.10.12", "logstash-2019.10.13", "logstash-2019.10.14", "logstash-2019.10.15", "logstash-2019.10.16", "logstash-2019.10.17", "logstash-2019.10.18", "logstash-2019.10.19", "logstash-2019.10.20", "logstash-2019.10.21", "logstash-2019.10.22", "logstash-2019.10.23", "logstash-2019.10.24", "logstash-2019.10.25", "logstash-2019.10.26", "logstash-2019.10.27", "logstash-2019.10.28", "logstash-2019.10.29", "logstash-2019.10.30", "logstash-2019.10.31" ],
    "state" : "PARTIAL",
    "start_time" : "2019-11-01T05:30:31.593Z",
    "start_time_in_millis" : 1572586231593,
    "end_time" : "2019-11-01T06:58:41.724Z",
    "end_time_in_millis" : 1572591521724,
    "duration_in_millis" : 5290131,
    "failures" : [ {
      "node_id" : "pcqYieAcSAK7http3p3yzQ",
      "index" : "logstash-2019.10.25",
      "reason" : "IndexShardSnapshotFailedException[[logstash-2019.10.25][2] Failed to perform snapshot (index files)]; nested: FileNotFoundException[/nlsrepcc/indices/logstash-2019.10.25/2/__10 (Stale file handle)]; ",
      "shard_id" : 2,
      "status" : "INTERNAL_SERVER_ERROR"
    }, {
      "node_id" : "pcqYieAcSAK7http3p3yzQ",
      "index" : "logstash-2019.10.21",
      "reason" : "IndexShardSnapshotFailedException[[logstash-2019.10.21][0] Failed to perform snapshot (index files)]; nested: FileNotFoundException[/nlsrepcc/indices/logstash-2019.10.21/0/__6 (Stale file handle)]; ",
      "shard_id" : 0,
      "status" : "INTERNAL_SERVER_ERROR"
    } ],
    "shards" : {
      "total" : 105,
      "failed" : 2,
      "successful" : 103
    }
  } ]
}

[cdienger]

Are the files available now:

Code: Select all

ls -lh /nlsrepcc/indices/logstash-2019.10.21/0/__6
ls -lh /nlsrepcc/indices/logstash-2019.10.25/2/__10

?

The error usually means there was problem accessing the files which could be due to minor network issues(given that's on a remote share).

[rferebee]

Yes, they appear to be:

Code: Select all

root@nagioslscc2:/root> ls -lh /nlsrepcc/indices/logstash-2019.10.21/0/__6
-rwxrwx--- 1 nagios nagios 1.9G Oct 29 10:48 /nlsrepcc/indices/logstash-2019.10.21/0/__6
root@nagioslscc2:/root> ls -lh /nlsrepcc/indices/logstash-2019.10.25/2/__10
-rwxrwx--- 1 nagios nagios 2.0G Oct 29 10:49 /nlsrepcc/indices/logstash-2019.10.25/2/__10

[cdienger]

These should be updated on the next run. I think we should be okay to just monitor it and follow up next week.

[rferebee]

Sounds good, thank you for your assistance.

[cdienger]

How are things looking today?

[rferebee]

All 3 snapshots from the weekend ran without issue. We can mark this resolved and lock the thread. Thank you.

[cdienger]

Glad to hear!