How to stop a currently running snapshot?

[scottwilkerson]

I think I may have some corrupt shards. After letting it go overnight without running a snapshot there are still 10 unassigned shards with 0 relocating and 0 initializing.

This is possible.

Also, the two snapshots I told you about that were still "IN_PROGRESS" are still running.

This would make sense if it was in the process of trying to backup a index that did not have a valid shard.

I would recommend doing a restart of elasticsearch on each node, this should clear the IN_PROGRESS

If you do not get back you green, status you can run the following to see which shards are not in a started state

Code: Select all

curl localhost:9200/_cat/shards|grep -v STARTED

From there you may be able to delete and restore those indexes from previous backups

[rferebee]

Do I only need to restart elasticsearch or should I restart logstash as well?

[rferebee]

Ok, I restarted elasticsearch on my three nodes. The two snapshots are still showing IN_PROGRESS and my cluster health is still showing RED.

[scottwilkerson]

Do I only need to restart elasticsearch or should I restart logstash as well?

no just ES

What is the output of this?

Code: Select all

curl localhost:9200/_cat/shards|grep -v STARTED

[rferebee]

Code: Select all

root@nagioslscc2:/root> curl localhost:9200/_cat/shards|grep -v STARTED
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30414  100 30414    0     0  32520      0 --:--:-- --:--:-- --:--:-- 32493
logstash-2019.10.21     0 r UNASSIGNED
logstash-2019.10.21     3 r INITIALIZING                  10.x.207.114 29dbb5cc-f936-4f0e-8a41-26b2277c7083
logstash-2019.10.21     4 r UNASSIGNED
logstash-2019.10.22     0 r INITIALIZING                  10.x.207.114 29dbb5cc-f936-4f0e-8a41-26b2277c7083
logstash-2019.10.03     1 r UNASSIGNED
logstash-2019.10.23     0 r UNASSIGNED
logstash-2019.10.23     3 r INITIALIZING                  10.x.207.112 fc00210a-5231-46f3-84f2-6e3c61c7ac0e
logstash-2019.10.23     1 r INITIALIZING                  10.x.207.111 4deb767e-abbd-43f0-8839-049760687e98
logstash-2019.10.23     4 r INITIALIZING                  10.x.207.111 4deb767e-abbd-43f0-8839-049760687e98
logstash-2019.10.02     2 r UNASSIGNED
logstash-2019.10.02     3 r UNASSIGNED
logstash-2019.10.07     3 r UNASSIGNED
logstash-2019.10.07     1 r UNASSIGNED
logstash-2019.10.08     2 r UNASSIGNED
logstash-2019.10.08     0 r UNASSIGNED
logstash-2019.10.09     4 r UNASSIGNED
logstash-2019.10.09     2 r UNASSIGNED
logstash-2019.10.10     2 r UNASSIGNED
logstash-2019.10.11     4 r UNASSIGNED
logstash-2019.10.14     0 r UNASSIGNED
logstash-2019.10.15     0 r UNASSIGNED
logstash-2019.10.12     0 r UNASSIGNED
logstash-2019.10.13     0 r UNASSIGNED
logstash-2019.10.13     3 r UNASSIGNED
logstash-2019.08.16     4 p UNASSIGNED
logstash-2019.08.16     4 r UNASSIGNED
logstash-2019.08.16     0 p UNASSIGNED
logstash-2019.08.16     0 r UNASSIGNED
logstash-2019.08.16     3 p UNASSIGNED
logstash-2019.08.16     3 r UNASSIGNED
logstash-2019.08.16     1 p UNASSIGNED
logstash-2019.08.16     1 r UNASSIGNED
logstash-2019.08.16     2 p UNASSIGNED
logstash-2019.08.16     2 r UNASSIGNED
logstash-2019.10.19     4 r UNASSIGNED
logstash-2019.10.16     0 r UNASSIGNED

[rferebee]

I think the issue is with the shards from 8/16. There are exactly 10 of them and that's how many shards were left this morning.

[cdienger]

The 8/16 index does appear to be the problem See if it will let you close it with:

Code: Select all

curl -XPOST "localhost:9200/logstash-2019.08.16/_close"

If it does then that should turn the status from red to yellow(and maybe eventually green if it is then able to assign the remaining redundant shards.

[rferebee]

Yep, that did the trick! Thank you, I think we're back in business.

[rferebee]

The snapshots described in my request are still showing IN_PROGRESS as of this morning and the snapshot that was supposed to run last night did not. I suspect because of the system thinking there are already two snapshots running.

I have restarted elasticsearch twice in the last two days and it's not causing the snapshots to finish. I need another way to stop them, please.

[rferebee]

This is what I'm seeing.