Page 2 of 6
Re: How to stop a currently running snapshot?
Posted: Wed Oct 23, 2019 9:31 am
by scottwilkerson
rferebee wrote:I think I may have some corrupt shards. After letting it go overnight without running a snapshot there are still 10 unassigned shards with 0 relocating and 0 initializing.
This is possible.
rferebee wrote:Also, the two snapshots I told you about that were still "IN_PROGRESS" are still running.
This would make sense if it was in the process of trying to backup a index that did not have a valid shard.
I would recommend doing a restart of elasticsearch on each node, this should clear the IN_PROGRESS
If you do not get back you green, status you can run the following to see which shards are not in a started state
Code: Select all
curl localhost:9200/_cat/shards|grep -v STARTED
From there you may be able to delete and restore those indexes from previous backups
Re: How to stop a currently running snapshot?
Posted: Wed Oct 23, 2019 9:43 am
by rferebee
Do I only need to restart elasticsearch or should I restart logstash as well?
Re: How to stop a currently running snapshot?
Posted: Wed Oct 23, 2019 10:15 am
by rferebee
Ok, I restarted elasticsearch on my three nodes. The two snapshots are still showing IN_PROGRESS and my cluster health is still showing RED.
Re: How to stop a currently running snapshot?
Posted: Wed Oct 23, 2019 11:07 am
by scottwilkerson
rferebee wrote:Do I only need to restart elasticsearch or should I restart logstash as well?
no just ES
What is the output of this?
Code: Select all
curl localhost:9200/_cat/shards|grep -v STARTED
Re: How to stop a currently running snapshot?
Posted: Wed Oct 23, 2019 11:14 am
by rferebee
Code: Select all
root@nagioslscc2:/root> curl localhost:9200/_cat/shards|grep -v STARTED
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 30414 100 30414 0 0 32520 0 --:--:-- --:--:-- --:--:-- 32493
logstash-2019.10.21 0 r UNASSIGNED
logstash-2019.10.21 3 r INITIALIZING 10.x.207.114 29dbb5cc-f936-4f0e-8a41-26b2277c7083
logstash-2019.10.21 4 r UNASSIGNED
logstash-2019.10.22 0 r INITIALIZING 10.x.207.114 29dbb5cc-f936-4f0e-8a41-26b2277c7083
logstash-2019.10.03 1 r UNASSIGNED
logstash-2019.10.23 0 r UNASSIGNED
logstash-2019.10.23 3 r INITIALIZING 10.x.207.112 fc00210a-5231-46f3-84f2-6e3c61c7ac0e
logstash-2019.10.23 1 r INITIALIZING 10.x.207.111 4deb767e-abbd-43f0-8839-049760687e98
logstash-2019.10.23 4 r INITIALIZING 10.x.207.111 4deb767e-abbd-43f0-8839-049760687e98
logstash-2019.10.02 2 r UNASSIGNED
logstash-2019.10.02 3 r UNASSIGNED
logstash-2019.10.07 3 r UNASSIGNED
logstash-2019.10.07 1 r UNASSIGNED
logstash-2019.10.08 2 r UNASSIGNED
logstash-2019.10.08 0 r UNASSIGNED
logstash-2019.10.09 4 r UNASSIGNED
logstash-2019.10.09 2 r UNASSIGNED
logstash-2019.10.10 2 r UNASSIGNED
logstash-2019.10.11 4 r UNASSIGNED
logstash-2019.10.14 0 r UNASSIGNED
logstash-2019.10.15 0 r UNASSIGNED
logstash-2019.10.12 0 r UNASSIGNED
logstash-2019.10.13 0 r UNASSIGNED
logstash-2019.10.13 3 r UNASSIGNED
logstash-2019.08.16 4 p UNASSIGNED
logstash-2019.08.16 4 r UNASSIGNED
logstash-2019.08.16 0 p UNASSIGNED
logstash-2019.08.16 0 r UNASSIGNED
logstash-2019.08.16 3 p UNASSIGNED
logstash-2019.08.16 3 r UNASSIGNED
logstash-2019.08.16 1 p UNASSIGNED
logstash-2019.08.16 1 r UNASSIGNED
logstash-2019.08.16 2 p UNASSIGNED
logstash-2019.08.16 2 r UNASSIGNED
logstash-2019.10.19 4 r UNASSIGNED
logstash-2019.10.16 0 r UNASSIGNED
Re: How to stop a currently running snapshot?
Posted: Wed Oct 23, 2019 11:32 am
by rferebee
I think the issue is with the shards from 8/16. There are exactly 10 of them and that's how many shards were left this morning.
Re: How to stop a currently running snapshot?
Posted: Wed Oct 23, 2019 1:03 pm
by cdienger
The 8/16 index does appear to be the problem See if it will let you close it with:
Code: Select all
curl -XPOST "localhost:9200/logstash-2019.08.16/_close"
If it does then that should turn the status from red to yellow(and maybe eventually green if it is then able to assign the remaining redundant shards.
Re: How to stop a currently running snapshot?
Posted: Wed Oct 23, 2019 1:38 pm
by rferebee
Yep, that did the trick! Thank you, I think we're back in business.
Re: How to stop a currently running snapshot?
Posted: Thu Oct 24, 2019 10:37 am
by rferebee
The snapshots described in my request are still showing IN_PROGRESS as of this morning and the snapshot that was supposed to run last night did not. I suspect because of the system thinking there are already two snapshots running.
I have restarted elasticsearch twice in the last two days and it's not causing the snapshots to finish. I need another way to stop them, please.
Re: How to stop a currently running snapshot?
Posted: Thu Oct 24, 2019 10:39 am
by rferebee
This is what I'm seeing.