Page 1 of 2
Indices re-opened?
Posted: Tue Dec 29, 2015 5:43 pm
by CFT6Server
After the holidays, I found our cluster status to be red. I am still not sure what happened and when this started, but looks like it's been a while and the shards are happy. I clicked on the index status page and looks like there's a lot of older indexes that are opened ?! I am not sure why that is....
What's the best way to tackle this? and to get our cluster back to green....
Re: Indices re-opened?
Posted: Tue Dec 29, 2015 5:47 pm
by CFT6Server
Looks like it re-opened all the previous index? As it shows that all index status as opened for some reason....
Lots of shards and pending tasks...
Code: Select all
# curl localhost:9200/_cluster/health?pretty
{
"cluster_name" : "80e9022e-f73f-429e-8927-f23d0d88dfd2",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 6,
"number_of_data_nodes" : 6,
"active_primary_shards" : 1501,
"active_shards" : 2869,
"relocating_shards" : 0,
"initializing_shards" : 12,
"unassigned_shards" : 121,
"number_of_pending_tasks" : 7013,
"number_of_in_flight_fetch" : 0
Re: Indices re-opened?
Posted: Tue Dec 29, 2015 7:36 pm
by CFT6Server
Update: After closing all previously closed, status went back to yellow and is now reallocating shards.
I wonder why it will open old shards, which are supposed to be removed and archived as well.
Re: Indices re-opened?
Posted: Wed Dec 30, 2015 11:44 am
by jolson
It'll be important to capture the elasticsearch log - it should give a reason why your indices were closed.
Code: Select all
tar czf elasticlog.tar.gz /var/log/elasticsearch/*
Are your indices still balancing? Was there any recent power outage or similar that may have triggered such a situation?
Re: Indices re-opened?
Posted: Wed Dec 30, 2015 3:22 pm
by CFT6Server
I'll capture the logs... looks like once it returned to green status, the closed indexes are "opened" again.... seems like backup/maintenance didn't run. I'll try to reset and see if that helps.
Re: Indices re-opened?
Posted: Wed Dec 30, 2015 4:03 pm
by jolson
Let me know if that helps - I'll keep an eye out for the logs once you collect them. Thanks!
Re: Indices re-opened?
Posted: Mon Jan 04, 2016 5:06 pm
by CFT6Server
So I think there must be some recovery information that's stuck? I rebooted a single node today and it is re-opening my indices again. I am not sure why this is happening.
Re: Indices re-opened?
Posted: Mon Jan 04, 2016 5:09 pm
by jolson
I think we should have a remote session - if you could mail into
[email protected] I would be happy to pick up the ticket and get the remote going as soon as possible.
I dealt with a very similar case today, and it's possible that your problem is similar.
Re: Indices re-opened?
Posted: Fri Jan 08, 2016 3:54 pm
by CFT6Server
Hi jolson, reporting back, the indexes did get reopened again, which means the configuration change didn't prevent if from re-reading everything. But after allowing it to run through it and re-delete, it seems to have settled.
Re: Indices re-opened?
Posted: Fri Jan 08, 2016 3:59 pm
by jolson
I'm happy to hear it - keep me updated if anything does wind up looking out of the ordinary.
