Indices re-opened?

[CFT6Server]

After the holidays, I found our cluster status to be red. I am still not sure what happened and when this started, but looks like it's been a while and the shards are happy. I clicked on the index status page and looks like there's a lot of older indexes that are opened ?! I am not sure why that is....

What's the best way to tackle this? and to get our cluster back to green....

[CFT6Server]

Looks like it re-opened all the previous index? As it shows that all index status as opened for some reason....

Lots of shards and pending tasks...

Code: Select all

# curl localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "80e9022e-f73f-429e-8927-f23d0d88dfd2",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 6,
  "number_of_data_nodes" : 6,
  "active_primary_shards" : 1501,
  "active_shards" : 2869,
  "relocating_shards" : 0,
  "initializing_shards" : 12,
  "unassigned_shards" : 121,
  "number_of_pending_tasks" : 7013,
  "number_of_in_flight_fetch" : 0

[CFT6Server]

Update: After closing all previously closed, status went back to yellow and is now reallocating shards.

I wonder why it will open old shards, which are supposed to be removed and archived as well.

[jolson]

It'll be important to capture the elasticsearch log - it should give a reason why your indices were closed.

Code: Select all

tar czf elasticlog.tar.gz /var/log/elasticsearch/*

Are your indices still balancing? Was there any recent power outage or similar that may have triggered such a situation?

[CFT6Server]

I'll capture the logs... looks like once it returned to green status, the closed indexes are "opened" again.... seems like backup/maintenance didn't run. I'll try to reset and see if that helps.

[jolson]

Let me know if that helps - I'll keep an eye out for the logs once you collect them. Thanks!

[CFT6Server]

So I think there must be some recovery information that's stuck? I rebooted a single node today and it is re-opening my indices again. I am not sure why this is happening.

[jolson]

I think we should have a remote session - if you could mail into [email protected] I would be happy to pick up the ticket and get the remote going as soon as possible.

I dealt with a very similar case today, and it's possible that your problem is similar.

[CFT6Server]

Hi jolson, reporting back, the indexes did get reopened again, which means the configuration change didn't prevent if from re-reading everything. But after allowing it to run through it and re-delete, it seems to have settled.

[jolson]

I'm happy to hear it - keep me updated if anything does wind up looking out of the ordinary.