Page 1 of 1
upgrade to 2.2
Posted: Fri Sep 04, 2015 12:52 pm
by jvestrum
Hello, I'm upgrading our two-node NLS cluster from R1.4 to R2.2. I ran the upgrade script on the first node, which seems to have gone well, but now the cluster state is Yellow and all the shards are Unassigned. The upgrade doc says to wait for the cluster state to go green before moving on to the next node, but nothing seems to be happening. Advice?
Re: upgrade to 2.2
Posted: Fri Sep 04, 2015 12:54 pm
by jvestrum
FYI:
[root@gtcs-nls01 nagioslogserver]# es cluster-health
{
"cluster_name" : "5e48d284-3c27-4184-b703-49e6a9830018",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 194,
"active_shards" : 194,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 194,
"number_of_pending_tasks" : -1,
"number_of_in_flight_fetch" : 0
}
Re: upgrade to 2.2
Posted: Fri Sep 04, 2015 1:02 pm
by jvestrum
I think this is what's going on:
https://www.elastic.co/guide/en/elastic ... g-upgrades
If it is not possible to assign the replica shards to another node with the higher version — e.g. if there is only one node with the higher version in the cluster — then the replica shards will remain unassigned, i.e. the cluster health will be status yellow. As soon as another node with the higher version joins the cluster, the replicas should be assigned and the cluster health will reach status green.
So I think I can just go ahead and upgrade the other node.
Re: upgrade to 2.2
Posted: Fri Sep 04, 2015 1:06 pm
by jolson
Quoted from the documentation:
During a rolling upgrade, primary shards assigned to a node with the higher version will never have their replicas assigned to a node with the lower version, because the newer version may have a different data format which is not understood by the older version.
If it is not possible to assign the replica shards to another node with the higher version — e.g. if there is only one node with the higher version in the cluster — then the replica shards will remain unassigned, i.e. the cluster health will be status yellow. As soon as another node with the higher version joins the cluster, the replicas should be assigned and the cluster health will reach status green.
Let's see where your shards are assigned:
Code: Select all
curl -s localhost:9200/_cat/shards
Are the primary shards (indicated by a 'p') assigned to your node with a higher version of elasticsearch?
You can check a nodes UUID by issuing the following command:
Code: Select all
cat /usr/local/nagioslogserver/var/node_uuid
If the primary shards are all on your recently upgraded node, you are good to upgrade your other 1.4 box. If not, let me know.
Re: upgrade to 2.2
Posted: Fri Sep 04, 2015 1:10 pm
by jvestrum
I went ahead and upgraded the other node. After that, it was still at Yellow. Then I re-enabled shard allocation (I had disabled it as per the docs) and now I'm back to Green. All looks good now, thanks.
Re: upgrade to 2.2
Posted: Fri Sep 04, 2015 1:11 pm
by jolson
No problem, glad to hear you're up and running.