Swapping out a cluster node?

[vAJ]

I have a question on a 2-node cluster. We're decommissioning a datacenter where one node resides. What steps need to be taken to drop this node?

I'm confident in adding the new node (just like creating the cluster in the first place), but I want to make sure the license frees up and that we don't have any shards lost.

Thanks,
Andrew

[scottwilkerson]

Andrew,

If you can take a short period of time without "searching & alerting" capabilities, here's what I would recommend.

1. add 3rd node
2. Once node is online and connected to the cluster, check cluster status and make sure you see Green
3. disconnect De-commissioned node

the alternative would be to
1. Check cluster status and make sure you see Green
2. turn off node you are De-commissioning
3. turn on replacement node

In either scenario you should not have a problem, the biggest thing to check is that the status is green before you start.
Once you shut down the node that is going offline it will switch to yellow while replica data is being copied to the new node, but everything will be fully functional while this is happening.

[vAJ]

Thanks Scott. Any issues with adding a 3rd node when I'm only licensed for 2?

[scottwilkerson]

Well, the only issue will be it will not allow searching until you decommission a node and get back to 2 nodes

[scottwilkerson]

going to add here that the timeline searching isn't allowed may be measured in seconds once you verify the 3rd node is up you can shut down the one you are removing

[vAJ]

So I added the 3rd node in, cluster status still shows green with no movement of shards. I do get the license violation at the top of all screens, though.

[scottwilkerson]

it will, then you shut down the machine you want to decommission and it will go to yellow and everything will replicate to the new node

[vAJ]

Cool, thanks.

I think we can lock this up. If anything hangs up on my, I'll ping you back.

[scottwilkerson]

Sounds good!