Remove instance from Nagios Log Server Cluster

[Sampath.Basireddy]

We have 4 instance NLS cluster. I would like to either isolate or delete 2 instances from the 4 instance cluster.

Is there any chance of loosing data in this case. If I just isolate the 2 instances, can I add them back at a later point of time?

Please suggest.

[cdienger]

There is a possibility of loosing data if you remove both nodes at the same time. The safer method would be to remove one node, wait for the cluster status to go back to green which indicates the data has been redistributed among the three nodes left in the cluster, and then remove the next node and again wait for the green status to indicate that the data is now distributed between the two remaining nodes.

One important note is that closed indices will not replicate. You'll need to open closed indices to allow them to replicate. A script to open closed indices to allow them to replicate can be found at https://github.com/elastic/elasticsearch/issues/12963. The steps would be:

-remove a node
-run the script
-wait for the status to go green
-remove second node
-run the script
-wait for the status to go green

https://assets.nagios.com/downloads/nag ... luster.pdf has information about isolating or removing nodes from a cluster. If you need to add nodes back to the cluster I would recommend just a new installation of NLS and then add them to the cluster as you usually do during the install process.

[Sampath.Basireddy]

Thanks @cdienger.

I don't have any closed indices, so I can simply go ahead with below steps to remove the instance from cluster, right?

Code: Select all

• Stop Elasticsearch on all instances
• Update configuration file
• Start Elasticsearch on all instances
• Verify instances in cluster
• Delete removed instance

[cdienger]

That should work. To stress it again though - remove one node at a time.

[Sampath.Basireddy]

Definitely, one node at a time.

With respect to the script: https://github.com/elastic/elasticsearch/issues/12963, does this script help in any other case where there are "Unassigned Shards"?

I have another instance of NLS which has yellow cluster health status. It has 60 open indices and 30 closed indices.

[Sampath.Basireddy]

Nevermind, NLS was running on low space, hence the shards were unassigned.

Adding space fixed it.

[cdienger]

It may help here, but I would first review https://support.nagios.com/kb/article/n ... h-469.html and run "curl -XGET 'http://localhost:9200/_cat/shards?v'" to get an idea of of which indices/shards are int this state and why. Please open a new thread if we need to examine this further.

[Sampath.Basireddy]

@cdienger Please close this topic.

I am all set now. Thank You so much for you support.

[cdienger]

Glad to hear!