Maintenance of Elastic Search Nodes

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
stecino
Posts: 248
Joined: Thu Mar 14, 2013 4:42 pm

Maintenance of Elastic Search Nodes

Post by stecino »

Hello,

I want to remove NLS nodes out of the cluster, in order to perform maintenance (disk expansion)


I have Step-by-step guide for ElasticSearch rolling restart, and whether it would apply to NLS

Check cluster setting: curl -u XXX:XXX -XGET "http://localhost:9200/_cluster/settings?pretty=true"

Disable shard allocation:
curl -u XXX:XXX -XPUT "http://localhost:9200/_cluster/settings" -d'
{
"transient" : {
"cluster.routing.allocation.enable" : "none"
}
}'

shutdown the node
curl -u xxx:xxx-XPOST "http://localhost:9200/_cluster/nodes/_local/_shutdown"
Wait for the connection clear out.
netstat -on | grep 9200|wc
netstat -on | grep 9300|wc

upgrade/server maintenance or .....

restart elasticsearch: service elasticsearch start

Enable shard allocation:
curl -u xxx:xxx -XPUT "http://localhost:9200/_cluster/settings" -d'
{
"transient" : {
"cluster.routing.allocation.enable" : "all"
}
}'
Wait for cluster status to green. Repeat step 1 to 6 for all other nodes.

* The speed of shard reallocation is related to following:

cluster.routing.allocation.cluster_concurrent_rebalance default is 2
indices.recovery.max_bytes_per_sec defaults to 40mb
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Maintenance of Elastic Search Nodes

Post by jolson »

The answer is yes - this applies to Nagios Log Server as well, and is very close to our own maintenance document: https://assets.nagios.com/downloads/nag ... Server.pdf

You could use either procedure without trouble. Thanks!
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
stecino
Posts: 248
Joined: Thu Mar 14, 2013 4:42 pm

Re: Maintenance of Elastic Search Nodes

Post by stecino »

jolson wrote:The answer is yes - this applies to Nagios Log Server as well, and is very close to our own maintenance document: https://assets.nagios.com/downloads/nag ... Server.pdf

You could use either procedure without trouble. Thanks!
Perfect!! Thanks
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:

Re: Maintenance of Elastic Search Nodes

Post by hsmith »

stecino wrote:
jolson wrote:The answer is yes - this applies to Nagios Log Server as well, and is very close to our own maintenance document: https://assets.nagios.com/downloads/nag ... Server.pdf

You could use either procedure without trouble. Thanks!
Perfect!! Thanks
Are we all right to mark this one as resolved and lock it up?
Former Nagios Employee.
me.
Locked