Node Maintenance

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
stecino
Posts: 248
Joined: Thu Mar 14, 2013 4:42 pm

Node Maintenance

Post by stecino »

Hi All,

I need to add more storage, CPU and memory to individual cluster nodes. How can I take a node out of rotation properly so I won't have to corrupt anything.
Also what would happen to the node where logstash is collecting locally?

Thanks in advance
User avatar
lgroschen
Posts: 384
Joined: Wed Nov 27, 2013 1:17 pm

Re: Node Maintenance

Post by lgroschen »

The shards have all been backed up on your other nodes so as long as you properly shut down your system the shards on the system that is going down will be recovered by the other nodes so you won't lose any data. Of course, when you turn the machine off to upgrade the hardware it won't report anything locally, but once you introduce it back into the cluster it will be allocated shards, and more than likely more since it's getting an upgrade.

Just do one at a time and give each some time to allocate shards. The best way to know if the cluster is ready to go is to navigate here:

Administration (navigation bar at top of page) > Cluster Status (left side menu) > Cluster Health > Status:

It will be either green, yellow or red. If the cluster shows a Green status then you know the cluster is finished allocating all the shards and you can move onto another server to upgrade.
/Luke
stecino
Posts: 248
Joined: Thu Mar 14, 2013 4:42 pm

Re: Node Maintenance

Post by stecino »

lgroschen wrote:The shards have all been backed up on your other nodes so as long as you properly shut down your system the shards on the system that is going down will be recovered by the other nodes so you won't lose any data. Of course, when you turn the machine off to upgrade the hardware it won't report anything locally, but once you introduce it back into the cluster it will be allocated shards, and more than likely more since it's getting an upgrade.

Just do one at a time and give each some time to allocate shards. The best way to know if the cluster is ready to go is to navigate here:

Administration (navigation bar at top of page) > Cluster Status (left side menu) > Cluster Health > Status:

It will be either green, yellow or red. If the cluster shows a Green status then you know the cluster is finished allocating all the shards and you can move onto another server to upgrade.
Thanks, I actually had stopped elasticSearch and logstash on the node. Then I just shutdown the VM and did the resource allocation. Once I joined it back, I noticed that non of the stats for the node were present. It took a while for it to appear. The hosts that I have are testing only, so don't really care about the data.
User avatar
lgroschen
Posts: 384
Joined: Wed Nov 27, 2013 1:17 pm

Re: Node Maintenance

Post by lgroschen »

Let us know how this goes for you or if you need any more assistance.
/Luke
stecino
Posts: 248
Joined: Thu Mar 14, 2013 4:42 pm

Re: Node Maintenance

Post by stecino »

lgroschen wrote:Let us know how this goes for you or if you need any more assistance.
Will do thanks
Locked