Replication to new server too slow
Posted: Mon Nov 12, 2018 4:16 pm
Hello,
Good afternoon,
I am virtualizing a 15TB physical server with NLS.
My cluster has 4 servers, each with 16TB (totaling 64TB).
Servers are under load; because; although we have a large volume of logs; only a few days of logs are open; and the rest is stored "closed"; for exporadic research.
Each node has the maximum memory that the elasticsearch supports (31GB); plus 20GB which are free for the operating system.
For virtualization, I shut down a server; and wait for the indexes to replicate, and the state of the cluster turns green; After, I installed a new virtual server; and added in the cluster.
The virtualization infrastructure is under an all-flash storage; what an excellent performance; and with 60 CPU cores available.
However, we are realizing that data replication will take a long time, because the partition where the data from the elasticsearch is is growing very vaguely (about 3-5MBs per second).
We have already verified that there are no errors in the logs, or any performance problems. We believe that the default elasticsearch configuration is conservative; and we would like to do a tuning for more replication.
In internet searches; we noticed that some people try to change the "max_bytes_per_sec" attribute to improve replication \ recovery.
There is some setting we can change to improve this replication. By calculations it will take months at this speed.
We have the latest version of NLS
Good afternoon,
I am virtualizing a 15TB physical server with NLS.
My cluster has 4 servers, each with 16TB (totaling 64TB).
Servers are under load; because; although we have a large volume of logs; only a few days of logs are open; and the rest is stored "closed"; for exporadic research.
Each node has the maximum memory that the elasticsearch supports (31GB); plus 20GB which are free for the operating system.
For virtualization, I shut down a server; and wait for the indexes to replicate, and the state of the cluster turns green; After, I installed a new virtual server; and added in the cluster.
The virtualization infrastructure is under an all-flash storage; what an excellent performance; and with 60 CPU cores available.
However, we are realizing that data replication will take a long time, because the partition where the data from the elasticsearch is is growing very vaguely (about 3-5MBs per second).
We have already verified that there are no errors in the logs, or any performance problems. We believe that the default elasticsearch configuration is conservative; and we would like to do a tuning for more replication.
In internet searches; we noticed that some people try to change the "max_bytes_per_sec" attribute to improve replication \ recovery.
There is some setting we can change to improve this replication. By calculations it will take months at this speed.
We have the latest version of NLS