(Massive) Data Transfer Between Sites

[myriad]

I have a three instance setup.
I have one server at each site.
Whenever one server is rebooted, we're seeing loads of traffic transfer from site to site.
Why is this happening?
Am I setup wrong?
All three log servers will need to be at a single site?
If I want coverage at the three sites, will I have to separate the servers so that they will not be in a cluster?

[lgroschen]

I think this is intended. When you have 3+ servers in your cluster there is a redundant 'shard' of data that is allocated for each instance. So with 3 instances there will be 1 primary and 1 backup shard for a total of 6 shards.

When you bring a server down the other 2 servers have to make up for the hole in the data and use the backup shards of the downed server to be able to run queries on your User Interface so that you can see all the data that was on the downed server. The traffic you are seeing is likely the allocation of shards and jobs after the 3rd server went down.

Are you having to reboot servers often? You shouldn't need to so this may be abnormal behavior depending on why you reboot them.

[myriad]

I have had to reboot frequently to keep the servers up. now I'm looking at the index status over the last two days. since the 21st of January I was logging about 1.6-6GB daily. the number of reporting hosts bounces around. right now, I see 98 but over 160 should be reporting.

[scottwilkerson]

Am I setup wrong?
All three log servers will need to be at a single site?
If I want coverage at the three sites, will I have to separate the servers so that they will not be in a cluster?

The servers in the cluster are in constant communication and need to be able to transfer significant data when a machine goes offline. This is expected behavior to protect your data, making sure there is always at least 1 primary and 1 replica shard for all of the log data you have.

Depending on your network infrastructure, it may be better to have multiple clusters if the data transfer between machines it problematic, however I would not recommend having any less than 2 instances in each cluster because you would not have any redundancy.