ES dead on all 3 nodes of the cluster

[gormank]

I restarted the server, stopped logstash and restarted the dead ES service.
Another maybe (or maybe not) interesting thing I see is that the 3 hosts in the cluster seem to randomly connect to either the fe80 address or the (correct) 2001 address.

Right now, hosts 1 and 3 are on fe80 and 2 is on 2001.

[STAGING root@sandcaykhsc-v-sweslog-01 logstash]# ip a | egrep 'fe80|2001'
inet6 2001:4888:a00:3154:f0:ff2:0:b01/64 scope global noprefixroute
inet6 fe80::4aa2:59e6:ddb8:9e17/64 scope link noprefixroute

When it connects to the fe80 address can see from the ES log that the address can't be resolved (which is correct)
Error injecting constructor, org.elasticsearch.ElasticsearchIllegalArgumentException: Failed to resolve address for [fe80:0:0:0:4aa2:59e6:ddb8:9e17]
and
Caused by: java.lang.NumberFormatException: For input string: "9e17"
which is the last bit of the fe80 address.

[gormank]

Ok,
Since ES was binding to the wrong address, I set network.host to the address of each server, stopped ES, edited cluster_hosts (probably not needed), and restarted ES.

[STAGING root@sandcaykhsc-v-sweslog-01 ~]# grep ^network /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml
network.host: 2001:4888:a00:3154:f0:ff2:0:b01

Now each host binds to the right address and joins the cluster.
Next I restarted logstash and watched things for a while. Things are stable and the cluster is green.

Let's let it sit for a day and I'll post the results.

Thanks

[pbroste]

Hello @gormank

Following up checking on how things are looking, please let us know if we can go ahead and close this case.

Thanks,
Perry

[gormank]

Yes, feel free to close this.