nagios log server cluster status is red

[hlyeung]

hi, we have three nodes cluster and found the cluster status is red. then i remove one node which had problems. now we left two nodes. how can we make the cluster become to green?

[tmcdonald]

That will of course depend on the individual status of each node in the cluster.

Are any of them experiencing a high CPU or memory usage? Are any of their disks full? What was the problem on the one node you removed?

[jomann]

Did you remove more than one node? Red might also indicate that you are missing shards (big sets of data) that are now missing.

[hlyeung]

The problem is solved as we found one of server's port 9300 is not opened so at last the cluser is in split brain mode. After we opened the firewall and restart the server, it become normal.
thanks for your help.

For prevent this happened again, we need setup discovery.zen.minimum_master_nodes: 3 (assume we have 4 nodes), where can we change the config file?

also for the cluster, do we need specify discovery.zen.ping.unicast.hosts: ["localhost"] and put all nodes IP?


thanks for your help again

[jomann]

The elasticsearch.yml config file is located in /usr/local/nagioslogserver/elasticsearch/config. As for the discovery.zen.ping.unicast.hosts settings you should not need to change this as long as you've added all instances via the setup when you install a new instance of Log Server and give it the cluster ID.