Page 1 of 1
nagios log server cluster status is red
Posted: Mon Dec 15, 2014 11:53 pm
by hlyeung
hi, we have three nodes cluster and found the cluster status is red. then i remove one node which had problems. now we left two nodes. how can we make the cluster become to green?
Re: nagios log server cluster status is red
Posted: Tue Dec 16, 2014 10:21 am
by tmcdonald
That will of course depend on the individual status of each node in the cluster.
Are any of them experiencing a high CPU or memory usage? Are any of their disks full? What was the problem on the one node you removed?
Re: nagios log server cluster status is red
Posted: Tue Dec 16, 2014 1:40 pm
by jomann
Did you remove more than one node? Red might also indicate that you are missing shards (big sets of data) that are now missing.
Re: nagios log server cluster status is red
Posted: Wed Dec 17, 2014 2:35 am
by hlyeung
The problem is solved as we found one of server's port 9300 is not opened so at last the cluser is in split brain mode. After we opened the firewall and restart the server, it become normal.
thanks for your help.
For prevent this happened again, we need setup discovery.zen.minimum_master_nodes: 3 (assume we have 4 nodes), where can we change the config file?
also for the cluster, do we need specify discovery.zen.ping.unicast.hosts: ["localhost"] and put all nodes IP?
thanks for your help again
Re: nagios log server cluster status is red
Posted: Wed Dec 17, 2014 12:04 pm
by jomann
The elasticsearch.yml config file is located in /usr/local/nagioslogserver/elasticsearch/config. As for the discovery.zen.ping.unicast.hosts settings you should not need to change this as long as you've added all instances via the setup when you install a new instance of Log Server and give it the cluster ID.