Instances don't see each other or sync

[vmesquita]

Hello,

We had a network shortage and now the two Log Server instance don't see each other anymore:

instance2.png

instance1.png

How can we fix this?

[rkennedy]

Just to check the obvious, from each machine can you ping the other one?

What is the output of these commands? cat /usr/local/nagioslogserver/var/cluster_hosts curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' and curl 'localhost:9200/_cat/nodes?v'?

[vmesquita]

Yes, they ping each other. This is the result in one instance:

Code: Select all

[root@sa585 vmesquita]# cat /usr/local/nagioslogserver/var/cluster_hosts | curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "155bf3a6-eb50-41aa-8f41-112a4db21fc6",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 1076,
  "active_shards" : 1076,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1076,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}
[root@sa585 vmesquita]# curl 'localhost:9200/_cat/nodes?v'
host  ip             heap.percent ram.percent load node.role master name
sa585 172.27.164.109           66          66 0.17 d         *      6a7ce4ea-e1b9-47a1-af18-1c4d47243d20

And in the other:

Code: Select all

[root@sb585 vmesquita]# cat /usr/local/nagioslogserver/var/cluster_hosts | curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "155bf3a6-eb50-41aa-8f41-112a4db21fc6",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 1076,
  "active_shards" : 1076,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1076,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}
[root@sb585 vmesquita]# curl 'localhost:9200/_cat/nodes?v'

host  ip            heap.percent ram.percent load node.role master name
sb585 172.27.36.109           23          73 0.27 d         *      636f559a-1fd5-4158-9dee-92e1a8403a1e

[hsmith]

Can you run this on both nodes? It looks like you piped it in to curl, so we didn't get the actual output of the command.

Code: Select all

cat /usr/local/nagioslogserver/var/cluster_hosts

[vmesquita]

ok:

Code: Select all

[root@sa585 tmp]# cat /usr/local/nagioslogserver/var/cluster_hosts
localhost

sb585.selic.bc
172.27.36.109
172.27.164.109

and

Code: Select all

[root@sb585 nagioslogserver]# cat /usr/local/nagioslogserver/var/cluster_hosts
172.27.164.109
172.27.36.109

[hsmith]

These should match. Make sure you add

Code: Select all

localhost

sb585.selic.bc

to the second one. Or take away nodes that are not actually part of the cluster in that file.