Cluster Health status yellow Indices not showing on 1 leg

[Jklre]

I recently setup 2 separate clusters for nagios log server. The new servers are showing there health as yellow and on one of the legs the Indices are not showing up while they are showing up on the other leg.

Cluster Health
Status Yellow
Timed Out? false
# Instances 2
# Data Instances 2
Active Primary Shards 61
Active Shards 91
Relocating Shards 0
Initializing Shards 0
Unassigned Shards 31

1.jpg

Any Idea whats causing this? Thank you.

[Box293]

What version of Nagios Log Server?

I recently setup 2 separate clusters for nagios log server. The new servers are showing there health as yellow and on one of the legs the Indices are not showing up while they are showing up on the other leg.

The two separate clusters will have no relationship with each other.

I want some clarification, how many nodes does each cluster have?

[Jklre]

What version of Nagios Log Server?

I recently setup 2 separate clusters for nagios log server. The new servers are showing there health as yellow and on one of the legs the Indices are not showing up while they are showing up on the other leg.

The two separate clusters will have no relationship with each other.

I want some clarification, how many nodes does each cluster have?

I have 3 different clusters (one per datacenter) 2 nodes each.

[jolson]

Jklre,

To get this straight-

You have a 3 instance cluster, and while two of the instances appear to be functioning properly, one of them appears to be disconnected. Is that correct?

[Jklre]

Jklre,

To get this straight-

You have a 3 instance cluster, and while two of the instances appear to be functioning properly, one of them appears to be disconnected. Is that correct?

Actually not quite

I have 3 different clusters all with 2 nodes. (see bad photoshop visual X means not showing them)

2.jpg

one node on each the other 2 are not showing the Indies. Thank you.

[jolson]

I have a couple of hunches about what might be happening here.

On *one* of your clusters, I'd like to see the output of the following on each instance:

Code: Select all

cat /usr/local/nagioslogserver/var/node_uuid
cat /usr/local/nagioslogserver/var/cluster_uuid
cat /usr/local/nagioslogserver/var/cluster_hosts
ip a

[Jklre]

I have a couple of hunches about what might be happening here.

On *one* of your clusters, I'd like to see the output of the following on each instance:

Code: Select all

cat /usr/local/nagioslogserver/var/node_uuid
cat /usr/local/nagioslogserver/var/cluster_uuid
cat /usr/local/nagioslogserver/var/cluster_hosts
ip a

Here's the output from one of the clusters.

cat /usr/local/nagioslogserver/var/node_uuid
d2372b37-3f94-4d4b-896d-d62812c1806a
cat /usr/local/nagioslogserver/var/cluster_uuid
886c2610-e1e0-4f9f-a33c-5de17cff9435
cat /usr/local/nagioslogserver/var/cluster_hosts
localhost
*.*.20.93
*.*.20.94
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:95:1d:28 brd ff:ff:ff:ff:ff:ff
inet *.*.20.93/23 brd *.*.21.255 scope global eth0
inet6 fe80::250:56ff:fe95:1d28/64 scope link
valid_lft forever preferred_lft forever

cat /usr/local/nagioslogserver/var/node_uuid
20370d97-c74e-4d34-9933-0646a06bf34e
cat /usr/local/nagioslogserver/var/cluster_uuid
886c2610-e1e0-4f9f-a33c-5de17cff9435
cat /usr/local/nagioslogserver/var/cluster_hosts
localhost

UNLS01lxv
*.*.20.93
*.*.20.94
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:95:06:0a brd ff:ff:ff:ff:ff:ff
inet *.*.20.94/23 brd *.*.21.255 scope global eth0
inet6 fe80::250:56ff:fe95:60a/64 scope link
valid_lft forever preferred_lft forever

[jolson]

Try removing the following from your cluster_hosts file:
UNLS01lxv

Does the cluster uuid 886c2610-e1e0-4f9f-a33c-5de17cff9435 match the UUID you see on the 'Administration -> Cluster Status' page of the Web GUI?

[Jklre]

Try removing the following from your cluster_hosts file:
UNLS01lxv

Does the cluster uuid 886c2610-e1e0-4f9f-a33c-5de17cff9435 match the UUID you see on the 'Administration -> Cluster Status' page of the Web GUI?

The UUID does match with 886c2610-e1e0-4f9f-a33c-5de17cff9435

The cluster_hosts file now matches on both.

localhost
172.24.20.93
172.24.20.94

I did a restart of each leg and now on UNLS01LXV "No results There were no results because no indices were found that match your selected time span"

[jolson]

I did a restart of each leg and now on UNLS01LXV "No results There were no results because no indices were found that match your selected time span"

Did the indices populate properly in the 'Index Status' page? Are the dates of your two servers synchronized?

Code: Select all

date

Please run the following command on each leg and report the output:

Code: Select all

curl -s localhost:9200/_cat/shards