Cluster is now red with unassigned shards and missing indice
Cluster is now red with unassigned shards and missing indice
We keep having problems with our cluster health, unassigned indices, and backups. Please see the screenshot:
You do not have the required permissions to view the files attached to this post.
Re: Cluster is now red with unassigned shards and missing in
Bump for urgency.
We do not appear to be receiving any new logs at this time.
We do not appear to be receiving any new logs at this time.
Re: Cluster is now red with unassigned shards and missing in
Bumping will not result in a faster response time to your thread. We have a dashboard system we use internally that tracks posts by age, and bumping it will make your thread look newer than it actually is.
Can I see the output of the following commands?
Can I see the output of the following commands?
Code: Select all
free -m
df -ih
df -h
curl 'localhost:9200/_cluster/health?level=indices&pretty'
Former Nagios Employee.
me.
me.
Re: Cluster is now red with unassigned shards and missing in
Code: Select all
free -m
total used free shared buffers cached
Mem: 32110 31881 228 0 251 10950
-/+ buffers/cache: 20679 11431
Swap: 4031 0 4031
Code: Select all
df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
61M 131K 61M 1% /
tmpfs 4.0M 2 4.0M 1% /dev/shm
/dev/sda1 126K 46 125K 1% /boot
/dev/mapper/VolGroup-lv_home
2.9M 17 2.9M 1% /home
192.168.29.20:/data/logging
349M 162K 349M 1% /mnt/repo
Code: Select all
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
958G 456G 454G 51% /
tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 477M 62M 390M 14% /boot
/dev/mapper/VolGroup-lv_home
45G 52M 43G 1% /home
192.168.29.20:/data/logging
5.4T 2.1T 3.0T 42% /mnt/repo
Code: Select all
curl 'localhost:9200/_cluster/health?level=indices&pretty'
{
"cluster_name" : "98ec1d45-73e0-4296-ba7f-d7d71953b7e8",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 171,
"active_shards" : 315,
"relocating_shards" : 0,
"initializing_shards" : 2,
"unassigned_shards" : 25,
"number_of_pending_tasks" : 1,
"number_of_in_flight_fetch" : 0,
"indices" : {
"nagioslogserver" : {
"status" : "yellow",
"number_of_shards" : 1,
"number_of_replicas" : 1,
"active_primary_shards" : 1,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
},
"logstash-2015.12.08" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.26" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.25" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 9,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
},
"logstash-2015.12.07" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5
},
"logstash-2015.12.28" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.09" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 9,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
},
"logstash-2015.12.27" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 2,
"unassigned_shards" : 3
},
"logstash-2016.01.03" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.01.04" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2016.01.02" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 6,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 4
},
"logstash-2015.12.29" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5
},
"logstash-2015.01.03" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.06" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.05" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2016.01.04" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.10" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"nagioslogserver_log" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.11" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.12" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.31" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.13" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 9,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
},
"logstash-2015.12.19" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.18" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.17" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 9,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
},
"logstash-2015.12.16" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.15" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.14" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.23" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.24" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2016.01.01" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.21" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.22" : {
"status" : "green",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"logstash-2015.12.20" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 9,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
},
"kibana-int" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 8,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2
}
}
}
Re: Cluster is now red with unassigned shards and missing in
@mike4vr , would you be able to send in a ticket to [email protected]? I discussed this with jolson, and this looks like something that may be easiest resolved in a remote session.
Thanks.
Thanks.
Former Nagios Employee.
me.
me.
Re: Cluster is now red with unassigned shards and missing in
Ticket sent and time booked. Thanks.
Re: Cluster is now red with unassigned shards and missing in
I'll go ahead and close this post, since we will handle the communication through the ticket.
Thank you.
Thank you.
Former Nagios Employee.
me.
me.