Nagios Support Forum

You are correct, after I ran out of disk space this started to occur. Screen shot attached.

You should just delete the following indexes as they are corrupt because they couldn't be created when you were out of disk space Once these are deleted your cluster should return to yellow state

I removed the files as requested. The Cluster status turned yellow. I then restarted the services and it appeared to work for a few minutes then stop collecting events again. I came back after lunch to find the cluster status back to Red again.

Please run the following command: The output will be messy, but we can find all of your failing shards this way. Look for anything with the health status of 'red'. You can delete an index with the following command: The cluster shows red - which means an index is red - which means shard(s) are red - typically corrupt or parentless. We need to figure out which indices the shards belong to. You can hold off on deleting for now if you'd like, and simply post the results of the first command - we'll come up with a plan from there.

It looks like it might be the latest index 3-30-15. Am I reading this correctly?

curl -XGET 'http://localhost:9200/_cluster/health/*?level=shards'
{"cluster_name":"3c75d8ce-06b4-409c-b5b7-1f834a9db496","status":"red","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":175,"active_shards":175,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":176,"indices":{"nagioslogserver":{"status":"yellow","number_of_shards":1,"number_of_replicas":1,"active_primary_shards":1,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.13":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.14":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.11":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.12":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.03.04":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.03.03":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.10":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"nagioslogserver_log":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.09":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.08":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.26":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.07":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.06":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.05":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.04":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.03.30":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":4,"active_shards":4,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.22":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.23":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.24":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.03.10":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.25":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.20":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.21":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.03.05":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.19":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.03.06":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.03.09":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.03":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.16":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.02":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.15":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.18":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"logstash-2015.02.17":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"a

You are reading it correctly - we could either delete the failed index if you're okay with losing the days data, or we could try re-assigning the shards since we're seeing (initializing_shards":1,"unassigned_shards":176).

If you want to delete the index, please proceed with the following: Status should turn to green shortly after deletion.


If you want to preserve the index, let's see the output of the following:

Ok removed the index. Waited a few hours and the cluster status is still yellow. Plus it will grab events for a short period and then stop. Screen shots are attached. Should I just cut my losses at this point, backup my dashboards, input filters, queries, etc and start over?

Yellow is a recovering state - can you please look at your 'Index Status' page and return the results? If you would like to start over, it may not be a bad idea - I don't think that the damage here is unrepairable.

Ok, thanks. Let's see where we get and we'll go from there. Index page attached.

Please note that your number of successful shards is half the number of total shards. How many nodes are in this cluster? If you have one node in your cluster, this is normal behavior. If you have more than one node, it is not normal behavior and I request that you run the following commands:

see shard status: see recovery: pending tasks: see nodes: Best,

Jesse