ElasticSearch red status after disk full

[chud]

We had an issue with our Cisco Firepower sending too many logs and it filled up our NFS share where Nagios Log Server stores its logs, sending the cluster into RED status.

I am trying to determine if ElasticSearch will be able to repair itself.

The number of unassigned shards has gone down, and I understand that this is a good sign, however I do not know enough about the inner workings of ElasticSearch yet to be certain.

How can I tell for sure?

[scottwilkerson]

Time will tell but one thing that can happen often with a full disk is the latest index fails to get created successfully, you can see this in Admin -> Index Status

If today's index has 0 documents, you should delete it and let it get re-created

[chud]

It looks like today's index has 205,472 documents, so that's good.
Thanks for your reply.

[scottwilkerson]

It looks like today's index has 205,472 documents, so that's good.
Thanks for your reply.

Hopefully all the shards will be able to restore

[chud]

Hopefully all the shards will be able to restore

Today's index has 187,504 documents.
However, the cluster status is still red, and there are still 92 unassigned shards, the same as last Friday.

Code: Select all

# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "409efd36-ffab-4d6f-a8ae-78e10d15a1ac",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 89,
  "active_shards" : 89,
  "relocating_shards" : 0,
  "initializing_shards" : 3,
  "unassigned_shards" : 92,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 1,
  "number_of_in_flight_fetch" : 0
}

[scottwilkerson]

Is this just a 1 instance cluster?

Can you show the output of this

Code: Select all

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

also, this

Code: Select all

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason

[chud]

Is this just a 1 instance cluster?

Yes, so far. I want to add a second instance after elasticsearch is back to green status.

Can you show the output of this

Code: Select all

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

# curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11040 100 11040 0 0 4664 0 0:00:02 0:00:02 --:--:-- 4664
logstash-2019.09.22 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 4 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 4 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 0 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 3 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 1 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.07 4 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 0 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 3 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 1 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 2 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 4 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 0 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 3 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 1 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 2 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 4 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 0 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 3 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 1 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 2 r UNASSIGNED INDEX_CREATED
logstash-2019.09.30 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 2 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver 0 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 4 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 0 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 3 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 1 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 2 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_history 0 r UNASSIGNED CLUSTER_RECOVERED

also, this

Code: Select all

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason

# curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason
logstash-2019.09.22 2 p STARTED
logstash-2019.09.22 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 0 p STARTED
logstash-2019.09.22 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 3 p STARTED
logstash-2019.09.22 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 1 p STARTED
logstash-2019.09.22 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 4 p STARTED
logstash-2019.09.22 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 2 p STARTED
logstash-2019.09.23 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 0 p STARTED
logstash-2019.09.23 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 3 p STARTED
logstash-2019.09.23 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 1 p STARTED
logstash-2019.09.23 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 4 p STARTED
logstash-2019.09.23 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 4 p STARTED
logstash-2019.09.24 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 0 p STARTED
logstash-2019.09.24 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 3 p STARTED
logstash-2019.09.24 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 1 p STARTED
logstash-2019.09.24 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 2 p STARTED
logstash-2019.09.24 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 2 p STARTED
logstash-2019.10.03 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 0 p STARTED
logstash-2019.10.03 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 3 p STARTED
logstash-2019.10.03 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 1 p STARTED
logstash-2019.10.03 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 4 p STARTED
logstash-2019.10.03 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 2 p STARTED
logstash-2019.09.25 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 0 p STARTED
logstash-2019.09.25 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 3 p STARTED
logstash-2019.09.25 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 1 p STARTED
logstash-2019.09.25 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 4 p STARTED
logstash-2019.09.25 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 4 p STARTED
logstash-2019.10.04 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 0 p STARTED
logstash-2019.10.04 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 3 p STARTED
logstash-2019.10.04 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 1 p STARTED
logstash-2019.10.04 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 2 p STARTED
logstash-2019.10.04 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 2 p STARTED
logstash-2019.09.26 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 0 p STARTED
logstash-2019.09.26 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 3 p STARTED
logstash-2019.09.26 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 1 p STARTED
logstash-2019.09.26 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 4 p STARTED
logstash-2019.09.26 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 4 p STARTED
logstash-2019.10.01 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 0 p STARTED
logstash-2019.10.01 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 3 p INITIALIZING ALLOCATION_FAILED
logstash-2019.10.01 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 1 p STARTED
logstash-2019.10.01 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 2 p STARTED
logstash-2019.10.01 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 2 p STARTED
logstash-2019.09.27 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 0 p STARTED
logstash-2019.09.27 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 3 p STARTED
logstash-2019.09.27 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 1 p STARTED
logstash-2019.09.27 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 4 p STARTED
logstash-2019.09.27 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 4 p STARTED
logstash-2019.10.02 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 0 p INITIALIZING ALLOCATION_FAILED
logstash-2019.10.02 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 3 p STARTED
logstash-2019.10.02 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 1 p STARTED
logstash-2019.10.02 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 2 p STARTED
logstash-2019.10.02 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 2 p INITIALIZING ALLOCATION_FAILED
logstash-2019.09.28 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 0 p STARTED
logstash-2019.09.28 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 3 p STARTED
logstash-2019.09.28 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 1 p STARTED
logstash-2019.09.28 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 4 p STARTED
logstash-2019.09.28 4 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 2 p STARTED
kibana-int 2 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 0 p STARTED
kibana-int 0 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 3 p STARTED
kibana-int 3 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 1 p STARTED
kibana-int 1 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 4 p STARTED
kibana-int 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.07 2 p STARTED
logstash-2019.10.07 2 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 0 p STARTED
logstash-2019.10.07 0 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 3 p STARTED
logstash-2019.10.07 3 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 1 p STARTED
logstash-2019.10.07 1 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 4 p STARTED
logstash-2019.10.07 4 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 2 p STARTED
logstash-2019.10.05 2 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 0 p STARTED
logstash-2019.10.05 0 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 3 p STARTED
logstash-2019.10.05 3 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 1 p STARTED
logstash-2019.10.05 1 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 4 p STARTED
logstash-2019.10.05 4 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 2 p STARTED
logstash-2019.10.06 2 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 0 p STARTED
logstash-2019.10.06 0 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 3 p STARTED
logstash-2019.10.06 3 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 1 p STARTED
logstash-2019.10.06 1 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 4 p STARTED
logstash-2019.10.06 4 r UNASSIGNED INDEX_CREATED
logstash-2019.09.30 2 p STARTED
logstash-2019.09.30 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 0 p STARTED
logstash-2019.09.30 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 3 p STARTED
logstash-2019.09.30 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 1 p STARTED
logstash-2019.09.30 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 4 p STARTED
logstash-2019.09.30 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 2 p STARTED
logstash-2019.09.29 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 0 p STARTED
logstash-2019.09.29 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 3 p STARTED
logstash-2019.09.29 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 1 p STARTED
logstash-2019.09.29 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 4 p STARTED
logstash-2019.09.29 4 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver 0 p STARTED
nagioslogserver 0 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 2 p STARTED
nagioslogserver_log 2 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 0 p STARTED
nagioslogserver_log 0 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 3 p STARTED
nagioslogserver_log 3 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 1 p STARTED
nagioslogserver_log 1 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 4 p STARTED
nagioslogserver_log 4 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_history 0 p STARTED
nagioslogserver_history 0 r UNASSIGNED CLUSTER_RECOVERED

[scottwilkerson]

Here are the 3 you are having problems with

Code: Select all

logstash-2019.10.01 3 p INITIALIZING ALLOCATION_FAILED
logstash-2019.10.02 0 p INITIALIZING ALLOCATION_FAILED
logstash-2019.09.28 2 p INITIALIZING ALLOCATION_FAILED

How much free space is available?

Do you have snapshots of these 3 indexes?
https://assets.nagios.com/downloads/nag ... enance.pdf

[chud]

Here are the 3 you are having problems with

Code: Select all

logstash-2019.10.01 3 p INITIALIZING ALLOCATION_FAILED
logstash-2019.10.02 0 p INITIALIZING ALLOCATION_FAILED
logstash-2019.09.28 2 p INITIALIZING ALLOCATION_FAILED

Thanks.

How much free space is available?

216G

Do you have snapshots of these 3 indexes?

Unfortunately, no. I didn't enable snapshots until recently.

[scottwilkerson]

Unfortunately you likely lost some of the data in each of those indexes. You can close these indexes in Admin -> Index status and that should get you back to Yellow status (cannot get green until you add another Instance to the cluster as there is no replication)

Code: Select all

logstash-2019.09.28
logstash-2019.10.01
logstash-2019.10.02