Page 1 of 2

ElasticSearch red status after disk full

Posted: Fri Oct 04, 2019 9:53 am
by chud
We had an issue with our Cisco Firepower sending too many logs and it filled up our NFS share where Nagios Log Server stores its logs, sending the cluster into RED status.

I am trying to determine if ElasticSearch will be able to repair itself.

The number of unassigned shards has gone down, and I understand that this is a good sign, however I do not know enough about the inner workings of ElasticSearch yet to be certain.

How can I tell for sure?

Re: ElasticSearch red status after disk full

Posted: Fri Oct 04, 2019 10:33 am
by scottwilkerson
Time will tell but one thing that can happen often with a full disk is the latest index fails to get created successfully, you can see this in Admin -> Index Status

If today's index has 0 documents, you should delete it and let it get re-created

Re: ElasticSearch red status after disk full

Posted: Fri Oct 04, 2019 11:15 am
by chud
It looks like today's index has 205,472 documents, so that's good.
Thanks for your reply.

Re: ElasticSearch red status after disk full

Posted: Fri Oct 04, 2019 11:22 am
by scottwilkerson
chud wrote:It looks like today's index has 205,472 documents, so that's good.
Thanks for your reply.
Hopefully all the shards will be able to restore

Re: ElasticSearch red status after disk full

Posted: Mon Oct 07, 2019 9:38 am
by chud
scottwilkerson wrote: Hopefully all the shards will be able to restore
Today's index has 187,504 documents.
However, the cluster status is still red, and there are still 92 unassigned shards, the same as last Friday.

Code: Select all

# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "409efd36-ffab-4d6f-a8ae-78e10d15a1ac",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 89,
  "active_shards" : 89,
  "relocating_shards" : 0,
  "initializing_shards" : 3,
  "unassigned_shards" : 92,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 1,
  "number_of_in_flight_fetch" : 0
}

Re: ElasticSearch red status after disk full

Posted: Mon Oct 07, 2019 10:05 am
by scottwilkerson
Is this just a 1 instance cluster?

Can you show the output of this

Code: Select all

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
also, this

Code: Select all

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason

Re: ElasticSearch red status after disk full

Posted: Mon Oct 07, 2019 10:38 am
by chud
scottwilkerson wrote:Is this just a 1 instance cluster?
Yes, so far. I want to add a second instance after elasticsearch is back to green status.
scottwilkerson wrote:Can you show the output of this

Code: Select all

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
# curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11040 100 11040 0 0 4664 0 0:00:02 0:00:02 --:--:-- 4664
logstash-2019.09.22 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 4 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 4 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 0 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 3 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 1 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.07 4 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 0 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 3 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 1 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 2 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 4 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 0 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 3 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 1 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 2 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 4 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 0 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 3 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 1 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 2 r UNASSIGNED INDEX_CREATED
logstash-2019.09.30 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 2 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver 0 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 4 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 0 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 3 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 1 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 2 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_history 0 r UNASSIGNED CLUSTER_RECOVERED
scottwilkerson wrote:also, this

Code: Select all

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason
# curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason
logstash-2019.09.22 2 p STARTED
logstash-2019.09.22 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 0 p STARTED
logstash-2019.09.22 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 3 p STARTED
logstash-2019.09.22 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 1 p STARTED
logstash-2019.09.22 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.22 4 p STARTED
logstash-2019.09.22 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 2 p STARTED
logstash-2019.09.23 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 0 p STARTED
logstash-2019.09.23 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 3 p STARTED
logstash-2019.09.23 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 1 p STARTED
logstash-2019.09.23 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.23 4 p STARTED
logstash-2019.09.23 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 4 p STARTED
logstash-2019.09.24 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 0 p STARTED
logstash-2019.09.24 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 3 p STARTED
logstash-2019.09.24 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 1 p STARTED
logstash-2019.09.24 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.24 2 p STARTED
logstash-2019.09.24 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 2 p STARTED
logstash-2019.10.03 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 0 p STARTED
logstash-2019.10.03 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 3 p STARTED
logstash-2019.10.03 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 1 p STARTED
logstash-2019.10.03 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.03 4 p STARTED
logstash-2019.10.03 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 2 p STARTED
logstash-2019.09.25 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 0 p STARTED
logstash-2019.09.25 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 3 p STARTED
logstash-2019.09.25 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 1 p STARTED
logstash-2019.09.25 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.25 4 p STARTED
logstash-2019.09.25 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 4 p STARTED
logstash-2019.10.04 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 0 p STARTED
logstash-2019.10.04 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 3 p STARTED
logstash-2019.10.04 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 1 p STARTED
logstash-2019.10.04 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.04 2 p STARTED
logstash-2019.10.04 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 2 p STARTED
logstash-2019.09.26 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 0 p STARTED
logstash-2019.09.26 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 3 p STARTED
logstash-2019.09.26 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 1 p STARTED
logstash-2019.09.26 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.26 4 p STARTED
logstash-2019.09.26 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 4 p STARTED
logstash-2019.10.01 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 0 p STARTED
logstash-2019.10.01 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 3 p INITIALIZING ALLOCATION_FAILED
logstash-2019.10.01 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 1 p STARTED
logstash-2019.10.01 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.01 2 p STARTED
logstash-2019.10.01 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 2 p STARTED
logstash-2019.09.27 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 0 p STARTED
logstash-2019.09.27 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 3 p STARTED
logstash-2019.09.27 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 1 p STARTED
logstash-2019.09.27 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.27 4 p STARTED
logstash-2019.09.27 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 4 p STARTED
logstash-2019.10.02 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 0 p INITIALIZING ALLOCATION_FAILED
logstash-2019.10.02 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 3 p STARTED
logstash-2019.10.02 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 1 p STARTED
logstash-2019.10.02 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.02 2 p STARTED
logstash-2019.10.02 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 2 p INITIALIZING ALLOCATION_FAILED
logstash-2019.09.28 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 0 p STARTED
logstash-2019.09.28 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 3 p STARTED
logstash-2019.09.28 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 1 p STARTED
logstash-2019.09.28 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.28 4 p STARTED
logstash-2019.09.28 4 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 2 p STARTED
kibana-int 2 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 0 p STARTED
kibana-int 0 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 3 p STARTED
kibana-int 3 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 1 p STARTED
kibana-int 1 r UNASSIGNED CLUSTER_RECOVERED
kibana-int 4 p STARTED
kibana-int 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.10.07 2 p STARTED
logstash-2019.10.07 2 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 0 p STARTED
logstash-2019.10.07 0 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 3 p STARTED
logstash-2019.10.07 3 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 1 p STARTED
logstash-2019.10.07 1 r UNASSIGNED INDEX_CREATED
logstash-2019.10.07 4 p STARTED
logstash-2019.10.07 4 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 2 p STARTED
logstash-2019.10.05 2 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 0 p STARTED
logstash-2019.10.05 0 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 3 p STARTED
logstash-2019.10.05 3 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 1 p STARTED
logstash-2019.10.05 1 r UNASSIGNED INDEX_CREATED
logstash-2019.10.05 4 p STARTED
logstash-2019.10.05 4 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 2 p STARTED
logstash-2019.10.06 2 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 0 p STARTED
logstash-2019.10.06 0 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 3 p STARTED
logstash-2019.10.06 3 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 1 p STARTED
logstash-2019.10.06 1 r UNASSIGNED INDEX_CREATED
logstash-2019.10.06 4 p STARTED
logstash-2019.10.06 4 r UNASSIGNED INDEX_CREATED
logstash-2019.09.30 2 p STARTED
logstash-2019.09.30 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 0 p STARTED
logstash-2019.09.30 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 3 p STARTED
logstash-2019.09.30 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 1 p STARTED
logstash-2019.09.30 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.30 4 p STARTED
logstash-2019.09.30 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 2 p STARTED
logstash-2019.09.29 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 0 p STARTED
logstash-2019.09.29 0 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 3 p STARTED
logstash-2019.09.29 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 1 p STARTED
logstash-2019.09.29 1 r UNASSIGNED CLUSTER_RECOVERED
logstash-2019.09.29 4 p STARTED
logstash-2019.09.29 4 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver 0 p STARTED
nagioslogserver 0 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 2 p STARTED
nagioslogserver_log 2 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 0 p STARTED
nagioslogserver_log 0 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 3 p STARTED
nagioslogserver_log 3 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 1 p STARTED
nagioslogserver_log 1 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_log 4 p STARTED
nagioslogserver_log 4 r UNASSIGNED CLUSTER_RECOVERED
nagioslogserver_history 0 p STARTED
nagioslogserver_history 0 r UNASSIGNED CLUSTER_RECOVERED

Re: ElasticSearch red status after disk full

Posted: Mon Oct 07, 2019 10:48 am
by scottwilkerson
Here are the 3 you are having problems with

Code: Select all

logstash-2019.10.01 3 p INITIALIZING ALLOCATION_FAILED
logstash-2019.10.02 0 p INITIALIZING ALLOCATION_FAILED
logstash-2019.09.28 2 p INITIALIZING ALLOCATION_FAILED
How much free space is available?

Do you have snapshots of these 3 indexes?
https://assets.nagios.com/downloads/nag ... enance.pdf

Re: ElasticSearch red status after disk full

Posted: Mon Oct 07, 2019 11:04 am
by chud
scottwilkerson wrote:Here are the 3 you are having problems with

Code: Select all

logstash-2019.10.01 3 p INITIALIZING ALLOCATION_FAILED
logstash-2019.10.02 0 p INITIALIZING ALLOCATION_FAILED
logstash-2019.09.28 2 p INITIALIZING ALLOCATION_FAILED
Thanks.
scottwilkerson wrote:How much free space is available?
216G
scottwilkerson wrote:Do you have snapshots of these 3 indexes?
Unfortunately, no. I didn't enable snapshots until recently.

Re: ElasticSearch red status after disk full

Posted: Mon Oct 07, 2019 11:13 am
by scottwilkerson
Unfortunately you likely lost some of the data in each of those indexes. You can close these indexes in Admin -> Index status and that should get you back to Yellow status (cannot get green until you add another Instance to the cluster as there is no replication)

Code: Select all

logstash-2019.09.28
logstash-2019.10.01
logstash-2019.10.02