We are getting "ClusterBlockException[blocked" in logstash.

[devops-india]

Hi Team,

From the past few days we are observing "ClusterBlockException[blocked" logs in logstash logs intermittently. Before receiving this logs , few shards of NLS cluster gets in UNASSIGNED or INITIALIZING state. When we checked elasticsearch logs at the same time we found below logs over there. When we received this strings in logs we restart the logstash and elasticsearch services on each NLS node in cluster. After restart cluster takes few time to assigned shards and start working properly.


logs string from logstash logs:

message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?", :error_message=>"[503] {\"error\":\"ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/2/no master]


logs string from elasticsearch:

[2020-12-27 22:27:03,250][INFO ][discovery.zen ] [9cb943f4-0239-415c-8529-5cb81fc735ab] master_left [[9fa4ac7d-d127-48aa-bf17-4e83b583b770][6-VRmeBeSFO3noB5WMV1Iw][nls02][inet[/10.27.18.49:9300]]{max_local_storage_nodes=1}], reason [do not exists on master, act as master failure]

Can you please help us to find the root cause of master failure ?

[vtrac]

Hi devops-india,
Looks like the Elasticsearch zen discovery process has failed to find its mater "10.27.18.49:9300".

I have talked to my teammates and was suggested that this might be related to network issue where logstash was not able to communicates with Elasticsearch.

A restart of Elasticsearch and logstash will help fixed the issue.

Code: Select all

# systemctl restart elasticsearch.service
# systemctl restart logstash.service

Please check with your network admin and see if there were network issue at the time of failure.

Regards,
Vinh

[devops-india]

Hi Team,


Thanks for looking in to this, But as mentioned in earlier email we are facing this issue frequently (twice a week). We are already restarting Logstash and Elasticsearch services whenever we faced this issue. Is there any way to permanently fix this issue?


Regards,
Akshay Pandhare,
Devops Team.

[vtrac]

Hi Akshay,
Couple times a week is quite often. Can you please check with your network team? .... are they doing network maintenance weekly?

Could you please upload more logs and the "elasticsearch.yml" file?

Code: Select all

/usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml

Also, please upload the results of the below commands:

Code: Select all

# curl -XGET 'http://localhost:9200'

# curl http://localhost:9200/_cluster/health?pretty

# curl -XGET 'http://localhost:9200/_cat/{nodes,shards,indices}?v'

Best Regards,
Vinh