Nagios Support Forum

Posted: **Mon Dec 28, 2020 7:41 am**

Hi Team,

From the past few days we are observing "ClusterBlockException[blocked" logs in logstash logs intermittently. Before receiving this logs , few shards of NLS cluster gets in UNASSIGNED or INITIALIZING state. When we checked elasticsearch logs at the same time we found below logs over there. When we received this strings in logs we restart the logstash and elasticsearch services on each NLS node in cluster. After restart cluster takes few time to assigned shards and start working properly.

logs string from logstash logs:

message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?", :error_message=>"[503] {\"error\":\"ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/2/no master]

logs string from elasticsearch:

[2020-12-27 22:27:03,250][INFO ][discovery.zen ] [9cb943f4-0239-415c-8529-5cb81fc735ab] master_left [[9fa4ac7d-d127-48aa-bf17-4e83b583b770][6-VRmeBeSFO3noB5WMV1Iw][nls02][inet[/10.27.18.49:9300]]{max_local_storage_nodes=1}], reason [do not exists on master, act as master failure]

Can you please help us to find the root cause of master failure ?

Posted: **Tue Dec 29, 2020 12:16 pm**

Hi devops-india,
Looks like the Elasticsearch zen discovery process has failed to find its mater "10.27.18.49:9300".

I have talked to my teammates and was suggested that this might be related to network issue where logstash was not able to communicates with Elasticsearch.

A restart of Elasticsearch and logstash will help fixed the issue.

Code: Select all

# systemctl restart elasticsearch.service
# systemctl restart logstash.service

Please check with your network admin and see if there were network issue at the time of failure.

Regards,
Vinh

Posted: **Thu Dec 31, 2020 2:18 am**

Hi Team,

Thanks for looking in to this, But as mentioned in earlier email we are facing this issue frequently (twice a week). We are already restarting Logstash and Elasticsearch services whenever we faced this issue. Is there any way to permanently fix this issue?

Regards,
Akshay Pandhare,
Devops Team.

Posted: **Mon Jan 04, 2021 12:50 pm**

Hi Akshay,
Couple times a week is quite often. Can you please check with your network team? .... are they doing network maintenance weekly?

Could you please upload more logs and the "elasticsearch.yml" file?

Code: Select all

/usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml

Also, please upload the results of the below commands:

Code: Select all

# curl -XGET 'http://localhost:9200'

# curl http://localhost:9200/_cluster/health?pretty

# curl -XGET 'http://localhost:9200/_cat/{nodes,shards,indices}?v'

Best Regards,
Vinh

Nagios Support Forum

We are getting "ClusterBlockException[blocked" in logstash.

We are getting "ClusterBlockException[blocked" in logstash.

Re: We are getting "ClusterBlockException[blocked" in logsta

Re: We are getting "ClusterBlockException[blocked" in logsta

Re: We are getting "ClusterBlockException[blocked" in logsta