Hello,
I have attached system profile from all 4 instances.
Below is the configuration change I made in elasticsearch.yml
discovery.zen.ping.unicast.hosts: ["IP address of node2","IP address of node 3"]
Updated network.host in elasticsearch.yml as below
network.host: IP address of node1
same has been done in all 4 nodes.
Below is the exception in elasticsearch
[2018-11-15 13:32:46,831][DEBUG][action.bulk ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2018-11-15 13:32:46,831][DEBUG][action.bulk ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
Below is the error in logstash and unable to see logs in dashboard
{:timestamp=>"2018-11-15T12:40:50.668000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"
http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:52.683000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"
http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:54.702000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"
http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:56.716000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"
http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:58.905000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"
http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:41:00.337000-0800", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2018-11-15T12:41:00.339000-0800", :message=>"stopping pipeline", :id=>"main"}