Empty dashboard but logstash is collecting logs

[uma K]

Dashboard is empty. However I am able to see logstash is collecting the logs but failed to display on the front end UI.
Please assist on this.

[npolovenko]

Hello, @uma K. Please remove all pinned queries and all filters and let me know if dashlets populate with data.
If not, please put the profile.sh script into the /tmp folder in the log server and execute it. That should generate a system profile archive in the same folder that you can upload here.

[uma K]

Hello, Now my dashboard is displaying data.

Increased fielddata size to 70%.

indices.breaker.fielddata.limit : 70%

Could you please help me in why this is required and causes behind this??

[uma K]

Hello team, My elasticsearch has stopped working again.
Please help on this

[root@X1LOGW01 _state]# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"error" : "MasterNotDiscoveredException[waited for [30s]]",
"status" : 503
}


Logs:
[2018-11-15 10:18:16,844][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
[2018-11-15 10:18:20,068][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
[2018-11-15 10:18:23,291][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
[2018-11-15 10:18:26,519][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
[2018-11-15 10:18:29,740][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times

[uma K]

I have attached my elasticsearch.yml

[uma K]

I have attached elasticsearch.yml herewith

[npolovenko]

@uma K, Please generate and send me a profile from each log server in the cluster. A profile can be generated under Admin > System > System Status or in the command line by running:

/usr/local/nagioslogserver/scripts/profile.sh

The profile can be found at /tmp/system-profile.tar.gz.

After you generate profiles please run the following command:

service elasticsearch restart

And if it crashes please generate another profile and send it in as well.

[uma K]

Hello,
I have attached system profile from all 4 instances.

Below is the configuration change I made in elasticsearch.yml

discovery.zen.ping.unicast.hosts: ["IP address of node2","IP address of node 3"]

Updated network.host in elasticsearch.yml as below
network.host: IP address of node1

same has been done in all 4 nodes.

Below is the exception in elasticsearch


[2018-11-15 13:32:46,831][DEBUG][action.bulk ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2018-11-15 13:32:46,831][DEBUG][action.bulk ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]



Below is the error in logstash and unable to see logs in dashboard

{:timestamp=>"2018-11-15T12:40:50.668000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:52.683000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:54.702000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:56.716000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:58.905000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:41:00.337000-0800", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2018-11-15T12:41:00.339000-0800", :message=>"stopping pipeline", :id=>"main"}

[uma K]

Can we have a webex or screen haring session??
None of our logs are showing up in logs anymore.

Can you please assist on this?

[npolovenko]

@uma K, You should revert the changes you made to the elasticsearch.yml file:
network.host should not point to node 1
The line should be commented out.
Once you fix that on all instances please run the following command on each log server:

service elasticsearch restart