Dashboard is empty. However I am able to see logstash is collecting the logs but failed to display on the front end UI.
Please assist on this.
Empty dashboard but logstash is collecting logs
Empty dashboard but logstash is collecting logs
You do not have the required permissions to view the files attached to this post.
-
- Support Tech
- Posts: 3457
- Joined: Mon May 15, 2017 5:00 pm
Re: Empty dashboard but logstash is collecting logs
Hello, @uma K. Please remove all pinned queries and all filters and let me know if dashlets populate with data.
If not, please put the profile.sh script into the /tmp folder in the log server and execute it. That should generate a system profile archive in the same folder that you can upload here.
If not, please put the profile.sh script into the /tmp folder in the log server and execute it. That should generate a system profile archive in the same folder that you can upload here.
You do not have the required permissions to view the files attached to this post.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: Empty dashboard but logstash is collecting logs
Hello, Now my dashboard is displaying data.
Increased fielddata size to 70%.
indices.breaker.fielddata.limit : 70%
Could you please help me in why this is required and causes behind this??
Increased fielddata size to 70%.
indices.breaker.fielddata.limit : 70%
Could you please help me in why this is required and causes behind this??
Re: Empty dashboard but logstash is collecting logs
Hello team, My elasticsearch has stopped working again.
Please help on this
[root@X1LOGW01 _state]# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"error" : "MasterNotDiscoveredException[waited for [30s]]",
"status" : 503
}
Logs:
[2018-11-15 10:18:16,844][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
[2018-11-15 10:18:20,068][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
[2018-11-15 10:18:23,291][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
[2018-11-15 10:18:26,519][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
[2018-11-15 10:18:29,740][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
Please help on this
[root@X1LOGW01 _state]# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"error" : "MasterNotDiscoveredException[waited for [30s]]",
"status" : 503
}
Logs:
[2018-11-15 10:18:16,844][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
[2018-11-15 10:18:20,068][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
[2018-11-15 10:18:23,291][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
[2018-11-15 10:18:26,519][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
[2018-11-15 10:18:29,740][INFO ][discovery.zen ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] failed to send join request to master [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][Qbxi8ooQRi2MwZ8fAURobA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[abd0aca5-8cbf-4f11-988e-be0d778f5f95][inet[/136.133.236.12:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}] not master for join request from [[abd0aca5-8cbf-4f11-988e-be0d778f5f95][246K8e7AQK2rRLyg_3SCvA][X1LOGW01.mnao.net][inet[/136.133.236.12:9300]]{max_local_storage_nodes=1, master=true}]]; ], tried [3] times
Re: Empty dashboard but logstash is collecting logs
I have attached my elasticsearch.yml
Re: Empty dashboard but logstash is collecting logs
I have attached elasticsearch.yml herewith
You do not have the required permissions to view the files attached to this post.
-
- Support Tech
- Posts: 3457
- Joined: Mon May 15, 2017 5:00 pm
Re: Empty dashboard but logstash is collecting logs
@uma K, Please generate and send me a profile from each log server in the cluster. A profile can be generated under Admin > System > System Status or in the command line by running:
After you generate profiles please run the following command:
The profile can be found at /tmp/system-profile.tar.gz./usr/local/nagioslogserver/scripts/profile.sh
After you generate profiles please run the following command:
And if it crashes please generate another profile and send it in as well.service elasticsearch restart
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: Empty dashboard but logstash is collecting logs
Hello,
I have attached system profile from all 4 instances.
Below is the configuration change I made in elasticsearch.yml
discovery.zen.ping.unicast.hosts: ["IP address of node2","IP address of node 3"]
Updated network.host in elasticsearch.yml as below
network.host: IP address of node1
same has been done in all 4 nodes.
Below is the exception in elasticsearch
[2018-11-15 13:32:46,831][DEBUG][action.bulk ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2018-11-15 13:32:46,831][DEBUG][action.bulk ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
Below is the error in logstash and unable to see logs in dashboard
{:timestamp=>"2018-11-15T12:40:50.668000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:52.683000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:54.702000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:56.716000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:58.905000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:41:00.337000-0800", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2018-11-15T12:41:00.339000-0800", :message=>"stopping pipeline", :id=>"main"}
I have attached system profile from all 4 instances.
Below is the configuration change I made in elasticsearch.yml
discovery.zen.ping.unicast.hosts: ["IP address of node2","IP address of node 3"]
Updated network.host in elasticsearch.yml as below
network.host: IP address of node1
same has been done in all 4 nodes.
Below is the exception in elasticsearch
[2018-11-15 13:32:46,831][DEBUG][action.bulk ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2018-11-15 13:32:46,831][DEBUG][action.bulk ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
Below is the error in logstash and unable to see logs in dashboard
{:timestamp=>"2018-11-15T12:40:50.668000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:52.683000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:54.702000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:56.716000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:40:58.905000-0800", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://127.0.0.1:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-11-15T12:41:00.337000-0800", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2018-11-15T12:41:00.339000-0800", :message=>"stopping pipeline", :id=>"main"}
You do not have the required permissions to view the files attached to this post.
Re: Empty dashboard but logstash is collecting logs
Can we have a webex or screen haring session??
None of our logs are showing up in logs anymore.
Can you please assist on this?
None of our logs are showing up in logs anymore.
Can you please assist on this?
-
- Support Tech
- Posts: 3457
- Joined: Mon May 15, 2017 5:00 pm
Re: Empty dashboard but logstash is collecting logs
@uma K, You should revert the changes you made to the elasticsearch.yml file:
network.host should not point to node 1
The line should be commented out.
Once you fix that on all instances please run the following command on each log server:
network.host should not point to node 1
The line should be commented out.
Once you fix that on all instances please run the following command on each log server:
service elasticsearch restart
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.