Unable to open the nagios log server web interface.

[mejokj]

Hello,

After opening some indices we are facing an issue in our Nagios log server. We are not able to open the web interface.

On checking the cluster status it shows below.

[root@localhost]# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "8df86e3a-8739-43ab-8a7f-110b464cae2d",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 503,
"active_shards" : 503,
"relocating_shards" : 0,
"initializing_shards" : 4,
"unassigned_shards" : 750,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 1,
"number_of_in_flight_fetch" : 0
}

On checking the elastic search logs it shows the below.
+++++++++++++++++++++++++++++
2019-08-21 18:23:28,088][DEBUG][action.search.type ] [e8a605f9-0c8e-4f9c-8bac-db39f8cda6d3] All shards failed for phase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null
+++++++++++++++++++++++++++++
We have restarted the service and rebooted the server, but the issue still exists.

Kindly help me to resolve the issue.

[mbellerue]

Let's try and close those indices from the command line.

First, get a list of your indices.

Code: Select all

ls -lh /usr/local/nagioslogserver/elasticsearch/data/<YOUR CLUSTER GUID>/nodes/0/indices/

Files named logstash-YYYY.MM.DD are the indices. If you remember which ones were opened, you can close them by running the following command.

Code: Select all

curl -X POST "localhost:9200/logstash-YYYY.MM.DD/_close?pretty=true"

[mejokj]

Hello,

Thanks for the update. Now the issue has been resolved and I am able to access the web interface and logs are coming.
But in the Home menu, the Number of logs per 15 min Dashboard shows blank. I have attached the screenshot.

[mbellerue]

The first thing I would check there would be that the date/time is still on point for the server. If it gets set to some other timezone so it looks like incoming logs are from the future, that can produce strange effects like this.

Also, I would recommend giving the server a reboot to make sure all of the services come up in proper order now that the previous issue has been resolved.