Nagios Support Forum

Posted: **Thu Nov 17, 2016 11:58 am**

Dear support,

Could you please help me what i can do in the following situation?

indices.JPG

I can not restore any index due to this message.

Thanks in advance!

Regards,
Zoltan

Posted: **Thu Nov 17, 2016 12:06 pm**

From the Administration -> Index Status page, you will need to be sure that the logstash-2016.09.28 index (or any other index you wish to restore) is closed before you restore it from a snapshot. You can do this by clicking the little "close" button next to the table entry of this index.

2016_11_17_11_06_04_Index_Status_Nagios_Log_Server.png

Posted: **Thu Nov 17, 2016 12:47 pm**

it's already deleted.

indices_2.JPG

Posted: **Thu Nov 17, 2016 1:23 pm**

What version of Nagios Log Server is this machine using? I was unable to replicate this issue against 1.4.4. There may be some commands we need to run specific to your version.

Posted: **Thu Nov 17, 2016 3:05 pm**

1.4.2

Posted: **Thu Nov 17, 2016 3:58 pm**

Hmm, I wasn't able to replicate this on NLS 1.4.2 either.

Share the output of the following commands executed from the CLI of your Nagios Log Server machine:

Code: Select all

curl 'localhost:9200/_snapshot/'
curl 'localhost:9200/_cat/indices?v'

Could you also tell me, from the GUI, what the date/time of the snapshot is that you are trying to restore logstash-2016.09.28 from? Could you also take a screenshot of the entire Backup & Maintenance page and share it here?

Posted: **Fri Nov 18, 2016 5:39 am**

Meanwhile in the night.

It seems according to log restoration logstash had been crashed with JAVA HEAP error, disk use went over 90%(shards will be relocated away from this node messages appeared in log). Cluster was in red state.
according to a knowledge base https://support.nagios.com/kb/article.php?id=90 i increased LS_HEAP_SIZE to "1024m" restart both nodes and delete all index before Oct28.
Now we are in yellow state:

Code: Select all

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "cfd94ef4-d466-4f3b-815b-0333a7b69252",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 121,
  "active_shards" : 187,
  "relocating_shards" : 0,
  "initializing_shards" : 2,
  "unassigned_shards" : 53,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}

Unassigned shareds is decreasing, so i hope we will in green state in a while.

Code: Select all

curl 'localhost:9200/_snapshot/'
{"lvpapp-NFS":{"type":"fs","settings":{"compress":"true","location":"/mnt/logstore"}}}

Code: Select all

curl 'localhost:9200/_cat/indices?v'
health status index               pri rep docs.count docs.deleted store.size pri.store.size
green  open   kibana-int            5   1         46            5    776.5kb        388.3kb
green  open   logstash-2016.10.28   5   1    7577989            0      7.6gb          3.8gb
yellow open   logstash-2016.11.16   5   1    7682792            0      3.8gb          3.8gb
green  open   logstash-2016.10.31   5   1    2203724            0      2.2gb          1.1gb
green  open   logstash-2016.11.18   5   1     871083            0   1009.4mb        506.6mb
green  open   logstash-2016.11.07   5   1    8025113            0        7gb          3.5gb
yellow open   logstash-2016.11.11   5   1    8335885            0        4gb            4gb
yellow open   nagioslogserver       1   1        723            3    390.9kb        390.9kb
green  open   logstash-2016.11.05   5   1    2427664            0      2.5gb          1.2gb
yellow open   logstash-2016.11.10   5   1    7310446            0      3.6gb          3.6gb
yellow open   logstash-2016.11.09   5   1    7225897            0      5.2gb          3.7gb
green  open   logstash-2016.11.08   5   1    6647047            0      6.7gb          3.3gb
green  open   logstash-2016.11.03   5   1    7119414            0        7gb          3.5gb
green  open   logstash-2016.11.06   5   1    2793149            0      2.6gb          1.3gb
green  open   logstash-2016.11.04   5   1    8102154            0        8gb            4gb
green  open   logstash-2016.11.01   5   1    2634621            0      2.7gb          1.3gb
green  open   logstash-2016.10.29   5   1    2091491            0      2.2gb          1.1gb
yellow open   logstash-2016.11.13   5   1    2389765            0      1.2gb          1.2gb
yellow open   logstash-2016.11.14   5   1    6852343            0        4gb          3.3gb
yellow open   logstash-2016.11.17   5   1    7160524            0      3.5gb          3.5gb
green  open   nagioslogserver_log   5   1    5576505            0      1.1gb        598.5mb
green  open   logstash-2016.10.30   5   1    1978619            0      1.9gb       1006.5mb
yellow open   logstash-2016.11.12   5   1    2323756            0      1.2gb          1.2gb
green  open   logstash-2016.11.02   5   1    6570303            0      6.6gb          3.3gb
yellow open   logstash-2016.11.15   5   1    7335914            0      3.6gb          3.6gb

So i guess that it was the root cause of all of our problems. I will increase disk space as well and try again and inform you.
Thank you for your support!

But i have an important question now:

Did we lost the logs during the time period when logstash was crashed, or is it possible somehow to get it from somewhere?

Best regards,
Zoltan

settings.JPG

to_restore.JPG

Posted: **Fri Nov 18, 2016 10:55 am**

patalenszki.zoltan wrote: Did we lost the logs during the time period when logstash was crashed, or is it possible somehow to get it from somewhere?

There are a lot of factors that could influence whether or not messages were lost (the time logstash was down, total traffic, agent configurations, etc). It's impossible for us to say for certain unfortunately, though it is a possibility.

Nagios Support Forum

Indices to restore

Indices to restore

Re: Indices to restore

Re: Indices to restore

Re: Indices to restore

Re: Indices to restore

Re: Indices to restore

Re: Indices to restore

Re: Indices to restore