Maintenance doesn't seem to run

[ljorg]

Hi,

Ive set maintenance to close any index older than seven days, but it doesn't do anything. I have to close indexes manually. It appears that with more than 130 million records (or so) the performance of Nagios Log Server crawls to a stop. Adding extra memory or cores doesn't seem to help, but closing old indexes helps tremendously.

My settings are shown in the attachment below

Clipboard01.jpg

[teirekos]

Have you tried to configure the heap memory? Check the link below:
http://www.elasticsearch.org/guide/en/e ... izing.html

[scottwilkerson]

I agree with @teirekos about the heap memory.

Also can you run the following command and return the output

Code: Select all

curl -XGET "http://localhost:9200/nagioslogserver/commands/_search?pretty&q=command:do_maintenance"

[ljorg]

I agree with @teirekos about the heap memory.

I'll get on top of that soon.

Also can you run the following command and return the output

This is the output:

Code: Select all

{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 4.749504,
    "hits" : [ {
      "_index" : "nagioslogserver",
      "_type" : "commands",
      "_id" : "ZM65w-m1QFy9is8Ebkdw0w",
      "_score" : 4.749504,
      "_source":{"created":"2014-12-09 15:02:37","active":1,"status":"waiting","type":"system","node":"global","command":"do_maintenance","run_time":1423749939,"frequency":"86400","last_run_output":"Maintenance and Backup jobs are being executed","last_run_time":"2015-02-11 15:05:39","last_run_status":"SUCCESS"}
    } ]
  }
}

[scottwilkerson]

It appears to be running, it was last executed on "2015-02-11 15:05:39"

One thing to note, on some systems we have noticed a bug releated to the addon that processes these. You can determine if you are affected by these by running the following from the command line.

Code: Select all

curator

this bug will be resolved in the next release due out soon.

[ljorg]

The output of curator:

Code: Select all

Traceback (most recent call last):
  File "/usr/bin/curator", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: elasticsearch>=1.0.0,<2.0.0

Does that mean I'm affected and is that why maintanence doesn't work although it runs?


Lars

[tmcdonald]

How did you install this server? One of our pre-built VMs, or did you use fullinstall on an existing server? It almost looks like python was already installed. What version is it?

Code: Select all

python --version

[ljorg]

It's installed on a RedHat Enterprise Linux 6. Python is probably installed by default. It's version 2.2.6.


Lars

[scottwilkerson]

Nagios Log Server 1.3 came out yesterday which has a fix for this problem.

http://library.nagios.com/library/produ ... /downloads

[teirekos]

I've installed NLS 1.3 I left it all the weekend and still no snapshots created.