Nagios Log Server - Backup and Maintenance

[roddergreg]

Code: Select all

curl -XGET 'http://localhost:9200/nagioslogserver/commands/_search?pretty&q=command:do_maintenance'
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 5.2696977,
    "hits" : [ {
      "_index" : "nagioslogserver",
      "_type" : "commands",
      "_id" : "7CefFIprSgmAv0HEM0dxPg",
      "_score" : 5.2696977,
      "_source":{"created":"2014-10-28 11:46:59","active":1,"status":"waiting","type":"system","node":"global","command":"do_maintenance","run_time":1418745383,"frequency":"86400","last_run_output":"Maintenance and Backup jobs are being executed","last_run_time":"2014-12-15 10:56:23","last_run_status":"SUCCESS"}
    } ]
  }
}

[roddergreg]

cluster-status-screenshot.png

[jomann]

So the maintenance and backup is done using an addon for elasticsearch called curator, which is written in python. The command that it runs (or would run on your system based on your configs) to close indexes is as follows:

Code: Select all

curator close --older-than 5

If you run this command in the terminal and give us the output:

Code: Select all

su nagios -c "curator close --older-than 5"

We should be able to see if there is a problem with curator running the close command on old indecies.

[roddergreg]

Code: Select all

 su - nagios -c "curator close --older-than 5"
Traceback (most recent call last):
  File "/usr/bin/curator", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: elasticsearch>=1.0.0,<2.0.0

[jomann]

Well we know what the problem is now. Interesting issue though - we install curator 1.2.2 ... and it should not have issues with the installed version of Elasticsearch. I have a few more things to run that might be able to help us pinpoint why it's not working.

Shows us the elasticsearch version:

Code: Select all

/usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v

Shows us the elasticsearch curator version:

Code: Select all

curator -v

And finally python version:

Code: Select all

python --version

A possible fix would be to ensure that the installed elasticsearch python module and the curator module are at the correct versions. You can reinstall the curator module by running these 3 commands in order:

Code: Select all

pip uninstall elasticsearch-curator -y
pip uninstall elasticsearch -y
pip install elasticsearch-curator==1.2.2

[roddergreg]

Code: Select all

/usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v 
Version: 1.3.2, Build: dee175d/2014-08-13T14:29:30Z, JVM: 1.7.0_65

Code: Select all

curator -v 
Traceback (most recent call last):
  File "/usr/bin/curator", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: elasticsearch>=1.0.0,<2.0.0

Code: Select all

 python -v 
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /usr/lib64/python2.6/site.pyc matches /usr/lib64/python2.6/site.py
import site # precompiled from /usr/lib64/python2.6/site.pyc
# /usr/lib64/python2.6/os.pyc matches /usr/lib64/python2.6/os.py
import os # precompiled from /usr/lib64/python2.6/os.pyc
import errno # builtin
import posix # builtin
# /usr/lib64/python2.6/posixpath.pyc matches /usr/lib64/python2.6/posixpath.py
import posixpath # precompiled from /usr/lib64/python2.6/posixpath.pyc
# /usr/lib64/python2.6/stat.pyc matches /usr/lib64/python2.6/stat.py
import stat # precompiled from /usr/lib64/python2.6/stat.pyc
# /usr/lib64/python2.6/genericpath.pyc matches /usr/lib64/python2.6/genericpath.py
import genericpath # precompiled from /usr/lib64/python2.6/genericpath.pyc
# /usr/lib64/python2.6/warnings.pyc matches /usr/lib64/python2.6/warnings.py
import warnings # precompiled from /usr/lib64/python2.6/warnings.pyc
# /usr/lib64/python2.6/linecache.pyc matches /usr/lib64/python2.6/linecache.py
import linecache # precompiled from /usr/lib64/python2.6/linecache.pyc
# /usr/lib64/python2.6/types.pyc matches /usr/lib64/python2.6/types.py
import types # precompiled from /usr/lib64/python2.6/types.pyc
# /usr/lib64/python2.6/UserDict.pyc matches /usr/lib64/python2.6/UserDict.py
import UserDict # precompiled from /usr/lib64/python2.6/UserDict.pyc
# /usr/lib64/python2.6/_abcoll.pyc matches /usr/lib64/python2.6/_abcoll.py
import _abcoll # precompiled from /usr/lib64/python2.6/_abcoll.pyc
# /usr/lib64/python2.6/abc.pyc matches /usr/lib64/python2.6/abc.py
import abc # precompiled from /usr/lib64/python2.6/abc.pyc
# /usr/lib64/python2.6/copy_reg.pyc matches /usr/lib64/python2.6/copy_reg.py
import copy_reg # precompiled from /usr/lib64/python2.6/copy_reg.pyc
# /usr/lib64/python2.6/site-packages/abrt_exception_handler.pyc matches /usr/lib64/python2.6/site-packages/abrt_exception_handler.py
import abrt_exception_handler # precompiled from /usr/lib64/python2.6/site-packages/abrt_exception_handler.pyc
import encodings # directory /usr/lib64/python2.6/encodings
# /usr/lib64/python2.6/encodings/__init__.pyc matches /usr/lib64/python2.6/encodings/__init__.py
import encodings # precompiled from /usr/lib64/python2.6/encodings/__init__.pyc
# /usr/lib64/python2.6/codecs.pyc matches /usr/lib64/python2.6/codecs.py
import codecs # precompiled from /usr/lib64/python2.6/codecs.pyc
import _codecs # builtin
# /usr/lib64/python2.6/encodings/aliases.pyc matches /usr/lib64/python2.6/encodings/aliases.py
import encodings.aliases # precompiled from /usr/lib64/python2.6/encodings/aliases.pyc
# /usr/lib64/python2.6/encodings/utf_8.pyc matches /usr/lib64/python2.6/encodings/utf_8.py
import encodings.utf_8 # precompiled from /usr/lib64/python2.6/encodings/utf_8.pyc
Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
dlopen("/usr/lib64/python2.6/lib-dynload/readline.so", 2);
import readline # dynamically loaded from /usr/lib64/python2.6/lib-dynload/readline.so

[jomann]

Based on curator not even able to give it's version, I'd do a complete re-install of it:

Code: Select all

pip uninstall elasticsearch-curator -y
pip uninstall elasticsearch -y
pip install elasticsearch-curator==1.2.2

[roddergreg]

I specified the full path to the python script it ran.

Code: Select all

su nagios -c "python /usr/lib/python2.6/site-packages/curator/curator.py close --older-than 5"
2014-12-15 17:03:43,644 INFO      Job starting...
2014-12-15 17:03:43,648 INFO      Beginning CLOSE operations...
2014-12-15 17:03:43,653 INFO      Attempting to close index logstash-2014.12.02.
2014-12-15 17:03:43,658 INFO      Skipping index logstash-2014.12.02: Already closed.
2014-12-15 17:03:43,658 INFO      Attempting to close index logstash-2014.12.03.
2014-12-15 17:03:43,662 INFO      Skipping index logstash-2014.12.03: Already closed.
2014-12-15 17:03:43,662 INFO      Attempting to close index logstash-2014.12.04.
2014-12-15 17:03:43,666 INFO      Skipping index logstash-2014.12.04: Already closed.
2014-12-15 17:03:43,666 INFO      Attempting to close index logstash-2014.12.05.
2014-12-15 17:03:43,671 INFO      Skipping index logstash-2014.12.05: Already closed.
2014-12-15 17:03:43,671 INFO      Attempting to close index logstash-2014.12.06.
2014-12-15 17:03:43,677 INFO      Skipping index logstash-2014.12.06: Already closed.
2014-12-15 17:03:43,677 INFO      Attempting to close index logstash-2014.12.07.
2014-12-15 17:03:43,681 INFO      Skipping index logstash-2014.12.07: Already closed.
2014-12-15 17:03:43,681 INFO      Attempting to close index logstash-2014.12.08.
2014-12-15 17:03:43,685 INFO      Skipping index logstash-2014.12.08: Already closed.
2014-12-15 17:03:43,685 INFO      Attempting to close index logstash-2014.12.09.
2014-12-15 17:03:43,690 INFO      Skipping index logstash-2014.12.09: Already closed.
2014-12-15 17:03:43,690 INFO      Attempting to close index logstash-2014.12.10.
2014-12-15 17:03:43,695 INFO      Skipping index logstash-2014.12.10: Already closed.
2014-12-15 17:03:43,696 INFO      logstash-2014.12.11 is within the threshold period (5 days).
2014-12-15 17:03:43,696 INFO      logstash-2014.12.12 is within the threshold period (5 days).
2014-12-15 17:03:43,696 INFO      logstash-2014.12.13 is within the threshold period (5 days).
2014-12-15 17:03:43,696 INFO      logstash-2014.12.14 is within the threshold period (5 days).
2014-12-15 17:03:43,696 INFO      logstash-2014.12.15 is within the threshold period (5 days).
2014-12-15 17:03:43,696 INFO      CLOSE index operations completed.
2014-12-15 17:03:43,696 INFO      Done in 0:00:00.061628.

[jomann]

Are there multiple versions of python on here? Is it possible the regular command is running in a different version of python that is causing the issue? Because our script runs exactly the command I gave you before. If it needs the full path on your system to run correctly then that is why the maintenance is not actually doing what it is supposed to. It's throwing that distribution not found error.

[roddergreg]

Just one version on python installed on the server python 2.6 located in /usr/lib/python2.6. The symbolic link for the python binary points here.