Page 2 of 5
Re: Nagios Log Server - Backup and Maintenance
Posted: Mon Dec 15, 2014 2:57 pm
by roddergreg
Code: Select all
curl -XGET 'http://localhost:9200/nagioslogserver/commands/_search?pretty&q=command:do_maintenance'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 5.2696977,
"hits" : [ {
"_index" : "nagioslogserver",
"_type" : "commands",
"_id" : "7CefFIprSgmAv0HEM0dxPg",
"_score" : 5.2696977,
"_source":{"created":"2014-10-28 11:46:59","active":1,"status":"waiting","type":"system","node":"global","command":"do_maintenance","run_time":1418745383,"frequency":"86400","last_run_output":"Maintenance and Backup jobs are being executed","last_run_time":"2014-12-15 10:56:23","last_run_status":"SUCCESS"}
} ]
}
}
Re: Nagios Log Server - Backup and Maintenance
Posted: Mon Dec 15, 2014 3:04 pm
by roddergreg
cluster-status-screenshot.png
Re: Nagios Log Server - Backup and Maintenance
Posted: Mon Dec 15, 2014 4:03 pm
by jomann
So the maintenance and backup is done using an addon for elasticsearch called curator, which is written in python. The command that it runs (or would run on your system based on your configs) to close indexes is as follows:
If you run this command in the terminal and give us the output:
Code: Select all
su nagios -c "curator close --older-than 5"
We should be able to see if there is a problem with curator running the close command on old indecies.
Re: Nagios Log Server - Backup and Maintenance
Posted: Mon Dec 15, 2014 4:30 pm
by roddergreg
Code: Select all
su - nagios -c "curator close --older-than 5"
Traceback (most recent call last):
File "/usr/bin/curator", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in <module>
working_set.require(__requires__)
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: elasticsearch>=1.0.0,<2.0.0
Re: Nagios Log Server - Backup and Maintenance
Posted: Mon Dec 15, 2014 4:51 pm
by jomann
Well we know what the problem is now. Interesting issue though - we install curator 1.2.2 ... and it should not have issues with the installed version of Elasticsearch. I have a few more things to run that might be able to help us pinpoint why it's not working.
Shows us the elasticsearch version:
Code: Select all
/usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v
Shows us the elasticsearch curator version:
And finally python version:
A possible fix would be to ensure that the installed elasticsearch python module and the curator module are at the correct versions. You can reinstall the curator module by running these 3 commands in order:
Code: Select all
pip uninstall elasticsearch-curator -y
pip uninstall elasticsearch -y
pip install elasticsearch-curator==1.2.2
Re: Nagios Log Server - Backup and Maintenance
Posted: Mon Dec 15, 2014 5:00 pm
by roddergreg
Code: Select all
/usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v
Version: 1.3.2, Build: dee175d/2014-08-13T14:29:30Z, JVM: 1.7.0_65
Code: Select all
curator -v
Traceback (most recent call last):
File "/usr/bin/curator", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in <module>
working_set.require(__requires__)
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: elasticsearch>=1.0.0,<2.0.0
Code: Select all
python -v
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /usr/lib64/python2.6/site.pyc matches /usr/lib64/python2.6/site.py
import site # precompiled from /usr/lib64/python2.6/site.pyc
# /usr/lib64/python2.6/os.pyc matches /usr/lib64/python2.6/os.py
import os # precompiled from /usr/lib64/python2.6/os.pyc
import errno # builtin
import posix # builtin
# /usr/lib64/python2.6/posixpath.pyc matches /usr/lib64/python2.6/posixpath.py
import posixpath # precompiled from /usr/lib64/python2.6/posixpath.pyc
# /usr/lib64/python2.6/stat.pyc matches /usr/lib64/python2.6/stat.py
import stat # precompiled from /usr/lib64/python2.6/stat.pyc
# /usr/lib64/python2.6/genericpath.pyc matches /usr/lib64/python2.6/genericpath.py
import genericpath # precompiled from /usr/lib64/python2.6/genericpath.pyc
# /usr/lib64/python2.6/warnings.pyc matches /usr/lib64/python2.6/warnings.py
import warnings # precompiled from /usr/lib64/python2.6/warnings.pyc
# /usr/lib64/python2.6/linecache.pyc matches /usr/lib64/python2.6/linecache.py
import linecache # precompiled from /usr/lib64/python2.6/linecache.pyc
# /usr/lib64/python2.6/types.pyc matches /usr/lib64/python2.6/types.py
import types # precompiled from /usr/lib64/python2.6/types.pyc
# /usr/lib64/python2.6/UserDict.pyc matches /usr/lib64/python2.6/UserDict.py
import UserDict # precompiled from /usr/lib64/python2.6/UserDict.pyc
# /usr/lib64/python2.6/_abcoll.pyc matches /usr/lib64/python2.6/_abcoll.py
import _abcoll # precompiled from /usr/lib64/python2.6/_abcoll.pyc
# /usr/lib64/python2.6/abc.pyc matches /usr/lib64/python2.6/abc.py
import abc # precompiled from /usr/lib64/python2.6/abc.pyc
# /usr/lib64/python2.6/copy_reg.pyc matches /usr/lib64/python2.6/copy_reg.py
import copy_reg # precompiled from /usr/lib64/python2.6/copy_reg.pyc
# /usr/lib64/python2.6/site-packages/abrt_exception_handler.pyc matches /usr/lib64/python2.6/site-packages/abrt_exception_handler.py
import abrt_exception_handler # precompiled from /usr/lib64/python2.6/site-packages/abrt_exception_handler.pyc
import encodings # directory /usr/lib64/python2.6/encodings
# /usr/lib64/python2.6/encodings/__init__.pyc matches /usr/lib64/python2.6/encodings/__init__.py
import encodings # precompiled from /usr/lib64/python2.6/encodings/__init__.pyc
# /usr/lib64/python2.6/codecs.pyc matches /usr/lib64/python2.6/codecs.py
import codecs # precompiled from /usr/lib64/python2.6/codecs.pyc
import _codecs # builtin
# /usr/lib64/python2.6/encodings/aliases.pyc matches /usr/lib64/python2.6/encodings/aliases.py
import encodings.aliases # precompiled from /usr/lib64/python2.6/encodings/aliases.pyc
# /usr/lib64/python2.6/encodings/utf_8.pyc matches /usr/lib64/python2.6/encodings/utf_8.py
import encodings.utf_8 # precompiled from /usr/lib64/python2.6/encodings/utf_8.pyc
Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
dlopen("/usr/lib64/python2.6/lib-dynload/readline.so", 2);
import readline # dynamically loaded from /usr/lib64/python2.6/lib-dynload/readline.so
Re: Nagios Log Server - Backup and Maintenance
Posted: Mon Dec 15, 2014 5:01 pm
by jomann
Based on curator not even able to give it's version, I'd do a complete re-install of it:
Code: Select all
pip uninstall elasticsearch-curator -y
pip uninstall elasticsearch -y
pip install elasticsearch-curator==1.2.2
Re: Nagios Log Server - Backup and Maintenance
Posted: Mon Dec 15, 2014 5:04 pm
by roddergreg
I specified the full path to the python script it ran.
Code: Select all
su nagios -c "python /usr/lib/python2.6/site-packages/curator/curator.py close --older-than 5"
2014-12-15 17:03:43,644 INFO Job starting...
2014-12-15 17:03:43,648 INFO Beginning CLOSE operations...
2014-12-15 17:03:43,653 INFO Attempting to close index logstash-2014.12.02.
2014-12-15 17:03:43,658 INFO Skipping index logstash-2014.12.02: Already closed.
2014-12-15 17:03:43,658 INFO Attempting to close index logstash-2014.12.03.
2014-12-15 17:03:43,662 INFO Skipping index logstash-2014.12.03: Already closed.
2014-12-15 17:03:43,662 INFO Attempting to close index logstash-2014.12.04.
2014-12-15 17:03:43,666 INFO Skipping index logstash-2014.12.04: Already closed.
2014-12-15 17:03:43,666 INFO Attempting to close index logstash-2014.12.05.
2014-12-15 17:03:43,671 INFO Skipping index logstash-2014.12.05: Already closed.
2014-12-15 17:03:43,671 INFO Attempting to close index logstash-2014.12.06.
2014-12-15 17:03:43,677 INFO Skipping index logstash-2014.12.06: Already closed.
2014-12-15 17:03:43,677 INFO Attempting to close index logstash-2014.12.07.
2014-12-15 17:03:43,681 INFO Skipping index logstash-2014.12.07: Already closed.
2014-12-15 17:03:43,681 INFO Attempting to close index logstash-2014.12.08.
2014-12-15 17:03:43,685 INFO Skipping index logstash-2014.12.08: Already closed.
2014-12-15 17:03:43,685 INFO Attempting to close index logstash-2014.12.09.
2014-12-15 17:03:43,690 INFO Skipping index logstash-2014.12.09: Already closed.
2014-12-15 17:03:43,690 INFO Attempting to close index logstash-2014.12.10.
2014-12-15 17:03:43,695 INFO Skipping index logstash-2014.12.10: Already closed.
2014-12-15 17:03:43,696 INFO logstash-2014.12.11 is within the threshold period (5 days).
2014-12-15 17:03:43,696 INFO logstash-2014.12.12 is within the threshold period (5 days).
2014-12-15 17:03:43,696 INFO logstash-2014.12.13 is within the threshold period (5 days).
2014-12-15 17:03:43,696 INFO logstash-2014.12.14 is within the threshold period (5 days).
2014-12-15 17:03:43,696 INFO logstash-2014.12.15 is within the threshold period (5 days).
2014-12-15 17:03:43,696 INFO CLOSE index operations completed.
2014-12-15 17:03:43,696 INFO Done in 0:00:00.061628.
Re: Nagios Log Server - Backup and Maintenance
Posted: Mon Dec 15, 2014 5:15 pm
by jomann
Are there multiple versions of python on here? Is it possible the regular command is running in a different version of python that is causing the issue? Because our script runs exactly the command I gave you before. If it needs the full path on your system to run correctly then that is why the maintenance is not actually doing what it is supposed to. It's throwing that distribution not found error.
Re: Nagios Log Server - Backup and Maintenance
Posted: Tue Dec 16, 2014 9:34 am
by roddergreg
Just one version on python installed on the server python 2.6 located in /usr/lib/python2.6. The symbolic link for the python binary points here.