Snapshots & Maintenance off of a Ext4 Partition?

[tvoll]

Here is the output that gets appended to the logs:

Code: Select all

[tvollscw@logserver ~]$ tail -Fn0 /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command do_maintenance with args ' ' for job id: snapshots_maintenance
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh optimize indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Return: 1
-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh close indices --older-than 2 --time-unit days --timestring %Y.%m.%d
Return: 1
-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete indices --older-than 30 --time-unit days --timestring %Y.%m.%d
Return: 1
-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "Default" --ignore_unavailable indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Return: 1
-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete snapshots --older-than 90 --time-unit days --timestring %Y%m%d --repository "Default"
Return: 1
-----
SUCCESS

On the web interface, it shows success, no errors upon running from the dashboard.

[ssax]

What is the output of this command?

Code: Select all

rpm -qa | grep python-

[tvoll]

The command responds back with the following packages:

python-gobject-base-3.22.0-1.el7_4.1.x86_64
python-libs-2.7.5-86.el7.x86_64
python-decorator-3.4.0-3.el7.noarch
python-javapackages-3.4.1-11.el7.noarch
python-perf-3.10.0-1062.18.1.el7.x86_64
newt-python-0.52.15-4.el7.x86_64
python-slip-dbus-0.4.0-4.el7.noarch
python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch
systemd-python-219-67.el7_7.4.x86_64
python-requests-2.6.0-9.el7_7.noarch
python-lxml-3.2.1-4.el7.x86_64
dbus-python-1.1.1-9.el7.x86_64
python-iniparse-0.4-9.el7.noarch
python-configobj-4.7.2-7.el7.noarch
python-slip-0.4.0-4.el7.noarch
python-urlgrabber-3.10-9.el7.noarch
python-backports-1.0-8.el7.x86_64
python-setuptools-0.9.8-7.el7.noarch
libxml2-python-2.9.1-6.el7_2.3.x86_64
python-babel-0.9.6-8.el7.noarch
python-six-1.9.0-2.el7.noarch
python-markupsafe-0.11-10.el7.x86_64
python-2.7.5-86.el7.x86_64
python-linux-procfs-0.4.11-4.el7.noarch
python-chardet-2.2.1-3.el7.noarch
rpm-python-4.11.3-40.el7.x86_64
python-jinja2-2.7.2-4.el7.noarch
python-kitchen-1.1.1-5.el7.noarch
python-firewall-0.6.3-2.el7_7.4.noarch
python-pyudev-0.15-9.el7.noarch
python-schedutils-0.4-6.el7.x86_64
python-pycurl-7.19.0-19.el7.x86_64
libselinux-python-2.5-14.1.el7.x86_64
python-ipaddress-1.0.16-2.el7.noarch

[ssax]

NOTE: Take a VM snapshot first just in case you need to revert

Try doing this and see if that resolves the issue:

Code: Select all

sudo pip uninstall urllib3 -y
sudo pip uninstall requests -y
sudo yum remove python-urllib3 -y
sudo yum remove python-requests -y
sudo pip install --upgrade urllib3
sudo pip install --upgrade requests

Then test again after restarting logstash/elasticsearch.

[tvoll]

After doing that, rebooting, and running the snapshot_maintenance command while tailing the logs exactly as I did before, i see this.

Running command do_maintenance with args ' ' for job id: snapshots_maintenance
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.

However, nothing new appears on the snapshots menu.

[ssax]

That looks good, are there any files in here at all:

Code: Select all

ls -l /mnt/logs-backup/

Please PM me a fresh copy of your profile, you can download it from Admin > System Status by clicking the Download System Profile button​.

[tvoll]

Code: Select all

[tvollscw@logserver ~]$ ls -l /mnt/logs-backup/
total 24
-rw-r--r-- 1 nagios users   61 May 21 11:31 index
drwxr-xr-x 3 nagios users 4096 May 20 19:11 indices
-rw-r--r-- 1 nagios users  519 May 20 19:11 metadata-curator-20200521001141
-rw-r--r-- 1 nagios users  519 May 21 11:29 metadata-curator-20200521162953
-rw-r--r-- 1 nagios users  210 May 20 19:24 snapshot-curator-20200521001141
-rw-r--r-- 1 nagios users  207 May 21 11:31 snapshot-curator-20200521162953

I'm also seeing something pop up on the snapshot menu!

Does this all look correct?
Will the index always be Logstash?
Will restoring just bring me back to an older config?

[ssax]

Yes, that all looks correct.

It will list the current index, since it only occurs once a day you should see them show up for each day.

Snapshots are point in time backups of your log data that exists in the Elasticsearch database, so restoring it will only restore the data from that snapshot.

Taken from here:

https://assets.nagios.com/downloads/nag ... enance.pdf

See here as well for config snapshots and system backups (different than regular snapshots):

https://assets.nagios.com/downloads/nag ... Server.pdf