Indexes backup

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
dariusz.nalazek
Posts: 39
Joined: Thu Nov 16, 2017 6:46 am

Indexes backup

Post by dariusz.nalazek »

Configuration done by the book (as below), NFS repo, connected to both nodes. (last version of LS from OVA image)

“Managing-Nagios-Log-Server-Snapshots-and-Maintenance.pdf
Snapshot-Repository-Considerations.pdf”

On both nodes and NFS server nagios user and group has the same UID/GID.
Both nodes can create test file (touch test.txt) on NFS share as nagios user.

But still I can't see any index backups coming to /mnt/snapshot_repository (NFS Share).
Nor old indexes are deleted.

"Enable Maintenance and Snapshots" is set to "Yes" on config page.
other settings on the page are: Optimize: 1d, Close: 2d, Delete: 7d, Repository: pointing_to_repo_defined.

What did I miss?
Why no indexes backup is perform, and why older then 7d indexes are not deleted?

Best regards,
D
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Indexes backup

Post by cdienger »

Check under Admin > System > Command Subsystem, and use the Reset All Jobs button to reset the jobs and then use the Run action to run the snapshots_maintenance job. If it still isn't working, run this from the command line:

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log
and then try forcing the job to run again(you may need to wait a minute after forcing it before it run) and gather the output that is displayed on the command line.

Please also provide a screenshot of the Snapshots & Maintenance page so we can verify the rest of the config on that page.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
dariusz.nalazek
Posts: 39
Joined: Thu Nov 16, 2017 6:46 am

Re: Indexes backup

Post by dariusz.nalazek »

Snapshots & Maintenance - screen as attachment.

I was trying to run it manualy a few times, all is success, it says...
cleanup_cmdsubsys Waiting SUCCESS 11/22/2019 12:20:47 1 hour 11/22/2019 13:20:47 System Edit Run
backups Waiting SUCCESS 11/21/2019 21:17:38 1 day 11/22/2019 21:17:38 System Edit Run
snapshots_maintenance Waiting SUCCESS 11/21/2019 21:17:28 1 day 11/22/2019 21:17:28 System Edit Run
run_all_alerts Waiting SUCCESS 11/22/2019 12:46:41 20 seconds 11/22/2019 12:47:01 System Edit Run
run_index_usage Waiting SUCCESS 11/22/2019 12:45:16 15 minutes 11/22/2019 13:00:16 System Edit Run
run_update_check Waiting SUCCESS 11/21/2019 21:18:02 1 day 11/22/2019 21:18:02 System Edit Run
jobs.log output:

on node2:

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log        
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 2 global jobs.
on node1:

Errors show up not afrer "Reset All Jobs"
But when i click "run", a few minutes later at snapshots_maintenance job ID.

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
^C
[root@wa-ls1-p ~]# tail -n 20 -f /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
^C
[root@wa-ls1-p ~]# tail -n 200 -f /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
^C
[root@wa-ls1-p ~]# tail -f /usr/local/nagioslogserver/var/jobs.log       
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command do_maintenance with args ' ' for job id: snapshots_maintenance
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh optimize indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Return: 1

-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh close indices --older-than 3 --time-unit days --timestring %Y.%m.%d
Return: 1

-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete indices --older-than 7 --time-unit days --timestring %Y.%m.%d
Return: 1

-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "Snapshot Repository" --ignore_unavailable indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Return: 1

-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete snapshots --older-than 365 --time-unit days --timestring %Y%m%d --repository "Snapshot Repository"
Return: 1

-----
SUCCESS
Processed 0 node jobs.
Processed 2 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
You do not have the required permissions to view the files attached to this post.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Indexes backup

Post by scottwilkerson »

It's like your missing part of the expected packages

Can you run the following

Code: Select all

sudo pip uninstall requests
sudo pip uninstall urllib3
sudo yum remove python-urllib3
sudo yum remove python-requests
sudo yum install python-urllib3
sudo yum install python-requests
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
dariusz.nalazek
Posts: 39
Joined: Thu Nov 16, 2017 6:46 am

Re: Indexes backup

Post by dariusz.nalazek »

OK. I'll check it out. I hope it will help.

D.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Indexes backup

Post by scottwilkerson »

dariusz.nalazek wrote:OK. I'll check it out. I hope it will help.

D.
Let us know how it turns out
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
dariusz.nalazek
Posts: 39
Joined: Thu Nov 16, 2017 6:46 am

Re: Indexes backup

Post by dariusz.nalazek »

Seems the problem is gone, after reinstalling the packages.

Nagios LS conflicts with client of SUSE Manager (Saltstack client) which I’m using to manage GNU/Linux servers. Now Nagios LS is in perfect health, but salt-minion can’t start… I’ll try to figure out how to marge the systems to work together. Always I can manage LS servers throught ssh based client instead of salt based client, but salt is worth of effort, to try, to make it running.

Maybe you have some experience in salt's minions and Nagios LS?

D.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Indexes backup

Post by scottwilkerson »

I'm not familiar with salt, the only thing that comes to mind would be if there was a port conflict, but I'm not familiar with what ports it uses, or what ports you have chosen to use with your logstash setup.
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
dariusz.nalazek
Posts: 39
Joined: Thu Nov 16, 2017 6:46 am

Re: Indexes backup

Post by dariusz.nalazek »

It's conflict between python modules/libraries (maybe versions), salt-minion is python sensitive.
I'll try to investigate it, since now I know where to dig :)
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Indexes backup

Post by scottwilkerson »

dariusz.nalazek wrote:It's conflict between python modules/libraries (maybe versions), salt-minion is python sensitive.
I'll try to investigate it, since now I know where to dig :)
Ohhh..

Good luck!
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
Locked