Indexes backup
-
- Posts: 39
- Joined: Thu Nov 16, 2017 6:46 am
Indexes backup
Configuration done by the book (as below), NFS repo, connected to both nodes. (last version of LS from OVA image)
“Managing-Nagios-Log-Server-Snapshots-and-Maintenance.pdf
Snapshot-Repository-Considerations.pdf”
On both nodes and NFS server nagios user and group has the same UID/GID.
Both nodes can create test file (touch test.txt) on NFS share as nagios user.
But still I can't see any index backups coming to /mnt/snapshot_repository (NFS Share).
Nor old indexes are deleted.
"Enable Maintenance and Snapshots" is set to "Yes" on config page.
other settings on the page are: Optimize: 1d, Close: 2d, Delete: 7d, Repository: pointing_to_repo_defined.
What did I miss?
Why no indexes backup is perform, and why older then 7d indexes are not deleted?
Best regards,
D
“Managing-Nagios-Log-Server-Snapshots-and-Maintenance.pdf
Snapshot-Repository-Considerations.pdf”
On both nodes and NFS server nagios user and group has the same UID/GID.
Both nodes can create test file (touch test.txt) on NFS share as nagios user.
But still I can't see any index backups coming to /mnt/snapshot_repository (NFS Share).
Nor old indexes are deleted.
"Enable Maintenance and Snapshots" is set to "Yes" on config page.
other settings on the page are: Optimize: 1d, Close: 2d, Delete: 7d, Repository: pointing_to_repo_defined.
What did I miss?
Why no indexes backup is perform, and why older then 7d indexes are not deleted?
Best regards,
D
Re: Indexes backup
Check under Admin > System > Command Subsystem, and use the Reset All Jobs button to reset the jobs and then use the Run action to run the snapshots_maintenance job. If it still isn't working, run this from the command line:
and then try forcing the job to run again(you may need to wait a minute after forcing it before it run) and gather the output that is displayed on the command line.
Please also provide a screenshot of the Snapshots & Maintenance page so we can verify the rest of the config on that page.
Code: Select all
tail -f /usr/local/nagioslogserver/var/jobs.log
Please also provide a screenshot of the Snapshots & Maintenance page so we can verify the rest of the config on that page.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
-
- Posts: 39
- Joined: Thu Nov 16, 2017 6:46 am
Re: Indexes backup
Snapshots & Maintenance - screen as attachment.
I was trying to run it manualy a few times, all is success, it says...
on node2:
on node1:
Errors show up not afrer "Reset All Jobs"
But when i click "run", a few minutes later at snapshots_maintenance job ID.
I was trying to run it manualy a few times, all is success, it says...
jobs.log output:cleanup_cmdsubsys Waiting SUCCESS 11/22/2019 12:20:47 1 hour 11/22/2019 13:20:47 System Edit Run
backups Waiting SUCCESS 11/21/2019 21:17:38 1 day 11/22/2019 21:17:38 System Edit Run
snapshots_maintenance Waiting SUCCESS 11/21/2019 21:17:28 1 day 11/22/2019 21:17:28 System Edit Run
run_all_alerts Waiting SUCCESS 11/22/2019 12:46:41 20 seconds 11/22/2019 12:47:01 System Edit Run
run_index_usage Waiting SUCCESS 11/22/2019 12:45:16 15 minutes 11/22/2019 13:00:16 System Edit Run
run_update_check Waiting SUCCESS 11/21/2019 21:18:02 1 day 11/22/2019 21:18:02 System Edit Run
on node2:
Code: Select all
tail -f /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 2 global jobs.
Errors show up not afrer "Reset All Jobs"
But when i click "run", a few minutes later at snapshots_maintenance job ID.
Code: Select all
tail -f /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
^C
[root@wa-ls1-p ~]# tail -n 20 -f /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
^C
[root@wa-ls1-p ~]# tail -n 200 -f /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
^C
[root@wa-ls1-p ~]# tail -f /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command do_maintenance with args ' ' for job id: snapshots_maintenance
Traceback (most recent call last):
File "/usr/bin/curator", line 7, in <module>
from curator.curator import main
File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
from .api import *
File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
from .utils import *
File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
import elasticsearch
File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
from .client import Elasticsearch
File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
from ..transport import Transport
File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
from .connection import Urllib3HttpConnection
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
from .http_urllib3 import Urllib3HttpConnection
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
import urllib3
File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
from .connectionpool import (
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
from .connection import (
File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
from .util.ssl_ import (
File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
from .request import make_headers
File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh optimize indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Return: 1
-----
Traceback (most recent call last):
File "/usr/bin/curator", line 7, in <module>
from curator.curator import main
File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
from .api import *
File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
from .utils import *
File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
import elasticsearch
File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
from .client import Elasticsearch
File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
from ..transport import Transport
File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
from .connection import Urllib3HttpConnection
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
from .http_urllib3 import Urllib3HttpConnection
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
import urllib3
File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
from .connectionpool import (
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
from .connection import (
File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
from .util.ssl_ import (
File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
from .request import make_headers
File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh close indices --older-than 3 --time-unit days --timestring %Y.%m.%d
Return: 1
-----
Traceback (most recent call last):
File "/usr/bin/curator", line 7, in <module>
from curator.curator import main
File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
from .api import *
File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
from .utils import *
File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
import elasticsearch
File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
from .client import Elasticsearch
File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
from ..transport import Transport
File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
from .connection import Urllib3HttpConnection
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
from .http_urllib3 import Urllib3HttpConnection
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
import urllib3
File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
from .connectionpool import (
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
from .connection import (
File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
from .util.ssl_ import (
File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
from .request import make_headers
File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete indices --older-than 7 --time-unit days --timestring %Y.%m.%d
Return: 1
-----
Traceback (most recent call last):
File "/usr/bin/curator", line 7, in <module>
from curator.curator import main
File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
from .api import *
File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
from .utils import *
File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
import elasticsearch
File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
from .client import Elasticsearch
File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
from ..transport import Transport
File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
from .connection import Urllib3HttpConnection
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
from .http_urllib3 import Urllib3HttpConnection
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
import urllib3
File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
from .connectionpool import (
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
from .connection import (
File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
from .util.ssl_ import (
File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
from .request import make_headers
File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "Snapshot Repository" --ignore_unavailable indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Return: 1
-----
Traceback (most recent call last):
File "/usr/bin/curator", line 7, in <module>
from curator.curator import main
File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
from .api import *
File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
from .utils import *
File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
import elasticsearch
File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
from .client import Elasticsearch
File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
from ..transport import Transport
File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
from .connection import Urllib3HttpConnection
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
from .http_urllib3 import Urllib3HttpConnection
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
import urllib3
File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
from .connectionpool import (
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
from .connection import (
File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
from .util.ssl_ import (
File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
from .request import make_headers
File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete snapshots --older-than 365 --time-unit days --timestring %Y%m%d --repository "Snapshot Repository"
Return: 1
-----
SUCCESS
Processed 0 node jobs.
Processed 2 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
You do not have the required permissions to view the files attached to this post.
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Indexes backup
It's like your missing part of the expected packages
Can you run the following
Can you run the following
Code: Select all
sudo pip uninstall requests
sudo pip uninstall urllib3
sudo yum remove python-urllib3
sudo yum remove python-requests
sudo yum install python-urllib3
sudo yum install python-requests
-
- Posts: 39
- Joined: Thu Nov 16, 2017 6:46 am
Re: Indexes backup
OK. I'll check it out. I hope it will help.
D.
D.
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Indexes backup
Let us know how it turns outdariusz.nalazek wrote:OK. I'll check it out. I hope it will help.
D.
-
- Posts: 39
- Joined: Thu Nov 16, 2017 6:46 am
Re: Indexes backup
Seems the problem is gone, after reinstalling the packages.
Nagios LS conflicts with client of SUSE Manager (Saltstack client) which I’m using to manage GNU/Linux servers. Now Nagios LS is in perfect health, but salt-minion can’t start… I’ll try to figure out how to marge the systems to work together. Always I can manage LS servers throught ssh based client instead of salt based client, but salt is worth of effort, to try, to make it running.
Maybe you have some experience in salt's minions and Nagios LS?
D.
Nagios LS conflicts with client of SUSE Manager (Saltstack client) which I’m using to manage GNU/Linux servers. Now Nagios LS is in perfect health, but salt-minion can’t start… I’ll try to figure out how to marge the systems to work together. Always I can manage LS servers throught ssh based client instead of salt based client, but salt is worth of effort, to try, to make it running.
Maybe you have some experience in salt's minions and Nagios LS?
D.
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Indexes backup
I'm not familiar with salt, the only thing that comes to mind would be if there was a port conflict, but I'm not familiar with what ports it uses, or what ports you have chosen to use with your logstash setup.
-
- Posts: 39
- Joined: Thu Nov 16, 2017 6:46 am
Re: Indexes backup
It's conflict between python modules/libraries (maybe versions), salt-minion is python sensitive.
I'll try to investigate it, since now I know where to dig
I'll try to investigate it, since now I know where to dig
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Indexes backup
Ohhh..dariusz.nalazek wrote:It's conflict between python modules/libraries (maybe versions), salt-minion is python sensitive.
I'll try to investigate it, since now I know where to dig
Good luck!