Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Indexes backup

https://support.nagios.com/forum/viewtopic.php?t=56492

Page 1 of 2

Indexes backup

Posted: Mon Nov 18, 2019 6:45 am
by dariusz.nalazek
Configuration done by the book (as below), NFS repo, connected to both nodes. (last version of LS from OVA image)

“Managing-Nagios-Log-Server-Snapshots-and-Maintenance.pdf
Snapshot-Repository-Considerations.pdf”

On both nodes and NFS server nagios user and group has the same UID/GID.
Both nodes can create test file (touch test.txt) on NFS share as nagios user.

But still I can't see any index backups coming to /mnt/snapshot_repository (NFS Share).
Nor old indexes are deleted.

"Enable Maintenance and Snapshots" is set to "Yes" on config page.
other settings on the page are: Optimize: 1d, Close: 2d, Delete: 7d, Repository: pointing_to_repo_defined.

What did I miss?
Why no indexes backup is perform, and why older then 7d indexes are not deleted?

Best regards,
D

Re: Indexes backup

Posted: Mon Nov 18, 2019 1:38 pm
by cdienger
Check under Admin > System > Command Subsystem, and use the Reset All Jobs button to reset the jobs and then use the Run action to run the snapshots_maintenance job. If it still isn't working, run this from the command line:

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log
and then try forcing the job to run again(you may need to wait a minute after forcing it before it run) and gather the output that is displayed on the command line.

Please also provide a screenshot of the Snapshots & Maintenance page so we can verify the rest of the config on that page.

Re: Indexes backup

Posted: Fri Nov 22, 2019 6:57 am
by dariusz.nalazek
Snapshots & Maintenance - screen as attachment.

I was trying to run it manualy a few times, all is success, it says...
cleanup_cmdsubsys Waiting SUCCESS 11/22/2019 12:20:47 1 hour 11/22/2019 13:20:47 System Edit Run
backups Waiting SUCCESS 11/21/2019 21:17:38 1 day 11/22/2019 21:17:38 System Edit Run
snapshots_maintenance Waiting SUCCESS 11/21/2019 21:17:28 1 day 11/22/2019 21:17:28 System Edit Run
run_all_alerts Waiting SUCCESS 11/22/2019 12:46:41 20 seconds 11/22/2019 12:47:01 System Edit Run
run_index_usage Waiting SUCCESS 11/22/2019 12:45:16 15 minutes 11/22/2019 13:00:16 System Edit Run
run_update_check Waiting SUCCESS 11/21/2019 21:18:02 1 day 11/22/2019 21:18:02 System Edit Run
jobs.log output:

on node2:

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log        
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 2 global jobs.
on node1:

Errors show up not afrer "Reset All Jobs"
But when i click "run", a few minutes later at snapshots_maintenance job ID.

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
^C
[root@wa-ls1-p ~]# tail -n 20 -f /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
^C
[root@wa-ls1-p ~]# tail -n 200 -f /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
^C
[root@wa-ls1-p ~]# tail -f /usr/local/nagioslogserver/var/jobs.log       
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command do_maintenance with args ' ' for job id: snapshots_maintenance
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh optimize indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Return: 1

-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh close indices --older-than 3 --time-unit days --timestring %Y.%m.%d
Return: 1

-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete indices --older-than 7 --time-unit days --timestring %Y.%m.%d
Return: 1

-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "Snapshot Repository" --ignore_unavailable indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Return: 1

-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 7, in <module>
    from curator.curator import main
  File "/usr/lib/python2.7/site-packages/curator/__init__.py", line 2, in <module>
    from .api import *
  File "/usr/lib/python2.7/site-packages/curator/api/__init__.py", line 1, in <module>
    from .utils import *
  File "/usr/lib/python2.7/site-packages/curator/api/utils.py", line 2, in <module>
    import elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/__init__.py", line 17, in <module>
    from .client import Elasticsearch
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 5, in <module>
    from ..transport import Transport
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 5, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/__init__.py", line 3, in <module>
    from .http_urllib3 import Urllib3HttpConnection
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 2, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 31, in <module>
    from .connection import (
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 45, in <module>
    from .util.ssl_ import (
  File "/usr/lib/python2.7/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .request import make_headers
  File "/usr/lib/python2.7/site-packages/urllib3/util/request.py", line 5, in <module>
    from ..exceptions import UnrewindableBodyError
ImportError: cannot import name UnrewindableBodyError
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete snapshots --older-than 365 --time-unit days --timestring %Y%m%d --repository "Snapshot Repository"
Return: 1

-----
SUCCESS
Processed 0 node jobs.
Processed 2 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 3 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS

Re: Indexes backup

Posted: Fri Nov 22, 2019 3:29 pm
by scottwilkerson
It's like your missing part of the expected packages

Can you run the following

Code: Select all

sudo pip uninstall requests
sudo pip uninstall urllib3
sudo yum remove python-urllib3
sudo yum remove python-requests
sudo yum install python-urllib3
sudo yum install python-requests

Re: Indexes backup

Posted: Fri Nov 22, 2019 6:04 pm
by dariusz.nalazek
OK. I'll check it out. I hope it will help.

D.

Re: Indexes backup

Posted: Mon Nov 25, 2019 7:53 am
by scottwilkerson
dariusz.nalazek wrote:OK. I'll check it out. I hope it will help.

D.
Let us know how it turns out

Re: Indexes backup

Posted: Tue Nov 26, 2019 1:07 pm
by dariusz.nalazek
Seems the problem is gone, after reinstalling the packages.

Nagios LS conflicts with client of SUSE Manager (Saltstack client) which I’m using to manage GNU/Linux servers. Now Nagios LS is in perfect health, but salt-minion can’t start… I’ll try to figure out how to marge the systems to work together. Always I can manage LS servers throught ssh based client instead of salt based client, but salt is worth of effort, to try, to make it running.

Maybe you have some experience in salt's minions and Nagios LS?

D.

Re: Indexes backup

Posted: Tue Nov 26, 2019 1:18 pm
by scottwilkerson
I'm not familiar with salt, the only thing that comes to mind would be if there was a port conflict, but I'm not familiar with what ports it uses, or what ports you have chosen to use with your logstash setup.

Re: Indexes backup

Posted: Tue Nov 26, 2019 1:27 pm
by dariusz.nalazek
It's conflict between python modules/libraries (maybe versions), salt-minion is python sensitive.
I'll try to investigate it, since now I know where to dig :)

Re: Indexes backup

Posted: Tue Nov 26, 2019 1:29 pm
by scottwilkerson
dariusz.nalazek wrote:It's conflict between python modules/libraries (maybe versions), salt-minion is python sensitive.
I'll try to investigate it, since now I know where to dig :)
Ohhh..

Good luck!
All times are UTC-05:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited