Log Server does not create backups

[vmesquita]

I changed the fstab line to:

Code: Select all

//s*****/logserver       /var/backup     cifs    uid=nagios,gid=nagios,rw,suid,file_mode=0770,dir_mode=0770,user=*******,pass=*******,ip=************** 0 0

But still I got the same error:

Code: Select all

2016-08-30 15:59:32,568 ERROR     Failed to verify all nodes have repository access.
2016-08-30 15:59:32,568 WARNING   Job did not complete successfully.

When trying to run in both nodes.

[mcapra]

Did you make those changes to fstab on every instance and remount? There could be other permissions considerations on the remote machine hosting the net share as well.

Have you also verified that every NLS instance has the exact same path set for the repository and the exact same fstab settings? Even something as simple as a misplaced uppercase character can cause this.

[vmesquita]

I unmounted and mounted the share again after the change. I also cut and pasted the same line on both machines to ensure they were exactly equal. How can I check about permission on the machine hosting the net share?

[mcapra]

As I said, not all net shares are created equal. There isn't really a "one size fits all" answer to that question. If you'd like, we can continue this in an email ticket and schedule a remote assistance session to troubleshoot this more intimately.

You can try skipping the repository validation as well, but curator isn't known for throwing out false positives for repository permissions:

Code: Select all

curator snapshot --skip-repo-validation --repository Backup indices --older-than 1 --time-unit days --timestring %Y.%m.%d

[vmesquita]

Ok, finally, I made it work. Actually the garbage from previous attempts was preventing the backup from working. I deleted the whole content of the backup folder and ran:

Code: Select all

curator snapshot --repository Backup indices --older-than 1 --time-unit days --timestring %Y.%m.%d

And this time it was successful:

Code: Select all

2016-09-20 18:14:40,413 INFO      Snapshot name: curator-20160920211440
2016-09-20 18:16:49,508 INFO      Snapshot curator-20160920211440 successfully completed.
2016-09-20 18:16:49,508 INFO      Job completed successfully.

The snapshot created is also shown on the web interface.

Now I am waiting to see if Nagios Log Server will create backups automatically.

[mcapra]

It's likely the previous backup got tripped-up due to memory limitations. This can upset curator permanently for that repository depending on where the process failed.

Let us know if the automatic backup isn't triggered!

[vmesquita]

Automatic backups have been working for the last days. Thanks!

[mcapra]

Awesome! Is it alright if we lock this thread and mark the issue as resolved?

[vmesquita]

Well, unfortunatelly a new issue came up. All the backups dissapeared from the interface, new ones are not being created and if I manually run:

Code: Select all

curator snapshot --repository Backup indices --older-than 1 --time-unit days --timestring %Y.%m.%d

I get:

Code: Select all

2016-10-07 11:42:47,334 INFO      Job starting: snapshot indices
2016-10-07 11:42:47,334 WARNING   Overriding default connection timeout.  New timeout: 21600
2016-10-07 11:42:47,401 INFO      Action snapshot will be performed on the following indices: [u'logstash-2015.01.01', u'logstash-2015.01.02', u'logstash-2015.01.03', u'logstash-2015.01.04', u'logstash-2015.01.05', u'logstash-2015.01.06', u'logstash-2015.01.07', u'logstash-2015.08.06', u'logstash-2015.11.16', u'logstash-2015.11.17', u'logstash-2015.11.18', u'logstash-2015.11.19', u'logstash-2015.11.20', u'logstash-2015.11.21', u'logstash-2015.11.22', u'logstash-2015.11.23', u'logstash-2015.11.24', u'logstash-2015.11.25', u'logstash-2015.11.26', u'logstash-2015.11.27', u'logstash-2015.11.28', u'logstash-2015.11.29', u'logstash-2015.11.30', u'logstash-2015.12.01', u'logstash-2015.12.02', u'logstash-2015.12.03', u'logstash-2015.12.04', u'logstash-2015.12.05', u'logstash-2015.12.06', u'logstash-2015.12.07', u'logstash-2015.12.08', u'logstash-2015.12.09', u'logstash-2015.12.10', u'logstash-2015.12.11', u'logstash-2015.12.12', u'logstash-2015.12.13', u'logstash-2015.12.14', u'logstash-2015.12.15', u'logstash-2015.12.16', u'logstash-2015.12.17', u'logstash-2015.12.18', u'logstash-2015.12.19', u'logstash-2015.12.20', u'logstash-2015.12.21', u'logstash-2015.12.22', u'logstash-2015.12.23', u'logstash-2015.12.24', u'logstash-2015.12.25', u'logstash-2015.12.26', u'logstash-2015.12.27', u'logstash-2015.12.28', u'logstash-2015.12.29', u'logstash-2015.12.30', u'logstash-2015.12.31', u'logstash-2016.01.01', u'logstash-2016.01.02', u'logstash-2016.01.03', u'logstash-2016.01.04', u'logstash-2016.01.05', u'logstash-2016.01.06', u'logstash-2016.01.07', u'logstash-2016.01.08', u'logstash-2016.01.09', u'logstash-2016.01.10', u'logstash-2016.01.11', u'logstash-2016.01.12', u'logstash-2016.01.13', u'logstash-2016.01.14', u'logstash-2016.01.15', u'logstash-2016.01.16', u'logstash-2016.01.17', u'logstash-2016.01.18', u'logstash-2016.01.19', u'logstash-2016.01.20', u'logstash-2016.01.21', u'logstash-2016.01.22', u'logstash-2016.01.23', u'logstash-2016.01.24', u'logstash-2016.01.25', u'logstash-2016.01.26', u'logstash-2016.01.27', u'logstash-2016.01.28', u'logstash-2016.01.29', u'logstash-2016.01.30', u'logstash-2016.01.31', u'logstash-2016.02.01', u'logstash-2016.02.02', u'logstash-2016.02.03', u'logstash-2016.02.04', u'logstash-2016.02.05', u'logstash-2016.02.06', u'logstash-2016.02.07', u'logstash-2016.02.08', u'logstash-2016.02.09', u'logstash-2016.02.10', u'logstash-2016.02.11', u'logstash-2016.02.12', u'logstash-2016.02.13', u'logstash-2016.02.14', u'logstash-2016.02.15', u'logstash-2016.02.16', u'logstash-2016.02.17', u'logstash-2016.02.18', u'logstash-2016.02.19', u'logstash-2016.02.20', u'logstash-2016.02.21', u'logstash-2016.02.22', u'logstash-2016.02.23', u'logstash-2016.02.24', u'logstash-2016.02.25', u'logstash-2016.02.26', u'logstash-2016.02.27', u'logstash-2016.02.28', u'logstash-2016.02.29', u'logstash-2016.03.01', u'logstash-2016.03.02', u'logstash-2016.03.03', u'logstash-2016.03.04', u'logstash-2016.03.05', u'logstash-2016.03.06', u'logstash-2016.03.07', u'logstash-2016.03.08', u'logstash-2016.03.09', u'logstash-2016.03.10', u'logstash-2016.03.11', u'logstash-2016.03.12', u'logstash-2016.03.13', u'logstash-2016.03.14', u'logstash-2016.03.15', u'logstash-2016.03.16', u'logstash-2016.03.17', u'logstash-2016.03.18', u'logstash-2016.03.19', u'logstash-2016.03.20', u'logstash-2016.03.21', u'logstash-2016.03.22', u'logstash-2016.03.23', u'logstash-2016.03.24', u'logstash-2016.03.25', u'logstash-2016.03.26', u'logstash-2016.03.27', u'logstash-2016.03.28', u'logstash-2016.03.29', u'logstash-2016.03.30', u'logstash-2016.03.31', u'logstash-2016.04.01', u'logstash-2016.04.02', u'logstash-2016.04.03', u'logstash-2016.04.04', u'logstash-2016.04.05', u'logstash-2016.04.06', u'logstash-2016.04.07', u'logstash-2016.04.08', u'logstash-2016.04.09', u'logstash-2016.04.10', u'logstash-2016.04.11', u'logstash-2016.04.12', u'logstash-2016.04.13', u'logstash-2016.04.14', u'logstash-2016.04.15', u'logstash-2016.04.16', u'logstash-2016.04.17', u'logstash-2016.04.18', u'logstash-2016.04.19', u'logstash-2016.04.20', u'logstash-2016.04.21', u'logstash-2016.04.22', u'logstash-2016.04.23', u'logstash-2016.04.24', u'logstash-2016.04.25', u'logstash-2016.04.26', u'logstash-2016.04.27', u'logstash-2016.04.28', u'logstash-2016.04.29', u'logstash-2016.04.30', u'logstash-2016.05.01', u'logstash-2016.05.02', u'logstash-2016.05.03', u'logstash-2016.05.04', u'logstash-2016.05.05', u'logstash-2016.05.06', u'logstash-2016.05.07', u'logstash-2016.05.08', u'logstash-2016.05.09', u'logstash-2016.05.10', u'logstash-2016.05.11', u'logstash-2016.05.12', u'logstash-2016.05.13', u'logstash-2016.05.14', u'logstash-2016.05.15', u'logstash-2016.05.16', u'logstash-2016.05.17', u'logstash-2016.05.18', u'logstash-2016.05.19', u'logstash-2016.05.20', u'logstash-2016.05.21', u'logstash-2016.05.22', u'logstash-2016.05.23', u'logstash-2016.05.24', u'logstash-2016.05.25', u'logstash-2016.05.26', u'logstash-2016.05.27', u'logstash-2016.05.28', u'logstash-2016.05.29', u'logstash-2016.05.30', u'logstash-2016.05.31', u'logstash-2016.06.01', u'logstash-2016.06.02', u'logstash-2016.06.03', u'logstash-2016.06.04', u'logstash-2016.06.05', u'logstash-2016.06.06', u'logstash-2016.06.07', u'logstash-2016.06.08', u'logstash-2016.06.09', u'logstash-2016.06.10', u'logstash-2016.06.11', u'logstash-2016.06.12', u'logstash-2016.06.13', u'logstash-2016.06.14', u'logstash-2016.06.15', u'logstash-2016.06.16', u'logstash-2016.06.17', u'logstash-2016.06.18', u'logstash-2016.06.19', u'logstash-2016.06.20', u'logstash-2016.06.21', u'logstash-2016.06.22', u'logstash-2016.06.23', u'logstash-2016.06.24', u'logstash-2016.06.25', u'logstash-2016.06.26', u'logstash-2016.06.27', u'logstash-2016.06.28', u'logstash-2016.06.29', u'logstash-2016.06.30', u'logstash-2016.07.01', u'logstash-2016.07.02', u'logstash-2016.07.03', u'logstash-2016.07.04', u'logstash-2016.07.05', u'logstash-2016.07.06', u'logstash-2016.07.07', u'logstash-2016.07.08', u'logstash-2016.07.09', u'logstash-2016.07.10', u'logstash-2016.07.11', u'logstash-2016.07.12', u'logstash-2016.07.13', u'logstash-2016.07.14', u'logstash-2016.07.15', u'logstash-2016.07.16', u'logstash-2016.07.17', u'logstash-2016.07.18', u'logstash-2016.07.19', u'logstash-2016.07.20', u'logstash-2016.07.21', u'logstash-2016.07.22', u'logstash-2016.07.23', u'logstash-2016.07.24', u'logstash-2016.07.25', u'logstash-2016.07.26', u'logstash-2016.07.27', u'logstash-2016.07.28', u'logstash-2016.07.29', u'logstash-2016.07.30', u'logstash-2016.07.31', u'logstash-2016.08.01', u'logstash-2016.08.02', u'logstash-2016.08.03', u'logstash-2016.08.04', u'logstash-2016.08.05', u'logstash-2016.08.06', u'logstash-2016.08.07', u'logstash-2016.08.08', u'logstash-2016.08.09', u'logstash-2016.08.10', u'logstash-2016.08.11', u'logstash-2016.08.12', u'logstash-2016.08.13', u'logstash-2016.08.14', u'logstash-2016.08.15', u'logstash-2016.08.16', u'logstash-2016.08.17', u'logstash-2016.08.18', u'logstash-2016.08.19', u'logstash-2016.08.20', u'logstash-2016.08.21', u'logstash-2016.08.22', u'logstash-2016.08.23', u'logstash-2016.08.24', u'logstash-2016.08.25', u'logstash-2016.08.26', u'logstash-2016.08.27', u'logstash-2016.08.28', u'logstash-2016.08.29', u'logstash-2016.08.30', u'logstash-2016.08.31', u'logstash-2016.09.01', u'logstash-2016.09.02', u'logstash-2016.09.03', u'logstash-2016.09.04', u'logstash-2016.09.05', u'logstash-2016.09.06', u'logstash-2016.09.07', u'logstash-2016.09.08', u'logstash-2016.09.09', u'logstash-2016.09.10', u'logstash-2016.09.11', u'logstash-2016.09.12', u'logstash-2016.09.13', u'logstash-2016.09.14', u'logstash-2016.09.15', u'logstash-2016.09.16', u'logstash-2016.09.17', u'logstash-2016.09.18', u'logstash-2016.09.19', u'logstash-2016.09.20', u'logstash-2016.09.21', u'logstash-2016.09.22', u'logstash-2016.09.23', u'logstash-2016.09.24', u'logstash-2016.09.25', u'logstash-2016.09.26', u'logstash-2016.09.27', u'logstash-2016.09.28', u'logstash-2016.09.29', u'logstash-2016.09.30', u'logstash-2016.10.01', u'logstash-2016.10.02', u'logstash-2016.10.03', u'logstash-2016.10.04', u'logstash-2016.10.05', u'logstash-2016.10.06']
2016-10-07 11:42:52,898 INFO      Snapshot name: curator-20161007144252
2016-10-07 11:42:53,066 ERROR     Unable to find all snapshots in repository: Backup
Traceback (most recent call last):
  File "/usr/bin/curator", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/curator/curator.py", line 5, in main
    cli( obj={ "filters": [] } )
  File "/usr/lib/python2.6/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.6/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.6/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 86, in augment_usage_errors
    yield
  File "/usr/lib/python2.6/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/curator/cli/index_selection.py", line 167, in indices
    retval = do_command(client, ctx.parent.info_name, working_list, ctx.parent.params, master_timeout)
  File "/usr/lib/python2.6/site-packages/curator/cli/utils.py", line 250, in do_command
    skip_repo_validation=params['skip_repo_validation'],
  File "/usr/lib/python2.6/site-packages/curator/api/snapshot.py", line 72, in create_snapshot
    if name in all_snaps:
TypeError: argument of type 'bool' is not iterable

Any ideas?

[rkennedy]

Could you show us a screenshot of your backup & maintenance page?