Log Server does not create backups

[vmesquita]

The indexes were closed, and I am attaching the output. So I should expect the backups to work correctly now?

[mcapra]

Lets test it from the CLI to verify. Share the output:

Code: Select all

curator snapshot --repository Backup indices --older-than 1 --time-unit days --timestring %Y.%m.%d

And see if there is a snapshot in the NLS GUI on the Backup & Maintenance page.

[vmesquita]

I got this message:

Code: Select all

2016-08-23 17:27:59,303 ERROR     Failed to verify all nodes have repository access.
2016-08-23 17:27:59,303 WARNING   Job did not complete successfully.

But I checked and the same share is mounted in both instances. How can I troubleshoot this?

[mcapra]

Is the repository an NFS or CIFS mount? Could be some underlying permissions there that are causing issues.

[vmesquita]

It's a cifs mount. But it seems that files are being created:

Code: Select all

# ls -ls
total 1
0 drwxr-xr-x 0 nagios root   0 Aug 17 19:21 indices
1 -rwxr-xr-x 0 nagios root 458 Aug 17 19:21 metadata-curator-20160817222121
0 -rwxr-xr-x 0 nagios root   0 Aug 17 19:51 snapshot-curator-20160817222121

[mcapra]

The 0 size on the snapshot-curator file is concerning.

Here's what my fairly new repository looks like:

Code: Select all

$ ls -al
total 80
drwxrwxrwx   3 root   root  4096 Aug 23 22:56 .
drwxr-xr-x. 21 root   root  4096 Aug 18 21:45 ..
-rw-r--r--   1 nagios users  119 Aug 23 22:56 index
drwxr-xr-x  34 nagios users 4096 Aug 23 20:15 indices
-rw-r--r--   1 nagios users  457 Aug 17 21:37 metadata-curator-20160817193707
-rw-r--r--   1 nagios users  457 Aug 18 21:37 metadata-curator-20160818193711
-rw-r--r--   1 nagios users  460 Aug 19 21:37 metadata-curator-20160819193711
-rw-r--r--   1 nagios users  460 Aug 20 21:37 metadata-curator-20160820193718
-rw-r--r--   1 nagios users  460 Aug 21 21:37 metadata-curator-20160821193714
-rw-r--r--   1 nagios users  457 Aug 22 21:37 metadata-curator-20160822193723
-rw-r--r--   1 nagios users  457 Aug 23 20:15 metadata-curator-20160823181504
-rw-r--r--   1 nagios users  457 Aug 23 22:56 metadata-curator-20160823205650
-rw-r--r--   1 nagios users  246 Aug 17 21:37 snapshot-curator-20160817193707
-rw-r--r--   1 nagios users  255 Aug 18 21:37 snapshot-curator-20160818193711
-rw-r--r--   1 nagios users  260 Aug 19 21:37 snapshot-curator-20160819193711
-rw-r--r--   1 nagios users  240 Aug 20 21:37 snapshot-curator-20160820193718
-rw-r--r--   1 nagios users  241 Aug 21 21:37 snapshot-curator-20160821193714
-rw-r--r--   1 nagios users  338 Aug 22 21:37 snapshot-curator-20160822193723
-rw-r--r--   1 nagios users  235 Aug 23 20:15 snapshot-curator-20160823181504
-rw-r--r--   1 nagios users  237 Aug 23 22:56 snapshot-curator-20160823205650

Can you see if giving the "users" group write permission on this path changes things?

[vmesquita]

No change:

Code: Select all

# ls -ls /var/backup*
total 1
0 drwxr-xr-x 0 nagios users   0 Aug 17 19:21 indices
1 -rwxr-xr-x 0 nagios users 458 Aug 17 19:21 metadata-curator-20160817222121
0 -rwxr-xr-x 0 nagios users   0 Aug 17 19:51 snapshot-curator-20160817222121
# ls -ls /var
total 88
4 drwxr-xr-x.  2 root   root  4096 Jul 10  2012 account
4 drwxr-xr-x   1 nagios users 4096 Aug 17 19:21 backup
4 drwxr-xr-x. 14 root   root  4096 Jan 25  2016 cache
4 drwxr-xr-x.  2 root   root  4096 Feb 25  2016 crash
4 drwxr-xr-x.  2 root   root  4096 Jun 24  2013 cvs
4 drwxr-xr-x.  3 root   root  4096 Jun 28  2011 db
4 drwxr-xr-x   3 root   root  4096 May 23 18:02 dell
4 drwxr-xr-x.  3 root   root  4096 Jun 28  2011 empty
4 drwxr-xr-x.  2 root   root  4096 Jun 28  2011 games
4 drwxr-xr-x. 32 root   root  4096 Aug 29 03:49 lib
4 drwxr-xr-x.  2 root   root  4096 Jun 28  2011 local
4 drwxrwxr-x.  5 root   lock  4096 Aug 29 03:49 lock
4 drwxr-xr-x. 15 root   root  4096 Aug 28 03:45 log
0 lrwxrwxrwx.  1 root   root    10 Jan 25  2016 mail -> spool/mail
4 drwxr-xr-x.  2 root   root  4096 Jun 28  2011 nis
4 drwxr-xr-x.  2 root   root  4096 Jun 28  2011 opt
4 drwxr-xr-x.  2 root   root  4096 Jun 28  2011 preserve
4 drwxr-xr-x. 26 root   root  4096 Aug 17 16:24 run
4 drwxr-xr-x   2 root   root  4096 Jul 18 17:32 sb530
4 drwxr-xr-x. 16 root   root  4096 Jan 25  2016 spool
4 drwxrwxrwt.  2 root   root  4096 Aug 16 14:35 tmp
4 drwxr-xr-x.  6 root   root  4096 Jan 25  2016 www
4 drwxr-xr-x.  3 root   root  4096 Jan 25  2016 yp

[mcapra]

Can you share the contents of your fstab file(s)?

Code: Select all

cat /etc/fstab

Be sure to sanitize any credentials that may exist in the entries.

[vmesquita]

Code: Select all

# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Oct 30 08:28:51 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/VolGroup-lv_root /                       ext4    defaults        1 1
UUID=f57459cb-d15a-4fce-b67d-57c62dde1b0a /boot                   ext4    defaults        1 2
/dev/mapper/VolGroup-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
//s******/logserver       /var/backup    cifs    user=************,pass=*********,ip=*************,uid=nagios,gid=users    0 0

[mcapra]

Here's an example CIFS mount i've been sending backups to for some time now without issues:

Code: Select all

\\192.168.67.201\NLSBackup /mnt/bkk cifs user,uid=nagios,gid=nagios,rw,suid,file_mode=0770,dir_mode=0770,credentials=/root/cifspass.txt 0 0

See if replicating those flags and remounting solves the issue. Specifically the uid=nagios,gid=nagios,rw,suid,file_mode=0770,dir_mode=0770 flags. Since you're passing the credentials in-line, a credentials file would be redundant.

Again, to run a test snapshot:

Code: Select all

curator snapshot --repository Backup indices --older-than 1 --time-unit days --timestring %Y.%m.%d