Log Server does not create backups

[vmesquita]

I attached to this message.

[vmesquita]

Also the backup files seem to still be there:

Code: Select all

# ls /var/backup/
index                            metadata-curator-20160926200807  snapshot-curator-20160920211440  snapshot-curator-20160927200725
indices                          metadata-curator-20160926200947  snapshot-curator-20160921200856  snapshot-curator-20160928202248
metadata-curator-20160920211440  metadata-curator-20160927200725  snapshot-curator-20160922200730  snapshot-curator-20160929201008
metadata-curator-20160921200856  metadata-curator-20160928202248  snapshot-curator-20160923204131  snapshot-curator-20160930200746
metadata-curator-20160922200730  metadata-curator-20160929201008  snapshot-curator-20160924201018  snapshot-curator-20161001200943
metadata-curator-20160923204131  metadata-curator-20160930200746  snapshot-curator-20160924202529  snapshot-curator-20161002201721
metadata-curator-20160924201018  metadata-curator-20161001200943  snapshot-curator-20160925200938  snapshot-curator-20161003200941
metadata-curator-20160924202529  metadata-curator-20161002201721  snapshot-curator-20160925201005  snapshot-curator-20161004202529
metadata-curator-20160925200938  metadata-curator-20161003200941  snapshot-curator-20160926200807
metadata-curator-20160925201005  metadata-curator-20161004202529  snapshot-curator-20160926200947

[rkennedy]

What are the permissions on the folder? ls -al /var/backup

[vmesquita]

Code: Select all

# ls -al /var/backup
total 22
drwxrwx---   1 nagios nagios    0 Oct  7 11:42 .
drwxr-xr-x. 24 root   root   4096 Aug 16 14:42 ..
-rwxrwx---   0 nagios nagios  222 Oct  4 17:26 index
drwxrwx---   0 nagios nagios    0 Oct  4 17:25 indices
-rwxrwx---   0 nagios nagios  458 Sep 20 18:14 metadata-curator-20160920211440
-rwxrwx---   0 nagios nagios  458 Sep 21 17:08 metadata-curator-20160921200856
-rwxrwx---   0 nagios nagios  458 Sep 22 17:07 metadata-curator-20160922200730
-rwxrwx---   0 nagios nagios  458 Sep 23 17:41 metadata-curator-20160923204131
-rwxrwx---   0 nagios nagios  458 Sep 24 17:10 metadata-curator-20160924201018
-rwxrwx---   0 nagios nagios  458 Sep 24 17:25 metadata-curator-20160924202529
-rwxrwx---   0 nagios nagios  458 Sep 25 17:09 metadata-curator-20160925200938
-rwxrwx---   0 nagios nagios  458 Sep 25 17:10 metadata-curator-20160925201005
-rwxrwx---   0 nagios nagios  458 Sep 26 17:08 metadata-curator-20160926200807
-rwxrwx---   0 nagios nagios  458 Sep 26 17:09 metadata-curator-20160926200947
-rwxrwx---   0 nagios nagios  458 Sep 27 17:07 metadata-curator-20160927200725
-rwxrwx---   0 nagios nagios  457 Sep 28 17:22 metadata-curator-20160928202248
-rwxrwx---   0 nagios nagios  458 Sep 29 17:10 metadata-curator-20160929201008
-rwxrwx---   0 nagios nagios  458 Sep 30 17:07 metadata-curator-20160930200746
-rwxrwx---   0 nagios nagios  458 Oct  1 17:09 metadata-curator-20161001200943
-rwxrwx---   0 nagios nagios  458 Oct  2 17:17 metadata-curator-20161002201721
-rwxrwx---   0 nagios nagios  458 Oct  3 17:09 metadata-curator-20161003200941
-rwxrwx---   0 nagios nagios  458 Oct  4 17:25 metadata-curator-20161004202529
-rwxrwx---   0 nagios nagios  236 Sep 20 18:16 snapshot-curator-20160920211440
-rwxrwx---   0 nagios nagios  241 Sep 21 17:09 snapshot-curator-20160921200856
-rwxrwx---   0 nagios nagios  244 Sep 22 17:08 snapshot-curator-20160922200730
-rwxrwx---   0 nagios nagios  243 Sep 23 17:43 snapshot-curator-20160923204131
-rwxrwx---   0 nagios nagios  241 Sep 24 17:12 snapshot-curator-20160924201018
-rwxrwx---   0 nagios nagios  243 Sep 24 17:29 snapshot-curator-20160924202529
-rwxrwx---   0 nagios nagios  242 Sep 25 17:11 snapshot-curator-20160925200938
-rwxrwx---   0 nagios nagios  241 Sep 25 17:12 snapshot-curator-20160925201005
-rwxrwx---   0 nagios nagios  242 Sep 26 17:08 snapshot-curator-20160926200807
-rwxrwx---   0 nagios nagios  241 Sep 26 17:11 snapshot-curator-20160926200947
-rwxrwx---   0 nagios nagios  240 Sep 27 17:07 snapshot-curator-20160927200725
-rwxrwx---   0 nagios nagios  241 Sep 28 17:24 snapshot-curator-20160928202248
-rwxrwx---   0 nagios nagios  239 Sep 29 17:11 snapshot-curator-20160929201008
-rwxrwx---   0 nagios nagios  241 Sep 30 17:09 snapshot-curator-20160930200746
-rwxrwx---   0 nagios nagios  241 Oct  1 17:13 snapshot-curator-20161001200943
-rwxrwx---   0 nagios nagios  245 Oct  2 17:20 snapshot-curator-20161002201721
-rwxrwx---   0 nagios nagios  248 Oct  3 17:10 snapshot-curator-20161003200941
-rwxrwx---   0 nagios nagios    0 Oct  4 17:26 snapshot-curator-20161004202529

[rkennedy]

Permissions look good, I wonder if you're hitting a memory throttle.

Could you post your curator log for us to look at? /usr/local/nagioslogserver/var/jobs.log

[vmesquita]

This logfile seem to be empty. I tried running the command again, but I got the same result.

[mcapra]

I suspect one of the snapshots is corrupted or there's some bad metadata being kept about the snapshots in general.

Let's see the outputs of:

Code: Select all

curator --debug snapshot --repository Backup indices --older-than 1 --time-unit days --timestring %Y.%m.%d
curl -XGET "http://localhost:9200/_snapshot/Backup/_all?pretty"

[vmesquita]

I attached the output of the first line, the second line output is below:

Code: Select all

# curl -XGET "http://localhost:9200/_snapshot/Backup/_all?pretty"
{
  "error" : "ElasticsearchParseException[Failed to derive xcontent from (offset=0, length=0): []]",
  "status" : 400
}

[mcapra]

It definitely looks like the repository contains corrupted data.

I would suggest creating a fresh repository and scheduling a snapshot. If that snapshot succeeds against the fresh repository, then this confirms my suspicions. If the snapshot still fails against a fresh repository, I would like to see a fresh copy of the elasticsearch logs (/var/log/elasticsearch/*.log).

[vmesquita]

How do I create a fresh snapshot? Should I just delete the files on the backup folder like I did before?