Page 4 of 7
Re: Log Server does not create backups
Posted: Fri Oct 07, 2016 10:05 am
by vmesquita
I attached to this message.
Re: Log Server does not create backups
Posted: Fri Oct 07, 2016 10:11 am
by vmesquita
Also the backup files seem to still be there:
Code: Select all
# ls /var/backup/
index metadata-curator-20160926200807 snapshot-curator-20160920211440 snapshot-curator-20160927200725
indices metadata-curator-20160926200947 snapshot-curator-20160921200856 snapshot-curator-20160928202248
metadata-curator-20160920211440 metadata-curator-20160927200725 snapshot-curator-20160922200730 snapshot-curator-20160929201008
metadata-curator-20160921200856 metadata-curator-20160928202248 snapshot-curator-20160923204131 snapshot-curator-20160930200746
metadata-curator-20160922200730 metadata-curator-20160929201008 snapshot-curator-20160924201018 snapshot-curator-20161001200943
metadata-curator-20160923204131 metadata-curator-20160930200746 snapshot-curator-20160924202529 snapshot-curator-20161002201721
metadata-curator-20160924201018 metadata-curator-20161001200943 snapshot-curator-20160925200938 snapshot-curator-20161003200941
metadata-curator-20160924202529 metadata-curator-20161002201721 snapshot-curator-20160925201005 snapshot-curator-20161004202529
metadata-curator-20160925200938 metadata-curator-20161003200941 snapshot-curator-20160926200807
metadata-curator-20160925201005 metadata-curator-20161004202529 snapshot-curator-20160926200947
Re: Log Server does not create backups
Posted: Fri Oct 07, 2016 12:38 pm
by rkennedy
What are the permissions on the folder? ls -al /var/backup
Re: Log Server does not create backups
Posted: Fri Oct 07, 2016 1:00 pm
by vmesquita
Code: Select all
# ls -al /var/backup
total 22
drwxrwx--- 1 nagios nagios 0 Oct 7 11:42 .
drwxr-xr-x. 24 root root 4096 Aug 16 14:42 ..
-rwxrwx--- 0 nagios nagios 222 Oct 4 17:26 index
drwxrwx--- 0 nagios nagios 0 Oct 4 17:25 indices
-rwxrwx--- 0 nagios nagios 458 Sep 20 18:14 metadata-curator-20160920211440
-rwxrwx--- 0 nagios nagios 458 Sep 21 17:08 metadata-curator-20160921200856
-rwxrwx--- 0 nagios nagios 458 Sep 22 17:07 metadata-curator-20160922200730
-rwxrwx--- 0 nagios nagios 458 Sep 23 17:41 metadata-curator-20160923204131
-rwxrwx--- 0 nagios nagios 458 Sep 24 17:10 metadata-curator-20160924201018
-rwxrwx--- 0 nagios nagios 458 Sep 24 17:25 metadata-curator-20160924202529
-rwxrwx--- 0 nagios nagios 458 Sep 25 17:09 metadata-curator-20160925200938
-rwxrwx--- 0 nagios nagios 458 Sep 25 17:10 metadata-curator-20160925201005
-rwxrwx--- 0 nagios nagios 458 Sep 26 17:08 metadata-curator-20160926200807
-rwxrwx--- 0 nagios nagios 458 Sep 26 17:09 metadata-curator-20160926200947
-rwxrwx--- 0 nagios nagios 458 Sep 27 17:07 metadata-curator-20160927200725
-rwxrwx--- 0 nagios nagios 457 Sep 28 17:22 metadata-curator-20160928202248
-rwxrwx--- 0 nagios nagios 458 Sep 29 17:10 metadata-curator-20160929201008
-rwxrwx--- 0 nagios nagios 458 Sep 30 17:07 metadata-curator-20160930200746
-rwxrwx--- 0 nagios nagios 458 Oct 1 17:09 metadata-curator-20161001200943
-rwxrwx--- 0 nagios nagios 458 Oct 2 17:17 metadata-curator-20161002201721
-rwxrwx--- 0 nagios nagios 458 Oct 3 17:09 metadata-curator-20161003200941
-rwxrwx--- 0 nagios nagios 458 Oct 4 17:25 metadata-curator-20161004202529
-rwxrwx--- 0 nagios nagios 236 Sep 20 18:16 snapshot-curator-20160920211440
-rwxrwx--- 0 nagios nagios 241 Sep 21 17:09 snapshot-curator-20160921200856
-rwxrwx--- 0 nagios nagios 244 Sep 22 17:08 snapshot-curator-20160922200730
-rwxrwx--- 0 nagios nagios 243 Sep 23 17:43 snapshot-curator-20160923204131
-rwxrwx--- 0 nagios nagios 241 Sep 24 17:12 snapshot-curator-20160924201018
-rwxrwx--- 0 nagios nagios 243 Sep 24 17:29 snapshot-curator-20160924202529
-rwxrwx--- 0 nagios nagios 242 Sep 25 17:11 snapshot-curator-20160925200938
-rwxrwx--- 0 nagios nagios 241 Sep 25 17:12 snapshot-curator-20160925201005
-rwxrwx--- 0 nagios nagios 242 Sep 26 17:08 snapshot-curator-20160926200807
-rwxrwx--- 0 nagios nagios 241 Sep 26 17:11 snapshot-curator-20160926200947
-rwxrwx--- 0 nagios nagios 240 Sep 27 17:07 snapshot-curator-20160927200725
-rwxrwx--- 0 nagios nagios 241 Sep 28 17:24 snapshot-curator-20160928202248
-rwxrwx--- 0 nagios nagios 239 Sep 29 17:11 snapshot-curator-20160929201008
-rwxrwx--- 0 nagios nagios 241 Sep 30 17:09 snapshot-curator-20160930200746
-rwxrwx--- 0 nagios nagios 241 Oct 1 17:13 snapshot-curator-20161001200943
-rwxrwx--- 0 nagios nagios 245 Oct 2 17:20 snapshot-curator-20161002201721
-rwxrwx--- 0 nagios nagios 248 Oct 3 17:10 snapshot-curator-20161003200941
-rwxrwx--- 0 nagios nagios 0 Oct 4 17:26 snapshot-curator-20161004202529
Re: Log Server does not create backups
Posted: Fri Oct 07, 2016 1:22 pm
by rkennedy
Permissions look good, I wonder if you're hitting a memory throttle.
Could you post your curator log for us to look at? /usr/local/nagioslogserver/var/jobs.log
Re: Log Server does not create backups
Posted: Fri Oct 07, 2016 2:42 pm
by vmesquita
This logfile seem to be empty. I tried running the command again, but I got the same result.
Re: Log Server does not create backups
Posted: Mon Oct 10, 2016 9:17 am
by mcapra
I suspect one of the snapshots is corrupted or there's some bad metadata being kept about the snapshots in general.
Let's see the outputs of:
Code: Select all
curator --debug snapshot --repository Backup indices --older-than 1 --time-unit days --timestring %Y.%m.%d
curl -XGET "http://localhost:9200/_snapshot/Backup/_all?pretty"
Re: Log Server does not create backups
Posted: Tue Oct 11, 2016 11:11 am
by vmesquita
I attached the output of the first line, the second line output is below:
Code: Select all
# curl -XGET "http://localhost:9200/_snapshot/Backup/_all?pretty"
{
"error" : "ElasticsearchParseException[Failed to derive xcontent from (offset=0, length=0): []]",
"status" : 400
}
Re: Log Server does not create backups
Posted: Tue Oct 11, 2016 2:10 pm
by mcapra
It definitely looks like the repository contains corrupted data.
I would suggest creating a fresh repository and scheduling a snapshot. If that snapshot succeeds against the fresh repository, then this confirms my suspicions. If the snapshot still fails against a fresh repository, I would like to see a fresh copy of the elasticsearch logs (/var/log/elasticsearch/*.log).
Re: Log Server does not create backups
Posted: Fri Oct 14, 2016 11:09 am
by vmesquita
How do I create a fresh snapshot? Should I just delete the files on the backup folder like I did before?