Page 1 of 3

Snapshots

Posted: Tue May 26, 2015 8:03 am
by teirekos
In Backup&Maintenance I have select to Delete Backup older that 10 days. Why do I have 36 logstash snaphots? I attach a screenshot.

Re: Snapshots

Posted: Tue May 26, 2015 9:00 am
by tmcdonald
What Logserver version is this?

Re: Snapshots

Posted: Tue May 26, 2015 9:04 am
by teirekos
latest 2015R1.4

Re: Snapshots

Posted: Tue May 26, 2015 9:19 am
by jolson
Please perform the following procedure:

SSH into Nagios Log Server on all of your nodes, and run this command:

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log
Once you are tailing jobs.log on every node, run a 'backup & maintenance job' from the NLS Web GUI:
2015-05-26 09_17_06-Command Subsystem • Nagios Log Server.png
One of your nodes will pick up the job and begin running a backup. Please report the output of that backup process here - it may give us some insight into why your backups are failing to delete old snapshots.

Re: Snapshots

Posted: Tue May 26, 2015 9:23 am
by teirekos
a txt is attached

Re: Snapshots

Posted: Tue May 26, 2015 1:43 pm
by jolson
It's interesting that our jobs are failing to automatically purge your old backups. What happens if you attempt to delete them manually? Run this on the CLI:

Code: Select all

curator snapshot --delete-older-than 30 --repository "name of repository"

Re: Snapshots

Posted: Wed May 27, 2015 1:52 am
by teirekos
[root@NagiosLogServer elasticsearch]# curator snapshot --delete-older-than 30 --repository NLSSnaps
2015-05-27 09:50:42,969 INFO Job starting...
2015-05-27 09:50:42,972 INFO Beginning SNAPSHOT operations...
2015-05-27 09:50:42,997 INFO logstash-2015.05.19 is within the threshold period (30 days).
2015-05-27 09:50:42,997 INFO logstash-2015.05.20 is within the threshold period (30 days).
2015-05-27 09:50:42,997 INFO logstash-2015.05.21 is within the threshold period (30 days).
2015-05-27 09:50:42,997 INFO logstash-2015.05.22 is within the threshold period (30 days).
2015-05-27 09:50:42,998 INFO logstash-2015.05.23 is within the threshold period (30 days).
2015-05-27 09:50:42,998 INFO logstash-2015.05.24 is within the threshold period (30 days).
2015-05-27 09:50:42,998 INFO Attempting to delete snapshot for index logstash-2015.04.26.
Traceback (most recent call last):
File "/usr/bin/curator", line 9, in <module>
load_entry_point('elasticsearch-curator==1.2.2', 'console_scripts', 'curator')()
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 731, in main
arguments.func(client, **argdict)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 585, in command_loop
skipped = op(client, index_name, **kwargs)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 424, in _delete_snapshot
client.snapshot.delete(repository=kwargs['repository'], snapshot=snap_name)
File "/usr/lib/python2.6/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.6/site-packages/elasticsearch/client/snapshot.py", line 34, in delete
_make_path('_snapshot', repository, snapshot), params=params)
File "/usr/lib/python2.6/site-packages/elasticsearch/transport.py", line 284, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.6/site-packages/elasticsearch/connection/http_urllib3.py", line 55, in perform_request
self._raise_error(response.status, raw_data)
File "/usr/lib/python2.6/site-packages/elasticsearch/connection/base.py", line 97, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.TransportError: TransportError(500, u'RemoteTransportException[[845bc07c-ed91-4920-8e23-747c9cc699f5][inet[/10.1.11.11:9300]][cluster/snapshot/delete]]; nested: RepositoryException[[NLSSnaps] failed to update snapshot in repository]; nested: SyncFailedException[sync failed]; ')

Re: Snapshots

Posted: Wed May 27, 2015 9:23 am
by jolson
failed to update snapshot in repository
I can't find anything about this error, but it's possible that the 'nagios' user cannot delete the snapshots in question. Are you certain your repository has proper permissions?

Code: Select all

ls -ld /NLSBackup

Code: Select all

ls -l /NLSBackup

Code: Select all

cat /etc/fstab
You could try logging in as the nagios user and writing a file, followed by deleting it:
su - nagios

Code: Select all

touch /NLSBackup/testfile
rm /NLSBackup/testfile

Re: Snapshots

Posted: Wed May 27, 2015 10:34 am
by teirekos
I was able to create and rm the file as nagios user. Below is the rest info...

[root@NagiosLogServer ~]# ls -ld /NLSBackup
drwxrwxr-x 3 nagios nagios 4096 May 27 09:51 /NLSBackup
[root@NagiosLogServer ~]# ls -l /NLSBackup
total 208
drwxr-xr-x 80 nagios nagios 4096 May 26 14:53 indices
-rw-r--r-- 1 nagios users 61 May 26 14:20 metadata-logstash-2015.04.28
-rw-r--r-- 1 nagios users 61 May 26 14:24 metadata-logstash-2015.04.29
-rw-r--r-- 1 nagios users 61 May 26 14:28 metadata-logstash-2015.04.30
-rw-r--r-- 1 nagios users 61 May 26 14:30 metadata-logstash-2015.05.01
-rw-r--r-- 1 nagios users 61 May 26 14:31 metadata-logstash-2015.05.02
-rw-r--r-- 1 nagios users 61 May 26 14:31 metadata-logstash-2015.05.03
-rw-r--r-- 1 nagios users 61 May 26 14:32 metadata-logstash-2015.05.04
-rw-r--r-- 1 nagios users 61 May 26 14:33 metadata-logstash-2015.05.05
-rw-r--r-- 1 nagios users 61 May 26 14:34 metadata-logstash-2015.05.06
-rw-r--r-- 1 nagios users 61 May 26 14:36 metadata-logstash-2015.05.07
-rw-r--r-- 1 nagios users 61 May 26 14:38 metadata-logstash-2015.05.08
-rw-r--r-- 1 nagios users 61 May 26 14:40 metadata-logstash-2015.05.09
-rw-r--r-- 1 nagios users 61 May 26 14:41 metadata-logstash-2015.05.10
-rw-r--r-- 1 nagios users 61 May 26 14:42 metadata-logstash-2015.05.11
-rw-r--r-- 1 nagios users 61 May 26 14:43 metadata-logstash-2015.05.12
-rw-r--r-- 1 nagios users 61 May 26 14:45 metadata-logstash-2015.05.13
-rw-r--r-- 1 nagios users 61 May 26 14:47 metadata-logstash-2015.05.14
-rw-r--r-- 1 nagios users 61 May 26 14:49 metadata-logstash-2015.05.15
-rw-r--r-- 1 nagios users 61 May 26 14:52 metadata-logstash-2015.05.16
-rw-r--r-- 1 nagios users 61 May 20 07:51 metadata-logstash-2015.05.19
-rw-r--r-- 1 nagios users 61 May 22 08:23 metadata-logstash-2015.05.20
-rw-r--r-- 1 nagios users 61 May 22 08:24 metadata-logstash-2015.05.21
-rw-r--r-- 1 nagios users 61 May 23 07:50 metadata-logstash-2015.05.22
-rw-r--r-- 1 nagios users 61 May 24 07:57 metadata-logstash-2015.05.23
-rw-r--r-- 1 nagios users 61 May 25 07:51 metadata-logstash-2015.05.24
-rw-r--r-- 1 nagios users 61 May 26 14:53 metadata-logstash-2015.05.25
-rw-r--r-- 1 nagios users 190 May 26 14:24 snapshot-logstash-2015.04.28
-rw-r--r-- 1 nagios users 190 May 26 14:28 snapshot-logstash-2015.04.29
-rw-r--r-- 1 nagios users 190 May 26 14:30 snapshot-logstash-2015.04.30
-rw-r--r-- 1 nagios users 190 May 26 14:31 snapshot-logstash-2015.05.01
-rw-r--r-- 1 nagios users 189 May 26 14:31 snapshot-logstash-2015.05.02
-rw-r--r-- 1 nagios users 189 May 26 14:32 snapshot-logstash-2015.05.03
-rw-r--r-- 1 nagios users 192 May 26 14:33 snapshot-logstash-2015.05.04
-rw-r--r-- 1 nagios users 188 May 26 14:34 snapshot-logstash-2015.05.05
-rw-r--r-- 1 nagios users 190 May 26 14:36 snapshot-logstash-2015.05.06
-rw-r--r-- 1 nagios users 190 May 26 14:38 snapshot-logstash-2015.05.07
-rw-r--r-- 1 nagios users 190 May 26 14:40 snapshot-logstash-2015.05.08
-rw-r--r-- 1 nagios users 189 May 26 14:41 snapshot-logstash-2015.05.09
-rw-r--r-- 1 nagios users 190 May 26 14:42 snapshot-logstash-2015.05.10
-rw-r--r-- 1 nagios users 189 May 26 14:43 snapshot-logstash-2015.05.11
-rw-r--r-- 1 nagios users 190 May 26 14:45 snapshot-logstash-2015.05.12
-rw-r--r-- 1 nagios users 190 May 26 14:47 snapshot-logstash-2015.05.13
-rw-r--r-- 1 nagios users 190 May 26 14:49 snapshot-logstash-2015.05.14
-rw-r--r-- 1 nagios users 191 May 26 14:52 snapshot-logstash-2015.05.15
-rw-r--r-- 1 nagios users 190 May 26 14:53 snapshot-logstash-2015.05.16
-rw-r--r-- 1 nagios users 190 May 20 07:52 snapshot-logstash-2015.05.19
-rw-r--r-- 1 nagios users 188 May 22 08:24 snapshot-logstash-2015.05.20
-rw-r--r-- 1 nagios users 190 May 22 08:25 snapshot-logstash-2015.05.21
-rw-r--r-- 1 nagios users 189 May 23 07:51 snapshot-logstash-2015.05.22
-rw-r--r-- 1 nagios users 190 May 24 07:59 snapshot-logstash-2015.05.23
-rw-r--r-- 1 nagios users 189 May 25 07:52 snapshot-logstash-2015.05.24
[root@NagiosLogServer ~]#
[root@NagiosLogServer ~]# cat /etc/fstab
LABEL=79d3d2d4 / ext4 defaults,noatime 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
/swapfile1 swap swap defaults 0 0
/dev/sdb /NLSBackup ext3 defaults 1 2

Re: Snapshots

Posted: Wed May 27, 2015 12:38 pm
by jolson
Do you have a single node in this cluster, or are there multiple?