Page 1 of 3
Snapshots
Posted: Tue May 26, 2015 8:03 am
by teirekos
In Backup&Maintenance I have select to Delete Backup older that 10 days. Why do I have 36 logstash snaphots? I attach a screenshot.
Re: Snapshots
Posted: Tue May 26, 2015 9:00 am
by tmcdonald
What Logserver version is this?
Re: Snapshots
Posted: Tue May 26, 2015 9:04 am
by teirekos
latest 2015R1.4
Re: Snapshots
Posted: Tue May 26, 2015 9:19 am
by jolson
Please perform the following procedure:
SSH into Nagios Log Server on all of your nodes, and run this command:
Code: Select all
tail -f /usr/local/nagioslogserver/var/jobs.log
Once you are tailing jobs.log on every node, run a 'backup & maintenance job' from the NLS Web GUI:
2015-05-26 09_17_06-Command Subsystem • Nagios Log Server.png
One of your nodes will pick up the job and begin running a backup. Please report the output of that backup process here - it may give us some insight into why your backups are failing to delete old snapshots.
Re: Snapshots
Posted: Tue May 26, 2015 9:23 am
by teirekos
a txt is attached
Re: Snapshots
Posted: Tue May 26, 2015 1:43 pm
by jolson
It's interesting that our jobs are failing to automatically purge your old backups. What happens if you attempt to delete them manually? Run this on the CLI:
Code: Select all
curator snapshot --delete-older-than 30 --repository "name of repository"
Re: Snapshots
Posted: Wed May 27, 2015 1:52 am
by teirekos
[root@NagiosLogServer elasticsearch]# curator snapshot --delete-older-than 30 --repository NLSSnaps
2015-05-27 09:50:42,969 INFO Job starting...
2015-05-27 09:50:42,972 INFO Beginning SNAPSHOT operations...
2015-05-27 09:50:42,997 INFO logstash-2015.05.19 is within the threshold period (30 days).
2015-05-27 09:50:42,997 INFO logstash-2015.05.20 is within the threshold period (30 days).
2015-05-27 09:50:42,997 INFO logstash-2015.05.21 is within the threshold period (30 days).
2015-05-27 09:50:42,997 INFO logstash-2015.05.22 is within the threshold period (30 days).
2015-05-27 09:50:42,998 INFO logstash-2015.05.23 is within the threshold period (30 days).
2015-05-27 09:50:42,998 INFO logstash-2015.05.24 is within the threshold period (30 days).
2015-05-27 09:50:42,998 INFO Attempting to delete snapshot for index logstash-2015.04.26.
Traceback (most recent call last):
File "/usr/bin/curator", line 9, in <module>
load_entry_point('elasticsearch-curator==1.2.2', 'console_scripts', 'curator')()
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 731, in main
arguments.func(client, **argdict)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 585, in command_loop
skipped = op(client, index_name, **kwargs)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 424, in _delete_snapshot
client.snapshot.delete(repository=kwargs['repository'], snapshot=snap_name)
File "/usr/lib/python2.6/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.6/site-packages/elasticsearch/client/snapshot.py", line 34, in delete
_make_path('_snapshot', repository, snapshot), params=params)
File "/usr/lib/python2.6/site-packages/elasticsearch/transport.py", line 284, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.6/site-packages/elasticsearch/connection/http_urllib3.py", line 55, in perform_request
self._raise_error(response.status, raw_data)
File "/usr/lib/python2.6/site-packages/elasticsearch/connection/base.py", line 97, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.TransportError: TransportError(500, u'RemoteTransportException[[845bc07c-ed91-4920-8e23-747c9cc699f5][inet[/10.1.11.11:9300]][cluster/snapshot/delete]]; nested: RepositoryException[[NLSSnaps] failed to update snapshot in repository]; nested: SyncFailedException[sync failed]; ')
Re: Snapshots
Posted: Wed May 27, 2015 9:23 am
by jolson
failed to update snapshot in repository
I can't find anything about this error, but it's possible that the 'nagios' user cannot delete the snapshots in question. Are you certain your repository has proper permissions?
You could try logging in as the nagios user and writing a file, followed by deleting it:
su - nagios
Code: Select all
touch /NLSBackup/testfile
rm /NLSBackup/testfile
Re: Snapshots
Posted: Wed May 27, 2015 10:34 am
by teirekos
I was able to create and rm the file as nagios user. Below is the rest info...
[root@NagiosLogServer ~]# ls -ld /NLSBackup
drwxrwxr-x 3 nagios nagios 4096 May 27 09:51 /NLSBackup
[root@NagiosLogServer ~]# ls -l /NLSBackup
total 208
drwxr-xr-x 80 nagios nagios 4096 May 26 14:53 indices
-rw-r--r-- 1 nagios users 61 May 26 14:20 metadata-logstash-2015.04.28
-rw-r--r-- 1 nagios users 61 May 26 14:24 metadata-logstash-2015.04.29
-rw-r--r-- 1 nagios users 61 May 26 14:28 metadata-logstash-2015.04.30
-rw-r--r-- 1 nagios users 61 May 26 14:30 metadata-logstash-2015.05.01
-rw-r--r-- 1 nagios users 61 May 26 14:31 metadata-logstash-2015.05.02
-rw-r--r-- 1 nagios users 61 May 26 14:31 metadata-logstash-2015.05.03
-rw-r--r-- 1 nagios users 61 May 26 14:32 metadata-logstash-2015.05.04
-rw-r--r-- 1 nagios users 61 May 26 14:33 metadata-logstash-2015.05.05
-rw-r--r-- 1 nagios users 61 May 26 14:34 metadata-logstash-2015.05.06
-rw-r--r-- 1 nagios users 61 May 26 14:36 metadata-logstash-2015.05.07
-rw-r--r-- 1 nagios users 61 May 26 14:38 metadata-logstash-2015.05.08
-rw-r--r-- 1 nagios users 61 May 26 14:40 metadata-logstash-2015.05.09
-rw-r--r-- 1 nagios users 61 May 26 14:41 metadata-logstash-2015.05.10
-rw-r--r-- 1 nagios users 61 May 26 14:42 metadata-logstash-2015.05.11
-rw-r--r-- 1 nagios users 61 May 26 14:43 metadata-logstash-2015.05.12
-rw-r--r-- 1 nagios users 61 May 26 14:45 metadata-logstash-2015.05.13
-rw-r--r-- 1 nagios users 61 May 26 14:47 metadata-logstash-2015.05.14
-rw-r--r-- 1 nagios users 61 May 26 14:49 metadata-logstash-2015.05.15
-rw-r--r-- 1 nagios users 61 May 26 14:52 metadata-logstash-2015.05.16
-rw-r--r-- 1 nagios users 61 May 20 07:51 metadata-logstash-2015.05.19
-rw-r--r-- 1 nagios users 61 May 22 08:23 metadata-logstash-2015.05.20
-rw-r--r-- 1 nagios users 61 May 22 08:24 metadata-logstash-2015.05.21
-rw-r--r-- 1 nagios users 61 May 23 07:50 metadata-logstash-2015.05.22
-rw-r--r-- 1 nagios users 61 May 24 07:57 metadata-logstash-2015.05.23
-rw-r--r-- 1 nagios users 61 May 25 07:51 metadata-logstash-2015.05.24
-rw-r--r-- 1 nagios users 61 May 26 14:53 metadata-logstash-2015.05.25
-rw-r--r-- 1 nagios users 190 May 26 14:24 snapshot-logstash-2015.04.28
-rw-r--r-- 1 nagios users 190 May 26 14:28 snapshot-logstash-2015.04.29
-rw-r--r-- 1 nagios users 190 May 26 14:30 snapshot-logstash-2015.04.30
-rw-r--r-- 1 nagios users 190 May 26 14:31 snapshot-logstash-2015.05.01
-rw-r--r-- 1 nagios users 189 May 26 14:31 snapshot-logstash-2015.05.02
-rw-r--r-- 1 nagios users 189 May 26 14:32 snapshot-logstash-2015.05.03
-rw-r--r-- 1 nagios users 192 May 26 14:33 snapshot-logstash-2015.05.04
-rw-r--r-- 1 nagios users 188 May 26 14:34 snapshot-logstash-2015.05.05
-rw-r--r-- 1 nagios users 190 May 26 14:36 snapshot-logstash-2015.05.06
-rw-r--r-- 1 nagios users 190 May 26 14:38 snapshot-logstash-2015.05.07
-rw-r--r-- 1 nagios users 190 May 26 14:40 snapshot-logstash-2015.05.08
-rw-r--r-- 1 nagios users 189 May 26 14:41 snapshot-logstash-2015.05.09
-rw-r--r-- 1 nagios users 190 May 26 14:42 snapshot-logstash-2015.05.10
-rw-r--r-- 1 nagios users 189 May 26 14:43 snapshot-logstash-2015.05.11
-rw-r--r-- 1 nagios users 190 May 26 14:45 snapshot-logstash-2015.05.12
-rw-r--r-- 1 nagios users 190 May 26 14:47 snapshot-logstash-2015.05.13
-rw-r--r-- 1 nagios users 190 May 26 14:49 snapshot-logstash-2015.05.14
-rw-r--r-- 1 nagios users 191 May 26 14:52 snapshot-logstash-2015.05.15
-rw-r--r-- 1 nagios users 190 May 26 14:53 snapshot-logstash-2015.05.16
-rw-r--r-- 1 nagios users 190 May 20 07:52 snapshot-logstash-2015.05.19
-rw-r--r-- 1 nagios users 188 May 22 08:24 snapshot-logstash-2015.05.20
-rw-r--r-- 1 nagios users 190 May 22 08:25 snapshot-logstash-2015.05.21
-rw-r--r-- 1 nagios users 189 May 23 07:51 snapshot-logstash-2015.05.22
-rw-r--r-- 1 nagios users 190 May 24 07:59 snapshot-logstash-2015.05.23
-rw-r--r-- 1 nagios users 189 May 25 07:52 snapshot-logstash-2015.05.24
[root@NagiosLogServer ~]#
[root@NagiosLogServer ~]# cat /etc/fstab
LABEL=79d3d2d4 / ext4 defaults,noatime 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
/swapfile1 swap swap defaults 0 0
/dev/sdb /NLSBackup ext3 defaults 1 2
Re: Snapshots
Posted: Wed May 27, 2015 12:38 pm
by jolson
Do you have a single node in this cluster, or are there multiple?