Snapshots

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
teirekos
Posts: 110
Joined: Wed Nov 26, 2014 6:06 am

Snapshots

Post by teirekos »

In Backup&Maintenance I have select to Delete Backup older that 10 days. Why do I have 36 logstash snaphots? I attach a screenshot.
Top
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: Snapshots

Post by tmcdonald »

What Logserver version is this?
Former Nagios employee
Top
teirekos
Posts: 110
Joined: Wed Nov 26, 2014 6:06 am

Re: Snapshots

Post by teirekos »

latest 2015R1.4
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Snapshots

Post by jolson »

Please perform the following procedure:

SSH into Nagios Log Server on all of your nodes, and run this command:

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log
Once you are tailing jobs.log on every node, run a 'backup & maintenance job' from the NLS Web GUI:
2015-05-26 09_17_06-Command Subsystem • Nagios Log Server.png
One of your nodes will pick up the job and begin running a backup. Please report the output of that backup process here - it may give us some insight into why your backups are failing to delete old snapshots.
You do not have the required permissions to view the files attached to this post.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
teirekos
Posts: 110
Joined: Wed Nov 26, 2014 6:06 am

Re: Snapshots

Post by teirekos »

a txt is attached
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Snapshots

Post by jolson »

It's interesting that our jobs are failing to automatically purge your old backups. What happens if you attempt to delete them manually? Run this on the CLI:

Code: Select all

curator snapshot --delete-older-than 30 --repository "name of repository"
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
teirekos
Posts: 110
Joined: Wed Nov 26, 2014 6:06 am

Re: Snapshots

Post by teirekos »

[root@NagiosLogServer elasticsearch]# curator snapshot --delete-older-than 30 --repository NLSSnaps
2015-05-27 09:50:42,969 INFO Job starting...
2015-05-27 09:50:42,972 INFO Beginning SNAPSHOT operations...
2015-05-27 09:50:42,997 INFO logstash-2015.05.19 is within the threshold period (30 days).
2015-05-27 09:50:42,997 INFO logstash-2015.05.20 is within the threshold period (30 days).
2015-05-27 09:50:42,997 INFO logstash-2015.05.21 is within the threshold period (30 days).
2015-05-27 09:50:42,997 INFO logstash-2015.05.22 is within the threshold period (30 days).
2015-05-27 09:50:42,998 INFO logstash-2015.05.23 is within the threshold period (30 days).
2015-05-27 09:50:42,998 INFO logstash-2015.05.24 is within the threshold period (30 days).
2015-05-27 09:50:42,998 INFO Attempting to delete snapshot for index logstash-2015.04.26.
Traceback (most recent call last):
File "/usr/bin/curator", line 9, in <module>
load_entry_point('elasticsearch-curator==1.2.2', 'console_scripts', 'curator')()
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 731, in main
arguments.func(client, **argdict)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 585, in command_loop
skipped = op(client, index_name, **kwargs)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 424, in _delete_snapshot
client.snapshot.delete(repository=kwargs['repository'], snapshot=snap_name)
File "/usr/lib/python2.6/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.6/site-packages/elasticsearch/client/snapshot.py", line 34, in delete
_make_path('_snapshot', repository, snapshot), params=params)
File "/usr/lib/python2.6/site-packages/elasticsearch/transport.py", line 284, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.6/site-packages/elasticsearch/connection/http_urllib3.py", line 55, in perform_request
self._raise_error(response.status, raw_data)
File "/usr/lib/python2.6/site-packages/elasticsearch/connection/base.py", line 97, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.TransportError: TransportError(500, u'RemoteTransportException[[845bc07c-ed91-4920-8e23-747c9cc699f5][inet[/10.1.11.11:9300]][cluster/snapshot/delete]]; nested: RepositoryException[[NLSSnaps] failed to update snapshot in repository]; nested: SyncFailedException[sync failed]; ')
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Snapshots

Post by jolson »

failed to update snapshot in repository
I can't find anything about this error, but it's possible that the 'nagios' user cannot delete the snapshots in question. Are you certain your repository has proper permissions?

Code: Select all

ls -ld /NLSBackup

Code: Select all

ls -l /NLSBackup

Code: Select all

cat /etc/fstab
You could try logging in as the nagios user and writing a file, followed by deleting it:
su - nagios

Code: Select all

touch /NLSBackup/testfile
rm /NLSBackup/testfile
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
teirekos
Posts: 110
Joined: Wed Nov 26, 2014 6:06 am

Re: Snapshots

Post by teirekos »

I was able to create and rm the file as nagios user. Below is the rest info...

[root@NagiosLogServer ~]# ls -ld /NLSBackup
drwxrwxr-x 3 nagios nagios 4096 May 27 09:51 /NLSBackup
[root@NagiosLogServer ~]# ls -l /NLSBackup
total 208
drwxr-xr-x 80 nagios nagios 4096 May 26 14:53 indices
-rw-r--r-- 1 nagios users 61 May 26 14:20 metadata-logstash-2015.04.28
-rw-r--r-- 1 nagios users 61 May 26 14:24 metadata-logstash-2015.04.29
-rw-r--r-- 1 nagios users 61 May 26 14:28 metadata-logstash-2015.04.30
-rw-r--r-- 1 nagios users 61 May 26 14:30 metadata-logstash-2015.05.01
-rw-r--r-- 1 nagios users 61 May 26 14:31 metadata-logstash-2015.05.02
-rw-r--r-- 1 nagios users 61 May 26 14:31 metadata-logstash-2015.05.03
-rw-r--r-- 1 nagios users 61 May 26 14:32 metadata-logstash-2015.05.04
-rw-r--r-- 1 nagios users 61 May 26 14:33 metadata-logstash-2015.05.05
-rw-r--r-- 1 nagios users 61 May 26 14:34 metadata-logstash-2015.05.06
-rw-r--r-- 1 nagios users 61 May 26 14:36 metadata-logstash-2015.05.07
-rw-r--r-- 1 nagios users 61 May 26 14:38 metadata-logstash-2015.05.08
-rw-r--r-- 1 nagios users 61 May 26 14:40 metadata-logstash-2015.05.09
-rw-r--r-- 1 nagios users 61 May 26 14:41 metadata-logstash-2015.05.10
-rw-r--r-- 1 nagios users 61 May 26 14:42 metadata-logstash-2015.05.11
-rw-r--r-- 1 nagios users 61 May 26 14:43 metadata-logstash-2015.05.12
-rw-r--r-- 1 nagios users 61 May 26 14:45 metadata-logstash-2015.05.13
-rw-r--r-- 1 nagios users 61 May 26 14:47 metadata-logstash-2015.05.14
-rw-r--r-- 1 nagios users 61 May 26 14:49 metadata-logstash-2015.05.15
-rw-r--r-- 1 nagios users 61 May 26 14:52 metadata-logstash-2015.05.16
-rw-r--r-- 1 nagios users 61 May 20 07:51 metadata-logstash-2015.05.19
-rw-r--r-- 1 nagios users 61 May 22 08:23 metadata-logstash-2015.05.20
-rw-r--r-- 1 nagios users 61 May 22 08:24 metadata-logstash-2015.05.21
-rw-r--r-- 1 nagios users 61 May 23 07:50 metadata-logstash-2015.05.22
-rw-r--r-- 1 nagios users 61 May 24 07:57 metadata-logstash-2015.05.23
-rw-r--r-- 1 nagios users 61 May 25 07:51 metadata-logstash-2015.05.24
-rw-r--r-- 1 nagios users 61 May 26 14:53 metadata-logstash-2015.05.25
-rw-r--r-- 1 nagios users 190 May 26 14:24 snapshot-logstash-2015.04.28
-rw-r--r-- 1 nagios users 190 May 26 14:28 snapshot-logstash-2015.04.29
-rw-r--r-- 1 nagios users 190 May 26 14:30 snapshot-logstash-2015.04.30
-rw-r--r-- 1 nagios users 190 May 26 14:31 snapshot-logstash-2015.05.01
-rw-r--r-- 1 nagios users 189 May 26 14:31 snapshot-logstash-2015.05.02
-rw-r--r-- 1 nagios users 189 May 26 14:32 snapshot-logstash-2015.05.03
-rw-r--r-- 1 nagios users 192 May 26 14:33 snapshot-logstash-2015.05.04
-rw-r--r-- 1 nagios users 188 May 26 14:34 snapshot-logstash-2015.05.05
-rw-r--r-- 1 nagios users 190 May 26 14:36 snapshot-logstash-2015.05.06
-rw-r--r-- 1 nagios users 190 May 26 14:38 snapshot-logstash-2015.05.07
-rw-r--r-- 1 nagios users 190 May 26 14:40 snapshot-logstash-2015.05.08
-rw-r--r-- 1 nagios users 189 May 26 14:41 snapshot-logstash-2015.05.09
-rw-r--r-- 1 nagios users 190 May 26 14:42 snapshot-logstash-2015.05.10
-rw-r--r-- 1 nagios users 189 May 26 14:43 snapshot-logstash-2015.05.11
-rw-r--r-- 1 nagios users 190 May 26 14:45 snapshot-logstash-2015.05.12
-rw-r--r-- 1 nagios users 190 May 26 14:47 snapshot-logstash-2015.05.13
-rw-r--r-- 1 nagios users 190 May 26 14:49 snapshot-logstash-2015.05.14
-rw-r--r-- 1 nagios users 191 May 26 14:52 snapshot-logstash-2015.05.15
-rw-r--r-- 1 nagios users 190 May 26 14:53 snapshot-logstash-2015.05.16
-rw-r--r-- 1 nagios users 190 May 20 07:52 snapshot-logstash-2015.05.19
-rw-r--r-- 1 nagios users 188 May 22 08:24 snapshot-logstash-2015.05.20
-rw-r--r-- 1 nagios users 190 May 22 08:25 snapshot-logstash-2015.05.21
-rw-r--r-- 1 nagios users 189 May 23 07:51 snapshot-logstash-2015.05.22
-rw-r--r-- 1 nagios users 190 May 24 07:59 snapshot-logstash-2015.05.23
-rw-r--r-- 1 nagios users 189 May 25 07:52 snapshot-logstash-2015.05.24
[root@NagiosLogServer ~]#
[root@NagiosLogServer ~]# cat /etc/fstab
LABEL=79d3d2d4 / ext4 defaults,noatime 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
/swapfile1 swap swap defaults 0 0
/dev/sdb /NLSBackup ext3 defaults 1 2
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Snapshots

Post by jolson »

Do you have a single node in this cluster, or are there multiple?
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
Locked

Return to “Nagios Log Server”