Nagios Support Forum

Hello,

My backup wasn't working for three or so weeks. Although, it's scheduled to run every night. And then passed couple of days it decided to work again.
What I have noticed, is that issues occur in the beginning of the month. Is it possible something with the underlying scheduling?

Here is reference to another thread that I had opened in regards my issues with backup I had before

http://support.nagios.com/forum/viewtop ... 37&t=31171

stecino,

The most common cause of your backups not firing is an improper/corrupt backup. If a backup did not fully complete, the backup process will stall. For instance, if the 'Delete backups older than' setting is set to three days, you may get a corrupt backup on day one - but on day 4 they may start working again because the corrupt backup can be skipped or deleted. This could explain some of the weirdness you are encountering. I will setup a test repository on my end to see if I have any issues with end of the month backups.

In the meantime, we may be able to get this working using a similar procedure as last time - tailing jobs.log and forcing a backup run to see if there are any errors.
Force the 'backup and maintenance' command from the Web GUI.

jolson wrote:stecino,

The most common cause of your backups not firing is an improper/corrupt backup. If a backup did not fully complete, the backup process will stall. For instance, if the 'Delete backups older than' setting is set to three days, you may get a corrupt backup on day one - but on day 4 they may start working again because the corrupt backup can be skipped or deleted. This could explain some of the weirdness you are encountering. I will setup a test repository on my end to see if I have any issues with end of the month backups.

In the meantime, we may be able to get this working using a similar procedure as last time - tailing jobs.log and forcing a backup run to see if there are any errors.

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log

Force the 'backup and maintenance' command from the Web GUI.

Forced a backup

Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command do_maintenance with args ' ' for job id: backup_maintenance
2015-05-22 14:04:02,920 INFO Job starting...
2015-05-22 14:04:02,928 INFO Beginning BLOOM operations...
2015-05-22 14:04:02,959 INFO Attempting to disable bloom filter for index logstash-2015.04.23.
2015-05-22 14:04:03,381 INFO logstash-2015.04.23: Successfully bloom filter disabled.
2015-05-22 14:04:03,382 INFO Attempting to disable bloom filter for index logstash-2015.04.24.
2015-05-22 14:04:03,820 INFO logstash-2015.04.24: Successfully bloom filter disabled.
2015-05-22 14:04:03,820 INFO Attempting to disable bloom filter for index logstash-2015.04.25.
2015-05-22 14:04:04,193 INFO logstash-2015.04.25: Successfully bloom filter disabled.
2015-05-22 14:04:04,194 INFO Attempting to disable bloom filter for index logstash-2015.04.26.
2015-05-22 14:04:04,581 INFO logstash-2015.04.26: Successfully bloom filter disabled.
2015-05-22 14:04:04,582 INFO Attempting to disable bloom filter for index logstash-2015.04.27.
2015-05-22 14:04:04,971 INFO logstash-2015.04.27: Successfully bloom filter disabled.
2015-05-22 14:04:04,971 INFO Attempting to disable bloom filter for index logstash-2015.04.28.
2015-05-22 14:04:05,340 INFO logstash-2015.04.28: Successfully bloom filter disabled.
2015-05-22 14:04:05,340 INFO Attempting to disable bloom filter for index logstash-2015.04.29.
2015-05-22 14:04:05,717 INFO logstash-2015.04.29: Successfully bloom filter disabled.
2015-05-22 14:04:05,718 INFO Attempting to disable bloom filter for index logstash-2015.04.30.
2015-05-22 14:04:06,083 INFO logstash-2015.04.30: Successfully bloom filter disabled.
2015-05-22 14:04:06,083 INFO Attempting to disable bloom filter for index logstash-2015.05.01.
2015-05-22 14:04:06,433 INFO logstash-2015.05.01: Successfully bloom filter disabled.
2015-05-22 14:04:06,433 INFO Attempting to disable bloom filter for index logstash-2015.05.02.
2015-05-22 14:04:06,803 INFO logstash-2015.05.02: Successfully bloom filter disabled.
2015-05-22 14:04:06,804 INFO Attempting to disable bloom filter for index logstash-2015.05.03.
2015-05-22 14:04:07,183 INFO logstash-2015.05.03: Successfully bloom filter disabled.
2015-05-22 14:04:07,184 INFO Attempting to disable bloom filter for index logstash-2015.05.04.
2015-05-22 14:04:07,535 INFO logstash-2015.05.04: Successfully bloom filter disabled.
2015-05-22 14:04:07,536 INFO Attempting to disable bloom filter for index logstash-2015.05.05.
2015-05-22 14:04:07,931 INFO logstash-2015.05.05: Successfully bloom filter disabled.
2015-05-22 14:04:07,932 INFO Attempting to disable bloom filter for index logstash-2015.05.06.
2015-05-22 14:04:08,302 INFO logstash-2015.05.06: Successfully bloom filter disabled.
2015-05-22 14:04:08,303 INFO Attempting to disable bloom filter for index logstash-2015.05.07.
2015-05-22 14:04:08,670 INFO logstash-2015.05.07: Successfully bloom filter disabled.
2015-05-22 14:04:08,671 INFO Attempting to disable bloom filter for index logstash-2015.05.08.
2015-05-22 14:04:09,044 INFO logstash-2015.05.08: Successfully bloom filter disabled.
2015-05-22 14:04:09,044 INFO Attempting to disable bloom filter for index logstash-2015.05.09.
2015-05-22 14:04:09,420 INFO logstash-2015.05.09: Successfully bloom filter disabled.
2015-05-22 14:04:09,420 INFO Attempting to disable bloom filter for index logstash-2015.05.10.
2015-05-22 14:04:09,791 INFO logstash-2015.05.10: Successfully bloom filter disabled.
2015-05-22 14:04:09,792 INFO Attempting to disable bloom filter for index logstash-2015.05.11.
2015-05-22 14:04:10,152 INFO logstash-2015.05.11: Successfully bloom filter disabled.
2015-05-22 14:04:10,152 INFO Attempting to disable bloom filter for index logstash-2015.05.12.
2015-05-22 14:04:10,532 INFO logstash-2015.05.12: Successfully bloom filter disabled.
2015-05-22 14:04:10,533 INFO Attempting to disable bloom filter for index logstash-2015.05.13.
2015-05-22 14:04:10,908 INFO logstash-2015.05.13: Successfully bloom filter disabled.
2015-05-22 14:04:10,909 INFO Attempting to disable bloom filter for index logstash-2015.05.20.
2015-05-22 14:04:11,111 INFO logstash-2015.05.20: Successfully bloom filter disabled.
2015-05-22 14:04:11,112 INFO Attempting to disable bloom filter for index logstash-2015.05.21.
2015-05-22 14:04:11,492 INFO logstash-2015.05.21: Successfully bloom filter disabled.
2015-05-22 14:04:11,493 INFO logstash-2015.05.22 is within the threshold period (1 days).
2015-05-22 14:04:11,493 INFO logstash-2015.05.23 is within the threshold period (1 days).
2015-05-22 14:04:11,493 INFO DISABLE BLOOM FILTER index operations completed.
2015-05-22 14:04:11,494 INFO Done in 0:00:08.604728.
2015-05-22 14:04:11,827 INFO Job starting...
2015-05-22 14:04:11,827 INFO Default timeout of 30 seconds is too low for command OPTIMIZE. Overriding to 21,600 seconds (6 hours).
2015-05-22 14:04:11,834 INFO Beginning OPTIMIZE operations...
2015-05-22 14:04:11,865 INFO Attempting to optimize index logstash-2015.04.23.
2015-05-22 14:04:11,908 INFO Job starting...
2015-05-22 14:04:11,915 INFO Beginning CLOSE operations...
2015-05-22 14:04:11,945 INFO logstash-2015.04.23 is within the threshold period (30 days).
2015-05-22 14:04:11,945 INFO logstash-2015.04.24 is within the threshold period (30 days).
2015-05-22 14:04:11,945 INFO logstash-2015.04.25 is within the threshold period (30 days).
2015-05-22 14:04:11,946 INFO logstash-2015.04.26 is within the threshold period (30 days).
2015-05-22 14:04:11,946 INFO logstash-2015.04.27 is within the threshold period (30 days).
2015-05-22 14:04:11,946 INFO logstash-2015.04.28 is within the threshold period (30 days).
2015-05-22 14:04:11,947 INFO logstash-2015.04.29 is within the threshold period (30 days).
2015-05-22 14:04:11,947 INFO logstash-2015.04.30 is within the threshold period (30 days).
2015-05-22 14:04:11,947 INFO logstash-2015.05.01 is within the threshold period (30 days).
2015-05-22 14:04:11,948 INFO logstash-2015.05.02 is within the threshold period (30 days).
2015-05-22 14:04:11,948 INFO logstash-2015.05.03 is within the threshold period (30 days).
2015-05-22 14:04:11,948 INFO logstash-2015.05.04 is within the threshold period (30 days).
2015-05-22 14:04:11,949 INFO logstash-2015.05.05 is within the threshold period (30 days).
2015-05-22 14:04:11,949 INFO logstash-2015.05.06 is within the threshold period (30 days).
2015-05-22 14:04:11,949 INFO logstash-2015.05.07 is within the threshold period (30 days).
2015-05-22 14:04:11,950 INFO logstash-2015.05.08 is within the threshold period (30 days).
2015-05-22 14:04:11,950 INFO logstash-2015.05.09 is within the threshold period (30 days).
2015-05-22 14:04:11,950 INFO logstash-2015.05.10 is within the threshold period (30 days).
2015-05-22 14:04:11,951 INFO logstash-2015.05.11 is within the threshold period (30 days).
2015-05-22 14:04:11,951 INFO logstash-2015.05.12 is within the threshold period (30 days).
2015-05-22 14:04:11,951 INFO logstash-2015.05.13 is within the threshold period (30 days).
2015-05-22 14:04:11,952 INFO logstash-2015.05.20 is within the threshold period (30 days).
2015-05-22 14:04:11,952 INFO logstash-2015.05.21 is within the threshold period (30 days).
2015-05-22 14:04:11,952 INFO logstash-2015.05.22 is within the threshold period (30 days).
2015-05-22 14:04:11,953 INFO logstash-2015.05.23 is within the threshold period (30 days).
2015-05-22 14:04:11,953 INFO CLOSE index operations completed.
2015-05-22 14:04:11,953 INFO Done in 0:00:00.071272.
2015-05-22 14:04:11,995 INFO Job starting...
2015-05-22 14:04:12,001 INFO Beginning DELETE operations...
SUCCESS
2015-05-22 14:04:12,032 INFO logstash-2015.04.23 is within the threshold period (30 days).
2015-05-22 14:04:12,033 INFO logstash-2015.04.24 is within the threshold period (30 days).
2015-05-22 14:04:12,033 INFO logstash-2015.04.25 is within the threshold period (30 days).
2015-05-22 14:04:12,034 INFO logstash-2015.04.26 is within the threshold period (30 days).
2015-05-22 14:04:12,034 INFO logstash-2015.04.27 is within the threshold period (30 days).
2015-05-22 14:04:12,034 INFO logstash-2015.04.28 is within the threshold period (30 days).
2015-05-22 14:04:12,035 INFO logstash-2015.04.29 is within the threshold period (30 days).
2015-05-22 14:04:12,035 INFO logstash-2015.04.30 is within the threshold period (30 days).
2015-05-22 14:04:12,035 INFO logstash-2015.05.01 is within the threshold period (30 days).
2015-05-22 14:04:12,036 INFO logstash-2015.05.02 is within the threshold period (30 days).
2015-05-22 14:04:12,036 INFO logstash-2015.05.03 is within the threshold period (30 days).
2015-05-22 14:04:12,036 INFO logstash-2015.05.04 is within the threshold period (30 days).
2015-05-22 14:04:12,037 INFO logstash-2015.05.05 is within the threshold period (30 days).
2015-05-22 14:04:12,037 INFO logstash-2015.05.06 is within the threshold period (30 days).
2015-05-22 14:04:12,037 INFO logstash-2015.05.07 is within the threshold period (30 days).
2015-05-22 14:04:12,038 INFO logstash-2015.05.08 is within the threshold period (30 days).
2015-05-22 14:04:12,038 INFO logstash-2015.05.09 is within the threshold period (30 days).
2015-05-22 14:04:12,038 INFO logstash-2015.05.10 is within the threshold period (30 days).
2015-05-22 14:04:12,039 INFO logstash-2015.05.11 is within the threshold period (30 days).
2015-05-22 14:04:12,039 INFO logstash-2015.05.12 is within the threshold period (30 days).
2015-05-22 14:04:12,039 INFO logstash-2015.05.13 is within the threshold period (30 days).
2015-05-22 14:04:12,039 INFO logstash-2015.05.20 is within the threshold period (30 days).
2015-05-22 14:04:12,040 INFO logstash-2015.05.21 is within the threshold period (30 days).
2015-05-22 14:04:12,040 INFO logstash-2015.05.22 is within the threshold period (30 days).
2015-05-22 14:04:12,040 INFO logstash-2015.05.23 is within the threshold period (30 days).
2015-05-22 14:04:12,041 INFO DELETE index operations completed.
2015-05-22 14:04:12,041 INFO Done in 0:00:00.072961.
2015-05-22 14:04:12,048 INFO Job starting...
2015-05-22 14:04:12,056 INFO Beginning SNAPSHOT operations...
2015-05-22 14:04:12,114 INFO Job starting...
2015-05-22 14:04:12,114 INFO Default timeout of 30 seconds is too low for command SNAPSHOT. Overriding to 21,600 seconds (6 hours).
2015-05-22 14:04:12,121 INFO Beginning SNAPSHOT operations...
2015-05-22 14:04:12,151 INFO Attempting to create snapshot for index logstash-2015.04.23.
2015-05-22 14:04:12,186 INFO Skipping index logstash-2015.04.23: Already optimized.
2015-05-22 14:04:12,186 INFO Attempting to optimize index logstash-2015.04.24.
2015-05-22 14:04:12,491 INFO Skipping index logstash-2015.04.24: Already optimized.
2015-05-22 14:04:12,492 INFO Attempting to optimize index logstash-2015.04.25.
2015-05-22 14:04:12,788 INFO Skipping index logstash-2015.04.25: Already optimized.
2015-05-22 14:04:12,789 INFO Attempting to optimize index logstash-2015.04.26.
2015-05-22 14:04:13,074 INFO Skipping index logstash-2015.04.26: Already optimized.
2015-05-22 14:04:13,075 INFO Attempting to optimize index logstash-2015.04.27.
2015-05-22 14:04:13,382 INFO Skipping index logstash-2015.04.27: Already optimized.
2015-05-22 14:04:13,382 INFO Attempting to optimize index logstash-2015.04.28.
2015-05-22 14:04:13,684 INFO Skipping index logstash-2015.04.28: Already optimized.
2015-05-22 14:04:13,685 INFO Attempting to optimize index logstash-2015.04.29.
2015-05-22 14:04:14,017 INFO Skipping index logstash-2015.04.29: Already optimized.
2015-05-22 14:04:14,018 INFO Attempting to optimize index logstash-2015.04.30.
2015-05-22 14:04:14,268 INFO Skipping index logstash-2015.04.30: Already optimized.
2015-05-22 14:04:14,268 INFO Attempting to optimize index logstash-2015.05.01.
2015-05-22 14:04:14,540 INFO Skipping index logstash-2015.05.01: Already optimized.
2015-05-22 14:04:14,540 INFO Attempting to optimize index logstash-2015.05.02.
2015-05-22 14:04:14,831 INFO Skipping index logstash-2015.05.02: Already optimized.
2015-05-22 14:04:14,831 INFO Attempting to optimize index logstash-2015.05.03.
2015-05-22 14:04:15,112 INFO Skipping index logstash-2015.05.03: Already optimized.
2015-05-22 14:04:15,113 INFO Attempting to optimize index logstash-2015.05.04.
2015-05-22 14:04:15,118 INFO Attempting to delete snapshot for index logstash-2015.04.20.
2015-05-22 14:04:15,398 INFO Skipping index logstash-2015.05.04: Already optimized.
2015-05-22 14:04:15,399 INFO Attempting to optimize index logstash-2015.05.05.
2015-05-22 14:04:15,698 INFO Skipping index logstash-2015.05.05: Already optimized.
2015-05-22 14:04:15,698 INFO Attempting to optimize index logstash-2015.05.06.
2015-05-22 14:04:15,964 INFO Skipping: A snapshot with name 'logstash-2015.04.23' already exists.
2015-05-22 14:04:15,965 INFO Attempting to create snapshot for index logstash-2015.04.24.
2015-05-22 14:04:15,994 INFO Skipping index logstash-2015.05.06: Already optimized.
2015-05-22 14:04:15,995 INFO Attempting to optimize index logstash-2015.05.07.
2015-05-22 14:04:16,300 INFO Skipping index logstash-2015.05.07: Already optimized.
2015-05-22 14:04:16,300 INFO Attempting to optimize index logstash-2015.05.08.
2015-05-22 14:04:16,591 INFO Skipping index logstash-2015.05.08: Already optimized.
2015-05-22 14:04:16,591 INFO Attempting to optimize index logstash-2015.05.09.
2015-05-22 14:04:16,708 INFO Skipping: A snapshot with name 'logstash-2015.04.24' already exists.
2015-05-22 14:04:16,708 INFO Attempting to create snapshot for index logstash-2015.04.25.
2015-05-22 14:04:16,845 INFO Skipping index logstash-2015.05.09: Already optimized.
2015-05-22 14:04:16,846 INFO Attempting to optimize index logstash-2015.05.10.
2015-05-22 14:04:17,042 INFO Skipping: A snapshot with name 'logstash-2015.04.25' already exists.
2015-05-22 14:04:17,043 INFO Attempting to create snapshot for index logstash-2015.04.26.
2015-05-22 14:04:17,129 INFO Skipping index logstash-2015.05.10: Already optimized.
2015-05-22 14:04:17,129 INFO Attempting to optimize index logstash-2015.05.11.
2015-05-22 14:04:17,396 INFO Skipping: A snapshot with name 'logstash-2015.04.26' already exists.
2015-05-22 14:04:17,397 INFO Attempting to create snapshot for index logstash-2015.04.27.
2015-05-22 14:04:17,433 INFO Skipping index logstash-2015.05.11: Already optimized.
2015-05-22 14:04:17,434 INFO Attempting to optimize index logstash-2015.05.12.
2015-05-22 14:04:17,751 INFO Skipping index logstash-2015.05.12: Already optimized.
2015-05-22 14:04:17,751 INFO Attempting to optimize index logstash-2015.05.13.
2015-05-22 14:04:17,868 INFO Skipping: A snapshot with name 'logstash-2015.04.27' already exists.
2015-05-22 14:04:17,869 INFO Attempting to create snapshot for index logstash-2015.04.28.
2015-05-22 14:04:18,113 INFO Skipping index logstash-2015.05.13: Already optimized.
2015-05-22 14:04:18,114 INFO Attempting to optimize index logstash-2015.05.20.
2015-05-22 14:04:18,241 INFO Skipping: A snapshot with name 'logstash-2015.04.28' already exists.
2015-05-22 14:04:18,242 INFO Attempting to create snapshot for index logstash-2015.04.29.
2015-05-22 14:04:18,267 INFO Optimizing index logstash-2015.05.20 to 2 segments per shard. Please wait...
2015-05-22 14:04:18,498 INFO logstash-2015.05.20: Successfully optimized.
2015-05-22 14:04:18,498 INFO logstash-2015.05.21 is within the threshold period (2 days).
2015-05-22 14:04:18,499 INFO logstash-2015.05.22 is within the threshold period (2 days).
2015-05-22 14:04:18,499 INFO logstash-2015.05.23 is within the threshold period (2 days).
2015-05-22 14:04:18,499 INFO OPTIMIZE index operations completed.
2015-05-22 14:04:18,500 INFO Done in 0:00:06.696626.
2015-05-22 14:04:18,610 INFO Skipping: A snapshot with name 'logstash-2015.04.29' already exists.
2015-05-22 14:04:18,611 INFO Attempting to create snapshot for index logstash-2015.04.30.
2015-05-22 14:04:18,965 INFO Skipping: A snapshot with name 'logstash-2015.04.30' already exists.
2015-05-22 14:04:18,965 INFO Attempting to create snapshot for index logstash-2015.05.01.
2015-05-22 14:04:19,299 INFO Skipping: A snapshot with name 'logstash-2015.05.01' already exists.
2015-05-22 14:04:19,300 INFO Attempting to create snapshot for index logstash-2015.05.02.
2015-05-22 14:04:19,646 INFO Skipping: A snapshot with name 'logstash-2015.05.02' already exists.
2015-05-22 14:04:19,647 INFO Attempting to create snapshot for index logstash-2015.05.03.
2015-05-22 14:04:20,050 INFO Skipping: A snapshot with name 'logstash-2015.05.03' already exists.
2015-05-22 14:04:20,050 INFO Attempting to create snapshot for index logstash-2015.05.04.
2015-05-22 14:04:20,421 INFO Skipping: A snapshot with name 'logstash-2015.05.04' already exists.
2015-05-22 14:04:20,421 INFO Attempting to create snapshot for index logstash-2015.05.05.
2015-05-22 14:04:20,799 INFO Skipping: A snapshot with name 'logstash-2015.05.05' already exists.
2015-05-22 14:04:20,800 INFO Attempting to create snapshot for index logstash-2015.05.06.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 736, in <module>
main()
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 731, in main
arguments.func(client, **argdict)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 585, in command_loop
skipped = op(client, index_name, **kwargs)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 424, in _delete_snapshot
client.snapshot.delete(repository=kwargs['repository'], snapshot=snap_name)
File "/usr/lib/python2.6/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.6/site-packages/elasticsearch/client/snapshot.py", line 34, in delete
_make_path('_snapshot', repository, snapshot), params=params)
File "/usr/lib/python2.6/site-packages/elasticsearch/transport.py", line 284, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.6/site-packages/elasticsearch/connection/http_urllib3.py", line 55, in perform_request
self._raise_error(response.status, raw_data)
File "/usr/lib/python2.6/site-packages/elasticsearch/connection/base.py", line 97, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.TransportError: TransportError(503, u'RemoteTransportException[[617c6cbc-2cb5-4d2b-aae3-445ce04d3040][inet[/xx.xx.1.247:9300]][cluster/snapshot/delete]]; nested: ConcurrentSnapshotExecutionException[[NLS_BACKUP:logstash-2015.04.20] another snapshot is currently running cannot delete]; ')
Processed 0 node jobs.
Processed 1 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated

Then 10 minutes later, I see this in the log

Processed 0 node jobs.
Processed 1 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
2015-05-22 14:13:13,634 INFO logstash-2015.05.06: Successfully created snapshot.
2015-05-22 14:13:13,635 INFO Attempting to create snapshot for index logstash-2015.05.07.

Backups all the way to yesterday's index are completed. I set the job to run tomorrow. Will let you know how it goes. Inconsistency is definitely a concern.

Let us know - I've seen cases of inconsistency before, it would be good to know if your issue is resolved.

elasticsearch.exceptions.TransportError: TransportError(503, u'RemoteTransportException[[617c6cbc-2cb5-4d2b-aae3-445ce04d3040][inet[/xx.xx.1.247:9300]][cluster/snapshot/delete]]; nested: ConcurrentSnapshotExecutionException[[NLS_BACKUP:logstash-2015.04.20] another snapshot is currently running cannot delete]; ')

It looks like this backup failed because another was already in progress.