How to stop a currently running snapshot?

[cdienger]

You can remove the stuck snapshots(data will get backed up in a new snapshot)

Get list of snapshots:

Code: Select all

curl -XGET 'http://localhost:9200/_snapshot/NLSREPCC/_all?pretty'

Delete the "in progress" snapshot:

Code: Select all

curl -XDELETE 'http://localhost:9200/_snapshot/NLSREPCC/curator-20191021...

Replace "curator-20191021..." with the actual snapshot name.

[rferebee]

So, should this command take some time to complete? When I run it, all I get returned is a blank next line in the CLI.

[rferebee]

For some reason it won't remove these two snapshots.

Is there any reason why I couldn't just delete the snapshot and metadata files for each snapshot directly from my repository? Would that break anything?

[rferebee]

Hold on, I've got a different problem to include... I don't have any new indexes after 10/21. I just noticed this. I understand why I don't have them. It's because the snapshots aren't running to create them.

I didn't even think to look at that because Log Server looks like it's collecting data just fine. I don't know what the heck is going on!

I need this resolved ASAP please. We have a State holiday tomorrow, so I won't be in the office for 3 days.

[cdienger]

PM me a profile as well as the output of:

Code: Select all

curl -XGET 'http://localhost:9200/_snapshot/NLSREPCC/_all?pretty'

[rferebee]

PM sent

PM me a profile as well as the output of:

Code: Select all

curl -XGET 'http://localhost:9200/_snapshot/NLSREPCC/_all?pretty'

[cdienger]

It's odd that "curl -XGET 'http://localhost:9200/_snapshot/NLSREPCC/_all?pretty'" didn't return anything.

Tail the jobs.log and force a maintenance job to see if it complains about anything:

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log

[rferebee]

The output of that command is in the PM I sent you.

[rferebee]

Here the output of the jobs log. Nothing, it went straight to SUCCESS in the subsystem, but didn't actually do anything.

Code: Select all

root@nagioslscc2:/root> tail -f /usr/local/nagioslogserver/var/jobs.log
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated

If you look at the screen shot on page 2. It's odd the end dates on the two snapshots that are running at 12/31/1969 @ 16:00:00.

[rferebee]

Here we go, I got something the second time I tried:

Code: Select all

root@nagioslscc2:/root> tail -f /usr/local/nagioslogserver/var/jobs.log
Running command do_maintenance with args ' ' for job id: snapshots_maintenance
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh close indices --older-than 22 --time-unit days --timestring %Y.%m.%d
Return: 0
2019-10-24 15:08:52,068 INFO      Job starting: close indices
2019-10-24 15:08:52,089 WARNING   No indices matched provided args: {'regex': None, 'index': (), 'suffix': None, 'newer_than': None, 'closed_only': False, 'prefix': None, 'time_unit': 'days', 'timestring': u'%Y.%m.%d', 'exclude': (), 'older_than': 22, 'all_indices': False}
No indices matched provided args: {'regex': None, 'index': (), 'suffix': None, 'newer_than': None, 'closed_only': False, 'prefix': None, 'time_unit': 'days', 'timestring': u'%Y.%m.%d', 'exclude': (), 'older_than': 22, 'all_indices': False}
-----
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete indices --older-than 22 --time-unit days --timestring %Y.%m.%d
Return: 0
2019-10-24 15:08:52,273 INFO      Job starting: delete indices
2019-10-24 15:08:52,286 INFO      Pruning Kibana-related indices to prevent accidental deletion.
2019-10-24 15:08:52,286 WARNING   No indices matched provided args: {'regex': None, 'index': (), 'suffix': None, 'newer_than': None, 'closed_only': False, 'prefix': None, 'time_unit': 'days', 'timestring': u'%Y.%m.%d', 'exclude': (), 'older_than': 22, 'all_indices': False}
No indices matched provided args: {'regex': None, 'index': (), 'suffix': None, 'newer_than': None, 'closed_only': False, 'prefix': None, 'time_unit': 'days', 'timestring': u'%Y.%m.%d', 'exclude': (), 'older_than': 22, 'all_indices': False}
-----
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh snapshot --repository "NLSREPCC" --ignore_unavailable indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Return: 1
2019-10-24 15:08:53,452 INFO      Job starting: snapshot indices
2019-10-24 15:08:53,452 WARNING   Overriding default connection timeout.  New timeout: 21600
2019-10-24 15:08:53,466 INFO      Action snapshot will be performed on the following indices: [u'logstash-2019.10.03', u'logstash-2019.10.04', u'logstash-2019.10.05', u'logstash-2019.10.06', u'logstash-2019.10.07', u'logstash-2019.10.08', u'logstash-2019.10.09', u'logstash-2019.10.10', u'logstash-2019.10.11', u'logstash-2019.10.12', u'logstash-2019.10.13', u'logstash-2019.10.14', u'logstash-2019.10.15', u'logstash-2019.10.16', u'logstash-2019.10.17', u'logstash-2019.10.18', u'logstash-2019.10.19', u'logstash-2019.10.20', u'logstash-2019.10.21', u'logstash-2019.10.22', u'logstash-2019.10.23']
2019-10-24 15:08:53,488 ERROR     Snapshot already in progress: [{u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'repository': u'NLSREPCC', u'shards_stats': {u'started': 0, u'failed': 21, u'done': 84, u'finalizing': 0, u'initializing': 0, u'total': 105}, u'state': u'ABORTED', u'snapshot': u'curator-20191022053043', u'indices': {u'logstash-2019.10.08': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.09': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}}, u'shards_stats': {u'started': 0, u'failed': 3, u'done': 2, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.19': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.10': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}}, u'shards_stats': {u'started': 0, u'failed': 1, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.05': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 2, u'done': 3, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.13': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 1, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.01': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.11': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.03': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.04': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.16': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}}, u'shards_stats': {u'started': 0, u'failed': 1, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.15': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}}, u'shards_stats': {u'started': 0, u'failed': 2, u'done': 3, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.14': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.20': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}}, u'shards_stats': {u'started': 0, u'failed': 3, u'done': 2, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.21': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 2, u'done': 3, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.06': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.18': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 1, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.12': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 1, u'done': 4, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.07': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.17': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'FAILURE'}}, u'shards_stats': {u'started': 0, u'failed': 4, u'done': 1, u'finalizing': 0, u'initializing': 0, u'total': 5}}, u'logstash-2019.10.02': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'shards': {u'1': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'0': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'3': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'2': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}, u'4': {u'stats': {u'number_of_files': 0, u'total_size_in_bytes': 0, u'time_in_millis': 0, u'processed_files': 0, u'processed_size_in_bytes': 0, u'start_time_in_millis': 0}, u'stage': u'DONE'}}, u'shards_stats': {u'started': 0, u'failed': 0, u'done': 5, u'finalizing': 0, u'initializing': 0, u'total': 5}}}}]
2019-10-24 15:08:53,488 WARNING   Job did not complete successfully.
-----
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh delete snapshots --older-than 731 --time-unit days --timestring %Y%m%d --repository "NLSREPCC"
Return: 0
2019-10-24 15:08:53,638 INFO      Job starting: delete snapshots
2019-10-24 15:08:53,703 WARNING   No snapshots matched provided args.
No snapshots matched provided args.
-----
SUCCESS
Processed 0 node jobs.
Processed 1 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated