Snapshot Repo Reconfig

[duchenz]

Our old snapshot repo ran out of space, so I removed all the snapshots from UI and recreated a bigger repo with the same name. But after the recreation of the snapshot repo, I could not back up the snapshot anymore. The snapshot job runs successfully with no error message in jobs.log, but no snapshot is created. Does anyone have any ideas?

[cdienger]

An incomplete snapshot in the repo can make it look like snapshots are not working. Please provide a directory listing of the repo. On the NLS command line run:

Code: Select all

ls -alh /location/of/mounted/repo

[duchenz]

Here is the result. It seems no files are present in the directory

[root@nagiosls ~]# ls -alh /mnt/snapshotrepo/
total 0
drwxrwxr-x 2 nagios nagios 6 Nov 11 10:28 .
drwxrwxrwx. 3 root root 26 Sep 16 13:59 ..

An incomplete snapshot in the repo can make it look like snapshots are not working. Please provide a directory listing of the repo. On the NLS command line run:

Code: Select all

ls -alh /location/of/mounted/repo

[cdienger]

Run the following on the command line and provide me the output:

Code: Select all

/usr/local/nagioslogserver/scripts/curator.sh snapshot --repository 'REPOSITORYNAME' --ignore_unavailable indices --older-than 1 --time-unit days --timestring %Y.%m.%d

Be sure to use the actual repository name instead of REPOSITORYNAME when running the command.

[duchenz]

2020-11-13 10:25:04,213 INFO Job starting: snapshot indices
2020-11-13 10:25:04,213 WARNING Overriding default connection timeout. New timeout: 21600
2020-11-13 10:25:04,226 INFO Action snapshot will be performed on the following indices: ['logstash-2020.0 9.10', 'logstash-2020.09.11', 'logstash-2020.09.12', 'logstash-2020.09.13', 'logstash-2020.09.14', 'logstash-20 20.09.15', 'logstash-2020.09.16', 'logstash-2020.09.17', 'logstash-2020.09.18', 'logstash-2020.09.19', 'logstas h-2020.09.20', 'logstash-2020.09.21', 'logstash-2020.09.22', 'logstash-2020.09.23', 'logstash-2020.09.24', 'log stash-2020.09.25', 'logstash-2020.09.26', 'logstash-2020.09.27', 'logstash-2020.09.28', 'logstash-2020.09.29', 'logstash-2020.09.30', 'logstash-2020.10.01', 'logstash-2020.10.02', 'logstash-2020.10.03', 'logstash-2020.10.0 4', 'logstash-2020.10.05', 'logstash-2020.10.06', 'logstash-2020.10.07', 'logstash-2020.10.08', 'logstash-2020. 10.09', 'logstash-2020.10.10', 'logstash-2020.10.11', 'logstash-2020.10.12', 'logstash-2020.10.13', 'logstash-2 020.10.14', 'logstash-2020.10.15', 'logstash-2020.10.16', 'logstash-2020.10.17', 'logstash-2020.10.18', 'logsta sh-2020.10.19', 'logstash-2020.10.20', 'logstash-2020.10.21', 'logstash-2020.10.22', 'logstash-2020.10.23', 'lo gstash-2020.10.24', 'logstash-2020.10.25', 'logstash-2020.10.26', 'logstash-2020.10.27', 'logstash-2020.10.28', 'logstash-2020.10.29', 'logstash-2020.10.30', 'logstash-2020.10.31', 'logstash-2020.11.01', 'logstash-2020.11. 02', 'logstash-2020.11.03', 'logstash-2020.11.04', 'logstash-2020.11.05', 'logstash-2020.11.06', 'logstash-2020 .11.07', 'logstash-2020.11.08', 'logstash-2020.11.09', 'logstash-2020.11.10', 'logstash-2020.11.11', 'logstash- 2020.11.12']
2020-11-13 10:25:04,237 ERROR Snapshot already in progress: [{'snapshot': 'curator-20201005133221', 'reposi tory': 'Snapshot Repository', 'state': 'ABORTED', 'shards_stats': {'initializing': 0, 'started': 0, 'finalizing ': 0, 'done': 89, 'failed': 36, 'total': 125}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_siz e_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'indices': {'log stash-2020.09.10': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in _bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'numbe r_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_mi llis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, ' total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': { 'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_siz e_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_f iles': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_ size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash- 2020.09.11': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total ': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes ': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_f iles': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_ size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage ': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_b ytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 't ime_in_millis': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_i n_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.0 9.30': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'0': {'stage': 'FAILURE', 'stats': {'number_of_file s': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_ size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage ': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_i n_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'FAILURE', 'stats': {'number_of_f iles': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'tot al_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logsta sh-2020.09.14': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'to tal': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_by tes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_o f_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_milli s': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'tot al_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'st age': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_i n_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_file s': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_siz e_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-202 0.10.03': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'1': {'stage': 'DONE', 'stats': {'number_of_file s': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_siz e_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_byte s': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time _in_millis': 0}}, '4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_b ytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.10.0 4': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 's tats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'st art_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'2': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time _in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_b ytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '4': {'stage': 'DONE' , 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'proc essed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_mi llis': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.15': {' shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_ti me_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'p rocessed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in _millis': 0}}, '3': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_b ytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'FAILU RE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time _in_millis': 0}}, '0': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_i n_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.0 9.12': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 't ime_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_i n_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DO NE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'p rocessed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in _millis': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.13': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stat s': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start _time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'p rocessed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in _millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', ' stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 's tart_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'process ed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_milli s': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.18': {'sha rds_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {' number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_ in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'process ed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_milli s': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats' : {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_t ime_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_fil es': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0} }, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'proc essed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.19': {'shards_st ats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number _of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_mil lis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_fil es': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0} }, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'proc essed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'nu mber_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in _millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0 , 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0' : {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_ size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.16': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'number_of_fi les': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files' : 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'proc essed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'FAILURE', 'stats': { 'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time _in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_fil es': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0} }, '0': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'p rocessed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.10.01': {'shards _stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'num ber_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_ millis': 0, 'time_in_millis': 0}, 'shards': {'1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'process ed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_milli s': 0}}, '0': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'FAILURE', ' stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 's tart_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'proc essed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_mi llis': 0}}, '4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.10.02': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stat s': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start _time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'p rocessed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in _millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', ' stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 's tart_time_in_millis': 0, 'time_in_millis': 0}}, '4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'process ed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_milli s': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.17': {'sha rds_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {' number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_ in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'process ed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_milli s': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats' : {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_t ime_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_fil es': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0} }, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'proc essed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.21': {'shards_st ats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number _of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_mil lis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_fil es': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0} }, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'proc essed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'nu mber_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in _millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0 , 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0' : {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_ size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.22': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_fi les': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0 , 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3' : {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_ size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_o f_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_milli s': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'tot al_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'st age': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_i n_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.20': {'shards_stats': {'init ializing': 0, 'started': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'ti me_in_millis': 0}, 'shards': {'4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, ' total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': { 'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_ size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'FAILURE', 'stats': {'numbe r_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_mi llis': 0, 'time_in_millis': 0}}, '1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0 , 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0' : {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'process ed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.25': {'shards_stats ': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of _files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis ': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files' : 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'process ed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'numbe r_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_mi llis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, ' total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': { 'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_siz e_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.26': {'shards_stats': {'i nitializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_files ': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, ' total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': { 'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_siz e_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_of_f iles': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_ size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage ': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_b ytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.23': {'shards_stats': {'initial izing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_ in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_ size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage ': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_b ytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 't ime_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_i n_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'DO NE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.24': {'shards_stats': {'initializing' : 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_files': 0, 'proce ssed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_mil lis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_i n_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DO NE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'p rocessed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in _millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'DONE', ' stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 's tart_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.29': {'shards_stats': {'initializing': 0, ' started': 0, 'finalizing': 0, 'done': 4, 'failed': 1, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_f iles': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', ' stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 's tart_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'process ed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_milli s': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'FAILURE', 'sta ts': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'star t_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.27': {'shards_stats': {'initializing': 0, 'sta rted': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_file s': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'FAILURE' , 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'p rocessed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in _millis': 0}}, '1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_b ytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'FAILU RE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.28': {'shards_stats': {'initializing' : 0, 'started': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'number_of_files': 0, 'proce ssed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_mil lis': 0}, 'shards': {'4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_siz e_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_b ytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'FAILURE', 'stats': {'number_of_file s': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_ size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage ': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_i n_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}}}]
2020-11-13 10:25:04,298 WARNING Job did not complete successfully.

[cdienger]

Check for curator jobs that may be running already on the NLS machine:

Code: Select all

ps aux | grep curator

and use the kill command to stop any that you may find:

Code: Select all

kill <PID>

and then try running the command again so that it tries to backup the indexes in smaller chunks:

Code: Select all

/usr/local/nagioslogserver/scripts/curator.sh snapshot --repository 'Snapshot Repository' --ignore_unavailable indices --older-than 60 --time-unit days --timestring %Y.%m.%d
/usr/local/nagioslogserver/scripts/curator.sh snapshot --repository 'Snapshot Repository' --ignore_unavailable indices --older-than 30 --time-unit days --timestring %Y.%m.%d
/usr/local/nagioslogserver/scripts/curator.sh snapshot --repository 'Snapshot Repository' --ignore_unavailable indices --older-than 1 --time-unit days --timestring %Y.%m.%d

[duchenz]

I ran the 3 commands, but all show "Job did not complete successfully." We have 2 nodes, does this need to be executed on both nodes? or any single node works?

[cdienger]

The last output that was provided showed the reason why it failed:

...
ERROR Snapshot already in progress:
...

Do you see this same message?

You only need to run the command on one machine in the cluster but both machines must have the same repo mounted.

[duchenz]

Yes，I saw the same message, but the "curator" process was not running on any of the nodes.

[cdienger]

Run:

Code: Select all

curl -XGET 'http://localhost:9200/_snapshot/Snapshot%20Repository/_all?pretty'

which should show you a snapshot in progress. Delete it. For example:

Code: Select all

curl -XDELETE 'http://localhost:9200/_snapshot/test/curator-20201217094956'