Nagios Support Forum

Our old snapshot repo ran out of space, so I removed all the snapshots from UI and recreated a bigger repo with the same name. But after the recreation of the snapshot repo, I could not back up the snapshot anymore. The snapshot job runs successfully with no error message in jobs.log, but no snapshot is created. Does anyone have any ideas?

An incomplete snapshot in the repo can make it look like snapshots are not working. Please provide a directory listing of the repo. On the NLS command line run:

Here is the result. It seems no files are present in the directory

[root@nagiosls ~]# ls -alh /mnt/snapshotrepo/
total 0
drwxrwxr-x 2 nagios nagios 6 Nov 11 10:28 .
drwxrwxrwx. 3 root root 26 Sep 16 13:59 ..

cdienger wrote:An incomplete snapshot in the repo can make it look like snapshots are not working. Please provide a directory listing of the repo. On the NLS command line run:

Code: Select all

ls -alh /location/of/mounted/repo

Run the following on the command line and provide me the output:
Be sure to use the actual repository name instead of REPOSITORYNAME when running the command.

2020-11-13 10:25:04,213 INFO Job starting: snapshot indices
2020-11-13 10:25:04,213 WARNING Overriding default connection timeout. New timeout: 21600
2020-11-13 10:25:04,226 INFO Action snapshot will be performed on the following indices: ['logstash-2020.0 9.10', 'logstash-2020.09.11', 'logstash-2020.09.12', 'logstash-2020.09.13', 'logstash-2020.09.14', 'logstash-20 20.09.15', 'logstash-2020.09.16', 'logstash-2020.09.17', 'logstash-2020.09.18', 'logstash-2020.09.19', 'logstas h-2020.09.20', 'logstash-2020.09.21', 'logstash-2020.09.22', 'logstash-2020.09.23', 'logstash-2020.09.24', 'log stash-2020.09.25', 'logstash-2020.09.26', 'logstash-2020.09.27', 'logstash-2020.09.28', 'logstash-2020.09.29', 'logstash-2020.09.30', 'logstash-2020.10.01', 'logstash-2020.10.02', 'logstash-2020.10.03', 'logstash-2020.10.0 4', 'logstash-2020.10.05', 'logstash-2020.10.06', 'logstash-2020.10.07', 'logstash-2020.10.08', 'logstash-2020. 10.09', 'logstash-2020.10.10', 'logstash-2020.10.11', 'logstash-2020.10.12', 'logstash-2020.10.13', 'logstash-2 020.10.14', 'logstash-2020.10.15', 'logstash-2020.10.16', 'logstash-2020.10.17', 'logstash-2020.10.18', 'logsta sh-2020.10.19', 'logstash-2020.10.20', 'logstash-2020.10.21', 'logstash-2020.10.22', 'logstash-2020.10.23', 'lo gstash-2020.10.24', 'logstash-2020.10.25', 'logstash-2020.10.26', 'logstash-2020.10.27', 'logstash-2020.10.28', 'logstash-2020.10.29', 'logstash-2020.10.30', 'logstash-2020.10.31', 'logstash-2020.11.01', 'logstash-2020.11. 02', 'logstash-2020.11.03', 'logstash-2020.11.04', 'logstash-2020.11.05', 'logstash-2020.11.06', 'logstash-2020 .11.07', 'logstash-2020.11.08', 'logstash-2020.11.09', 'logstash-2020.11.10', 'logstash-2020.11.11', 'logstash- 2020.11.12']
2020-11-13 10:25:04,237 ERROR Snapshot already in progress: [{'snapshot': 'curator-20201005133221', 'reposi tory': 'Snapshot Repository', 'state': 'ABORTED', 'shards_stats': {'initializing': 0, 'started': 0, 'finalizing ': 0, 'done': 89, 'failed': 36, 'total': 125}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_siz e_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'indices': {'log stash-2020.09.10': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in _bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'numbe r_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_mi llis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, ' total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': { 'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_siz e_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_f iles': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_ size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash- 2020.09.11': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total ': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes ': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_f iles': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_ size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage ': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_b ytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 't ime_in_millis': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_i n_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.0 9.30': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'0': {'stage': 'FAILURE', 'stats': {'number_of_file s': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_ size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage ': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_i n_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'FAILURE', 'stats': {'number_of_f iles': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'tot al_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logsta sh-2020.09.14': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'to tal': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_by tes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_o f_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_milli s': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'tot al_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'st age': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_i n_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_file s': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_siz e_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-202 0.10.03': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'1': {'stage': 'DONE', 'stats': {'number_of_file s': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_siz e_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_byte s': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time _in_millis': 0}}, '4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_b ytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.10.0 4': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 's tats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'st art_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'2': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time _in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_b ytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '4': {'stage': 'DONE' , 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'proc essed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_mi llis': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.15': {' shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_ti me_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'p rocessed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in _millis': 0}}, '3': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_b ytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'FAILU RE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time _in_millis': 0}}, '0': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_i n_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.0 9.12': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 't ime_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_i n_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DO NE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'p rocessed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in _millis': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.13': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stat s': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start _time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'p rocessed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in _millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', ' stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 's tart_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'process ed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_milli s': 0}}, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.18': {'sha rds_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {' number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_ in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'process ed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_milli s': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats' : {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_t ime_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_fil es': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0} }, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'proc essed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.19': {'shards_st ats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number _of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_mil lis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_fil es': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0} }, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'proc essed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'nu mber_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in _millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0 , 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0' : {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_ size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.16': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'number_of_fi les': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files' : 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'proc essed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'FAILURE', 'stats': { 'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time _in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_fil es': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0} }, '0': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'p rocessed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.10.01': {'shards _stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'num ber_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_ millis': 0, 'time_in_millis': 0}, 'shards': {'1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'process ed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_milli s': 0}}, '0': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'FAILURE', ' stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 's tart_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'proc essed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_mi llis': 0}}, '4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.10.02': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stat s': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start _time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'p rocessed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in _millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', ' stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 's tart_time_in_millis': 0, 'time_in_millis': 0}}, '4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'process ed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_milli s': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.17': {'sha rds_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {' number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_ in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'process ed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_milli s': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats' : {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_t ime_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_fil es': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0} }, '0': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'proc essed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.21': {'shards_st ats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number _of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_mil lis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_fil es': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0} }, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'proc essed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'nu mber_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in _millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0 , 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0' : {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_ size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.22': {'shards_stats': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_fi les': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0 , 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3' : {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_ size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_o f_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_milli s': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'tot al_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'st age': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_i n_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.20': {'shards_stats': {'init ializing': 0, 'started': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'ti me_in_millis': 0}, 'shards': {'4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, ' total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': { 'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_ size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'FAILURE', 'stats': {'numbe r_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_mi llis': 0, 'time_in_millis': 0}}, '1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0 , 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0' : {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'process ed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.25': {'shards_stats ': {'initializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of _files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis ': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files' : 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'process ed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'numbe r_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_mi llis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, ' total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': { 'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_siz e_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.26': {'shards_stats': {'i nitializing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_files ': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, ' total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': { 'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_siz e_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_of_f iles': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_ size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage ': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_b ytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.23': {'shards_stats': {'initial izing': 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_ in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_ size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage ': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_b ytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 't ime_in_millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_i n_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'DO NE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.24': {'shards_stats': {'initializing' : 0, 'started': 0, 'finalizing': 0, 'done': 5, 'failed': 0, 'total': 5}, 'stats': {'number_of_files': 0, 'proce ssed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_mil lis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_i n_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DO NE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'p rocessed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in _millis': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'DONE', ' stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 's tart_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.29': {'shards_stats': {'initializing': 0, ' started': 0, 'finalizing': 0, 'done': 4, 'failed': 1, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_f iles': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'DONE', ' stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 's tart_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'process ed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_milli s': 0}}, '1': {'stage': 'DONE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'FAILURE', 'sta ts': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'star t_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.27': {'shards_stats': {'initializing': 0, 'sta rted': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'number_of_files': 0, 'processed_file s': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}, 'shards': {'4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_byte s': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'FAILURE' , 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'p rocessed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in _millis': 0}}, '1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_b ytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage': 'FAILU RE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}, 'logstash-2020.09.28': {'shards_stats': {'initializing' : 0, 'started': 0, 'finalizing': 0, 'done': 0, 'failed': 5, 'total': 5}, 'stats': {'number_of_files': 0, 'proce ssed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_mil lis': 0}, 'shards': {'4': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_siz e_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '3': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_b ytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '2': {'stage': 'FAILURE', 'stats': {'number_of_file s': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '1': {'stage': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_ size_in_bytes': 0, 'processed_size_in_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}, '0': {'stage ': 'FAILURE', 'stats': {'number_of_files': 0, 'processed_files': 0, 'total_size_in_bytes': 0, 'processed_size_i n_bytes': 0, 'start_time_in_millis': 0, 'time_in_millis': 0}}}}}}]
2020-11-13 10:25:04,298 WARNING Job did not complete successfully.

Check for curator jobs that may be running already on the NLS machine:
and use the kill command to stop any that you may find:
and then try running the command again so that it tries to backup the indexes in smaller chunks:

I ran the 3 commands, but all show "Job did not complete successfully." We have 2 nodes, does this need to be executed on both nodes? or any single node works?

The last output that was provided showed the reason why it failed:

...
ERROR Snapshot already in progress:
...

Do you see this same message?

You only need to run the command on one machine in the cluster but both machines must have the same repo mounted.

Yes，I saw the same message, but the "curator" process was not running on any of the nodes.

Run:
which should show you a snapshot in progress. Delete it. For example: