dwindling disk space and backups failing

[jpconsilio]

Hi,
Something happened and my backups have been failing I'm now at 91% usage on my cluster nodes.
/dev/mapper/centos-root 1005G 869G 95G 91% /

Failure looks like this:
[root@lntnls1 snmp]# curator --loglevel debug snapshot --repository "common" indices --all-indices
2021-04-29 05:51:24,523 INFO Job starting: snapshot indices
2021-04-29 05:51:24,523 DEBUG Params: {'url_prefix': u'', 'http_auth': None, 'dry_run': False, 'certificate': None, 'loglevel': u'debug', 'logformat': u'default', 'quiet': False, 'host': u'localhost', 'timeout': 30, 'debug': False, 'use_ssl': False, 'logfile': None, 'master_only': False, 'port': 9200, 'ssl_no_validate': False}
2021-04-29 05:51:24,523 WARNING Overriding default connection timeout. New timeout: 21600
2021-04-29 05:51:24,523 DEBUG kwargs = {'url_prefix': u'', 'http_auth': None, 'dry_run': False, 'certificate': None, 'loglevel': u'debug', 'quiet': False, 'debug': False, 'logformat': u'default', 'timeout': 21600, 'host': u'localhost', 'use_ssl': False, 'logfile': None, 'master_only': False, 'port': 9200, 'ssl_no_validate': False}
2021-04-29 05:51:24,530 DEBUG Detected Elasticsearch version 1.7.6
2021-04-29 05:51:24,540 DEBUG All indices: [u'logstash-2021.04.17', u'logstash-2021.04.16', u'logstash-2021.04.15', u'logstash-2021.04.14', u'logstash-2021.04.13', u'logstash-2021.04.12', u'logstash-2021.04.11', u'logstash-2021.04.10', u'logstash-2021.04.19', u'logstash-2021.04.18', u'logstash-2021.10.27', u'logstash-2021.10.26', u'logstash-2021.10.25', u'logstash-2021.10.24', u'logstash-2021.10.23', u'logstash-2021.10.29', u'logstash-2021.10.28', u'logstash-2021.04.04', u'logstash-2021.04.05', u'logstash-2021.04.06', u'logstash-2021.04.07', u'logstash-2021.04.01', u'logstash-2021.04.02', u'logstash-2021.04.03', u'logstash-2021.04.08', u'logstash-2021.04.09', u'logstash-2021.12.04', u'logstash-2021.12.03', u'logstash-2021.12.02', u'logstash-2021.12.01', u'logstash-2021.10.30', u'logstash-2021.10.31', u'logstash-2021.11.02', u'logstash-2021.03.08', u'logstash-2021.11.30', u'logstash-2021.10.09', u'logstash-2021.10.08', u'logstash-2021.10.05', u'logstash-2021.10.04', u'logstash-2021.10.07', u'logstash-2021.04.28', u'logstash-2021.04.29', u'logstash-2021.04.26', u'logstash-2021.04.27', u'logstash-2021.04.24', u'logstash-2021.04.25', u'logstash-2021.04.22', u'logstash-2021.04.23', u'logstash-2021.04.20', u'logstash-2021.04.21', u'logstash-2021.11.08', u'logstash-2021.11.28', u'logstash-2021.11.29', u'logstash-2021.11.09', u'logstash-2021.11.20', u'logstash-2021.11.21', u'logstash-2021.11.22', u'logstash-2021.11.23', u'logstash-2021.11.24', u'logstash-2021.11.25', u'logstash-2021.11.26', u'logstash-2021.11.27', u'logstash-2021.03.27', u'logstash-2021.03.26', u'logstash-2021.03.25', u'logstash-2021.03.24', u'logstash-2021.03.23', u'logstash-2021.03.22', u'logstash-2021.03.21', u'logstash-2021.10.19', u'logstash-2021.10.16', u'logstash-2021.10.17', u'logstash-2021.10.14', u'logstash-2021.10.15', u'logstash-2021.10.13', u'logstash-2021.10.10', u'logstash-2021.03.28', u'nagioslogserver_log', u'logstash-2021.11.12', u'logstash-2021.11.19', u'logstash-2021.11.18', u'nagioslogserver', u'logstash-2021.11.15', u'logstash-2021.11.14', u'logstash-2021.11.17', u'logstash-2021.11.16', u'logstash-2021.11.11', u'logstash-2021.11.10', u'logstash-2021.11.13', u'logstash-2021.12.31', u'logstash-2021.03.30', u'logstash-2021.03.31', u'logstash-2021.03.09', u'logstash-2021.11.03', u'logstash-2021.11.01', u'logstash-2021.11.06', u'logstash-2021.11.07', u'logstash-2021.11.04', u'logstash-2021.11.05', u'logstash-2021.03.01', u'logstash-2021.03.03', u'logstash-2021.03.02', u'logstash-2021.03.05', u'logstash-2021.03.04', u'logstash-2021.03.07', u'logstash-2021.03.06', u'kibana-int', u'logstash-2021.03.18', u'logstash-2021.03.19', u'logstash-2021.03.16', u'logstash-2021.03.17', u'logstash-2021.03.14', u'logstash-2021.03.15', u'logstash-2021.03.12', u'logstash-2021.03.13', u'logstash-2021.03.10', u'logstash-2021.03.11', u'logstash-2021.10.18', u'logstash-2021.03.20', u'logstash-2021.03.29', u'logstash-2021.10.11']
2021-04-29 05:51:24,540 DEBUG Full list of indices: [u'logstash-2021.04.17', u'logstash-2021.04.16', u'logstash-2021.04.15', u'logstash-2021.04.14', u'logstash-2021.04.13', u'logstash-2021.04.12', u'logstash-2021.04.11', u'logstash-2021.04.10', u'logstash-2021.04.19', u'logstash-2021.04.18', u'logstash-2021.10.27', u'logstash-2021.10.26', u'logstash-2021.10.25', u'logstash-2021.10.24', u'logstash-2021.10.23', u'logstash-2021.10.29', u'logstash-2021.10.28', u'logstash-2021.04.04', u'logstash-2021.04.05', u'logstash-2021.04.06', u'logstash-2021.04.07', u'logstash-2021.04.01', u'logstash-2021.04.02', u'logstash-2021.04.03', u'logstash-2021.04.08', u'logstash-2021.04.09', u'logstash-2021.12.04', u'logstash-2021.12.03', u'logstash-2021.12.02', u'logstash-2021.12.01', u'logstash-2021.10.30', u'logstash-2021.10.31', u'logstash-2021.11.02', u'logstash-2021.03.08', u'logstash-2021.11.30', u'logstash-2021.10.09', u'logstash-2021.10.08', u'logstash-2021.10.05', u'logstash-2021.10.04', u'logstash-2021.10.07', u'logstash-2021.04.28', u'logstash-2021.04.29', u'logstash-2021.04.26', u'logstash-2021.04.27', u'logstash-2021.04.24', u'logstash-2021.04.25', u'logstash-2021.04.22', u'logstash-2021.04.23', u'logstash-2021.04.20', u'logstash-2021.04.21', u'logstash-2021.11.08', u'logstash-2021.11.28', u'logstash-2021.11.29', u'logstash-2021.11.09', u'logstash-2021.11.20', u'logstash-2021.11.21', u'logstash-2021.11.22', u'logstash-2021.11.23', u'logstash-2021.11.24', u'logstash-2021.11.25', u'logstash-2021.11.26', u'logstash-2021.11.27', u'logstash-2021.03.27', u'logstash-2021.03.26', u'logstash-2021.03.25', u'logstash-2021.03.24', u'logstash-2021.03.23', u'logstash-2021.03.22', u'logstash-2021.03.21', u'logstash-2021.10.19', u'logstash-2021.10.16', u'logstash-2021.10.17', u'logstash-2021.10.14', u'logstash-2021.10.15', u'logstash-2021.10.13', u'logstash-2021.10.10', u'logstash-2021.03.28', u'nagioslogserver_log', u'logstash-2021.11.12', u'logstash-2021.11.19', u'logstash-2021.11.18', u'nagioslogserver', u'logstash-2021.11.15', u'logstash-2021.11.14', u'logstash-2021.11.17', u'logstash-2021.11.16', u'logstash-2021.11.11', u'logstash-2021.11.10', u'logstash-2021.11.13', u'logstash-2021.12.31', u'logstash-2021.03.30', u'logstash-2021.03.31', u'logstash-2021.03.09', u'logstash-2021.11.03', u'logstash-2021.11.01', u'logstash-2021.11.06', u'logstash-2021.11.07', u'logstash-2021.11.04', u'logstash-2021.11.05', u'logstash-2021.03.01', u'logstash-2021.03.03', u'logstash-2021.03.02', u'logstash-2021.03.05', u'logstash-2021.03.04', u'logstash-2021.03.07', u'logstash-2021.03.06', u'kibana-int', u'logstash-2021.03.18', u'logstash-2021.03.19', u'logstash-2021.03.16', u'logstash-2021.03.17', u'logstash-2021.03.14', u'logstash-2021.03.15', u'logstash-2021.03.12', u'logstash-2021.03.13', u'logstash-2021.03.10', u'logstash-2021.03.11', u'logstash-2021.10.18', u'logstash-2021.03.20', u'logstash-2021.03.29', u'logstash-2021.10.11']
2021-04-29 05:51:24,540 INFO Matching all indices. Ignoring flags other than --exclude.
2021-04-29 05:51:24,540 DEBUG All filters: []
2021-04-29 05:51:24,540 INFO Action snapshot will be performed on the following indices: [u'kibana-int', u'logstash-2021.03.01', u'logstash-2021.03.02', u'logstash-2021.03.03', u'logstash-2021.03.04', u'logstash-2021.03.05', u'logstash-2021.03.06', u'logstash-2021.03.07', u'logstash-2021.03.08', u'logstash-2021.03.09', u'logstash-2021.03.10', u'logstash-2021.03.11', u'logstash-2021.03.12', u'logstash-2021.03.13', u'logstash-2021.03.14', u'logstash-2021.03.15', u'logstash-2021.03.16', u'logstash-2021.03.17', u'logstash-2021.03.18', u'logstash-2021.03.19', u'logstash-2021.03.20', u'logstash-2021.03.21', u'logstash-2021.03.22', u'logstash-2021.03.23', u'logstash-2021.03.24', u'logstash-2021.03.25', u'logstash-2021.03.26', u'logstash-2021.03.27', u'logstash-2021.03.28', u'logstash-2021.03.29', u'logstash-2021.03.30', u'logstash-2021.03.31', u'logstash-2021.04.01', u'logstash-2021.04.02', u'logstash-2021.04.03', u'logstash-2021.04.04', u'logstash-2021.04.05', u'logstash-2021.04.06', u'logstash-2021.04.07', u'logstash-2021.04.08', u'logstash-2021.04.09', u'logstash-2021.04.10', u'logstash-2021.04.11', u'logstash-2021.04.12', u'logstash-2021.04.13', u'logstash-2021.04.14', u'logstash-2021.04.15', u'logstash-2021.04.16', u'logstash-2021.04.17', u'logstash-2021.04.18', u'logstash-2021.04.19', u'logstash-2021.04.20', u'logstash-2021.04.21', u'logstash-2021.04.22', u'logstash-2021.04.23', u'logstash-2021.04.24', u'logstash-2021.04.25', u'logstash-2021.04.26', u'logstash-2021.04.27', u'logstash-2021.04.28', u'logstash-2021.04.29', u'logstash-2021.10.04', u'logstash-2021.10.05', u'logstash-2021.10.07', u'logstash-2021.10.08', u'logstash-2021.10.09', u'logstash-2021.10.10', u'logstash-2021.10.11', u'logstash-2021.10.13', u'logstash-2021.10.14', u'logstash-2021.10.15', u'logstash-2021.10.16', u'logstash-2021.10.17', u'logstash-2021.10.18', u'logstash-2021.10.19', u'logstash-2021.10.23', u'logstash-2021.10.24', u'logstash-2021.10.25', u'logstash-2021.10.26', u'logstash-2021.10.27', u'logstash-2021.10.28', u'logstash-2021.10.29', u'logstash-2021.10.30', u'logstash-2021.10.31', u'logstash-2021.11.01', u'logstash-2021.11.02', u'logstash-2021.11.03', u'logstash-2021.11.04', u'logstash-2021.11.05', u'logstash-2021.11.06', u'logstash-2021.11.07', u'logstash-2021.11.08', u'logstash-2021.11.09', u'logstash-2021.11.10', u'logstash-2021.11.11', u'logstash-2021.11.12', u'logstash-2021.11.13', u'logstash-2021.11.14', u'logstash-2021.11.15', u'logstash-2021.11.16', u'logstash-2021.11.17', u'logstash-2021.11.18', u'logstash-2021.11.19', u'logstash-2021.11.20', u'logstash-2021.11.21', u'logstash-2021.11.22', u'logstash-2021.11.23', u'logstash-2021.11.24', u'logstash-2021.11.25', u'logstash-2021.11.26', u'logstash-2021.11.27', u'logstash-2021.11.28', u'logstash-2021.11.29', u'logstash-2021.11.30', u'logstash-2021.12.01', u'logstash-2021.12.02', u'logstash-2021.12.03', u'logstash-2021.12.04', u'logstash-2021.12.31', u'nagioslogserver', u'nagioslogserver_log']
2021-04-29 05:51:24,582 DEBUG Including index kibana-int: Opened.
2021-04-29 05:51:24,601 DEBUG Skipping index logstash-2021.03.01: Opened.
2021-04-29 05:51:24,619 DEBUG Skipping index logstash-2021.03.02: Opened.
2021-04-29 05:51:24,647 DEBUG Skipping index logstash-2021.03.03: Opened.
2021-04-29 05:51:24,668 DEBUG Skipping index logstash-2021.03.04: Opened.
2021-04-29 05:51:24,691 DEBUG Skipping index logstash-2021.03.05: Opened.
2021-04-29 05:51:24,734 DEBUG Skipping index logstash-2021.03.06: Opened.
2021-04-29 05:51:24,757 DEBUG Skipping index logstash-2021.03.07: Opened.
2021-04-29 05:51:24,777 DEBUG Skipping index logstash-2021.03.08: Opened.
2021-04-29 05:51:24,814 DEBUG Skipping index logstash-2021.03.09: Opened.
2021-04-29 05:51:24,836 DEBUG Skipping index logstash-2021.03.10: Opened.
2021-04-29 05:51:24,859 DEBUG Skipping index logstash-2021.03.11: Opened.
2021-04-29 05:51:24,879 DEBUG Skipping index logstash-2021.03.12: Opened.
2021-04-29 05:51:24,897 DEBUG Skipping index logstash-2021.03.13: Opened.
2021-04-29 05:51:24,919 DEBUG Skipping index logstash-2021.03.14: Opened.
2021-04-29 05:51:24,940 DEBUG Skipping index logstash-2021.03.15: Opened.
2021-04-29 05:51:24,976 DEBUG Including index logstash-2021.03.16: Opened.
2021-04-29 05:51:25,001 DEBUG Including index logstash-2021.03.17: Opened.
2021-04-29 05:51:25,026 DEBUG Including index logstash-2021.03.18: Opened.
2021-04-29 05:51:25,063 DEBUG Including index logstash-2021.03.19: Opened.
2021-04-29 05:51:25,089 DEBUG Including index logstash-2021.03.20: Opened.
2021-04-29 05:51:25,112 DEBUG Including index logstash-2021.03.21: Opened.
2021-04-29 05:51:25,131 DEBUG Including index logstash-2021.03.22: Opened.
2021-04-29 05:51:25,150 DEBUG Including index logstash-2021.03.23: Opened.
2021-04-29 05:51:25,193 DEBUG Including index logstash-2021.03.24: Opened.
2021-04-29 05:51:25,217 DEBUG Including index logstash-2021.03.25: Opened.
2021-04-29 05:51:25,240 DEBUG Including index logstash-2021.03.26: Opened.
2021-04-29 05:51:25,258 DEBUG Including index logstash-2021.03.27: Opened.
2021-04-29 05:51:25,290 DEBUG Including index logstash-2021.03.28: Opened.
2021-04-29 05:51:25,311 DEBUG Including index logstash-2021.03.29: Opened.
2021-04-29 05:51:25,328 DEBUG Including index logstash-2021.03.30: Opened.
2021-04-29 05:51:25,346 DEBUG Including index logstash-2021.03.31: Opened.
2021-04-29 05:51:25,367 DEBUG Including index logstash-2021.04.01: Opened.
2021-04-29 05:51:25,385 DEBUG Including index logstash-2021.04.02: Opened.
2021-04-29 05:51:25,420 DEBUG Including index logstash-2021.04.03: Opened.
2021-04-29 05:51:25,446 DEBUG Including index logstash-2021.04.04: Opened.
2021-04-29 05:51:25,467 DEBUG Including index logstash-2021.04.05: Opened.
2021-04-29 05:51:25,487 DEBUG Including index logstash-2021.04.06: Opened.
2021-04-29 05:51:25,520 DEBUG Including index logstash-2021.04.07: Opened.
2021-04-29 05:51:25,545 DEBUG Including index logstash-2021.04.08: Opened.
2021-04-29 05:51:25,567 DEBUG Including index logstash-2021.04.09: Opened.
2021-04-29 05:51:25,589 DEBUG Including index logstash-2021.04.10: Opened.
2021-04-29 05:51:25,634 DEBUG Including index logstash-2021.04.11: Opened.
2021-04-29 05:51:25,654 DEBUG Including index logstash-2021.04.12: Opened.
2021-04-29 05:51:25,676 DEBUG Including index logstash-2021.04.13: Opened.
2021-04-29 05:51:25,699 DEBUG Including index logstash-2021.04.14: Opened.
2021-04-29 05:51:25,718 DEBUG Including index logstash-2021.04.15: Opened.
2021-04-29 05:51:25,758 DEBUG Including index logstash-2021.04.16: Opened.
2021-04-29 05:51:25,778 DEBUG Including index logstash-2021.04.17: Opened.
2021-04-29 05:51:25,803 DEBUG Including index logstash-2021.04.18: Opened.
2021-04-29 05:51:25,843 DEBUG Including index logstash-2021.04.19: Opened.
2021-04-29 05:51:25,869 DEBUG Including index logstash-2021.04.20: Opened.
2021-04-29 05:51:25,892 DEBUG Including index logstash-2021.04.21: Opened.
2021-04-29 05:51:25,915 DEBUG Including index logstash-2021.04.22: Opened.
2021-04-29 05:51:25,937 DEBUG Including index logstash-2021.04.23: Opened.
2021-04-29 05:51:25,957 DEBUG Including index logstash-2021.04.24: Opened.
2021-04-29 05:51:25,992 DEBUG Including index logstash-2021.04.25: Opened.
2021-04-29 05:51:26,010 DEBUG Including index logstash-2021.04.26: Opened.
2021-04-29 05:51:26,030 DEBUG Including index logstash-2021.04.27: Opened.
2021-04-29 05:51:26,048 DEBUG Including index logstash-2021.04.28: Opened.
2021-04-29 05:51:26,084 DEBUG Including index logstash-2021.04.29: Opened.
2021-04-29 05:51:26,105 DEBUG Including index logstash-2021.10.04: Opened.
2021-04-29 05:51:26,125 DEBUG Including index logstash-2021.10.05: Opened.
2021-04-29 05:51:26,147 DEBUG Including index logstash-2021.10.07: Opened.
2021-04-29 05:51:26,168 DEBUG Including index logstash-2021.10.08: Opened.
2021-04-29 05:51:26,190 DEBUG Including index logstash-2021.10.09: Opened.
2021-04-29 05:51:26,226 DEBUG Including index logstash-2021.10.10: Opened.
2021-04-29 05:51:26,252 DEBUG Including index logstash-2021.10.11: Opened.
2021-04-29 05:51:26,274 DEBUG Including index logstash-2021.10.13: Opened.
2021-04-29 05:51:26,310 DEBUG Including index logstash-2021.10.14: Opened.
2021-04-29 05:51:26,328 DEBUG Including index logstash-2021.10.15: Opened.
2021-04-29 05:51:26,349 DEBUG Including index logstash-2021.10.16: Opened.
2021-04-29 05:51:26,371 DEBUG Including index logstash-2021.10.17: Opened.
2021-04-29 05:51:26,390 DEBUG Including index logstash-2021.10.18: Opened.
2021-04-29 05:51:26,407 DEBUG Including index logstash-2021.10.19: Opened.
2021-04-29 05:51:26,448 DEBUG Including index logstash-2021.10.23: Opened.
2021-04-29 05:51:26,473 DEBUG Including index logstash-2021.10.24: Opened.
2021-04-29 05:51:26,500 DEBUG Including index logstash-2021.10.25: Opened.
2021-04-29 05:51:26,545 DEBUG Including index logstash-2021.10.26: Opened.
2021-04-29 05:51:26,569 DEBUG Including index logstash-2021.10.27: Opened.
2021-04-29 05:51:26,589 DEBUG Including index logstash-2021.10.28: Opened.
2021-04-29 05:51:26,611 DEBUG Including index logstash-2021.10.29: Opened.
2021-04-29 05:51:26,630 DEBUG Including index logstash-2021.10.30: Opened.
2021-04-29 05:51:26,652 DEBUG Including index logstash-2021.10.31: Opened.
2021-04-29 05:51:26,691 DEBUG Including index logstash-2021.11.01: Opened.
2021-04-29 05:51:26,716 DEBUG Including index logstash-2021.11.02: Opened.
2021-04-29 05:51:26,742 DEBUG Including index logstash-2021.11.03: Opened.
2021-04-29 05:51:26,764 DEBUG Including index logstash-2021.11.04: Opened.
2021-04-29 05:51:26,804 DEBUG Including index logstash-2021.11.05: Opened.
2021-04-29 05:51:26,827 DEBUG Including index logstash-2021.11.06: Opened.
2021-04-29 05:51:26,848 DEBUG Including index logstash-2021.11.07: Opened.
2021-04-29 05:51:26,871 DEBUG Including index logstash-2021.11.08: Opened.
2021-04-29 05:51:26,896 DEBUG Including index logstash-2021.11.09: Opened.
2021-04-29 05:51:26,921 DEBUG Including index logstash-2021.11.10: Opened.
2021-04-29 05:51:26,961 DEBUG Including index logstash-2021.11.11: Opened.
2021-04-29 05:51:26,989 DEBUG Including index logstash-2021.11.12: Opened.
2021-04-29 05:51:27,013 DEBUG Including index logstash-2021.11.13: Opened.
2021-04-29 05:51:27,039 DEBUG Including index logstash-2021.11.14: Opened.
2021-04-29 05:51:27,072 DEBUG Including index logstash-2021.11.15: Opened.
2021-04-29 05:51:27,096 DEBUG Including index logstash-2021.11.16: Opened.
2021-04-29 05:51:27,123 DEBUG Including index logstash-2021.11.17: Opened.
2021-04-29 05:51:27,148 DEBUG Including index logstash-2021.11.18: Opened.
2021-04-29 05:51:27,171 DEBUG Including index logstash-2021.11.19: Opened.
2021-04-29 05:51:27,213 DEBUG Including index logstash-2021.11.20: Opened.
2021-04-29 05:51:27,235 DEBUG Including index logstash-2021.11.21: Opened.
2021-04-29 05:51:27,254 DEBUG Including index logstash-2021.11.22: Opened.
2021-04-29 05:51:27,274 DEBUG Including index logstash-2021.11.23: Opened.
2021-04-29 05:51:27,311 DEBUG Including index logstash-2021.11.24: Opened.
2021-04-29 05:51:27,331 DEBUG Including index logstash-2021.11.25: Opened.
2021-04-29 05:51:27,353 DEBUG Including index logstash-2021.11.26: Opened.
2021-04-29 05:51:27,378 DEBUG Including index logstash-2021.11.27: Opened.
2021-04-29 05:51:27,402 DEBUG Including index logstash-2021.11.28: Opened.
2021-04-29 05:51:27,442 DEBUG Including index logstash-2021.11.29: Opened.
2021-04-29 05:51:27,463 DEBUG Including index logstash-2021.11.30: Opened.
2021-04-29 05:51:27,488 DEBUG Including index logstash-2021.12.01: Opened.
2021-04-29 05:51:27,510 DEBUG Including index logstash-2021.12.02: Opened.
2021-04-29 05:51:27,529 DEBUG Including index logstash-2021.12.03: Opened.
2021-04-29 05:51:27,565 DEBUG Including index logstash-2021.12.04: Opened.
2021-04-29 05:51:27,588 DEBUG Including index logstash-2021.12.31: Opened.
2021-04-29 05:51:27,614 DEBUG Including index nagioslogserver: Opened.
2021-04-29 05:51:27,638 DEBUG Including index nagioslogserver_log: Opened.
2021-04-29 05:51:27,739 DEBUG Nodes with verified repository access: {u'JocVAvkORBynNF_ptl-hOA': {u'name': u'47ef3064-d6c5-4517-aee2-22255deaafc1'}, u'axHzfikURUuM6PFUhs0v1A': {u'name': u'63791bb4-9825-42f6-8cb9-a336bccba928'}}
2021-04-29 05:51:27,740 INFO Snapshot name: curator-20210429055127
2021-04-29 05:51:31,147 ERROR Unable to find all snapshots in repository: common
Traceback (most recent call last):
File "/usr/bin/curator", line 11, in <module>
sys.exit(main())
File "/usr/lib/python2.7/site-packages/curator/curator.py", line 5, in main
cli( obj={ "filters": [] } )
File "/usr/lib64/python2.7/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib64/python2.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib64/python2.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib64/python2.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib64/python2.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/lib/python2.7/site-packages/curator/cli/index_selection.py", line 167, in indices
retval = do_command(client, ctx.parent.info_name, working_list, ctx.parent.params, master_timeout)
File "/usr/lib/python2.7/site-packages/curator/cli/utils.py", line 250, in do_command
skip_repo_validation=params['skip_repo_validation'],
File "/usr/lib/python2.7/site-packages/curator/api/snapshot.py", line 72, in create_snapshot
if name in all_snaps:
TypeError: argument of type 'bool' is not iterable

I've confirmed the backup repository is available to both nodes from the CLI, and permissions and ownership look right.


Oddly enough the GUI does not show any snapshots on either node. Rather the GUI says, "No backups yet." The backup repository is a cifs share with about 322GB remaining free on a 2.4TB volume. How can I tell if the old backups have been purged? and how can I overcome the backup failures and free up some disk space?
Many thanks for your assistance.

[ssax]

Please PM me a copy of your profile, you can download it from Admin > System Status by clicking the Download System Profile button.

Send the output of these commands as well:
- Change the path to the proper path for your snapshot repo before running

Code: Select all

ls -ld /path/to/your/snapshot/repo
ls -lh /path/to/your/snapshot/repo

[ssax]

Please remove this (it's showing zero bytes and that apparently causes an issue):

Code: Select all

rm -rf /mnt/common_snapshot_repository/snapshot-curator-20201206202423

Then check the web interface to see if the snapshots are now showing.

I didn't see the profile attached, please PM me a copy of your profile, you can download it from Admin > System Status by clicking the Download System Profile button.

Please follow this, you may be hitting a watermark for the disk space:

https://support.nagios.com/kb/article/n ... h-469.html