Nagios Support Forum

our snapshot stop at 3/25 with the index name " logstash-2020.03.25 " and no new snapshot is created afterward

checked snaphots02 folder have no 0kb file

how to fix this , thanks!

Try running the snapshot manually from the command line of the NLS machine to try and get some details. This can be done with:
Make sure to replace $repository with the name of your repo which can be found under Admin > System > Snapshots & Maintenance > Maintenance and Repository Settings.

got this after run the command, thanks

[root@nxlog03 ~]# /usr/local/nagioslogserver/scripts/curator.sh snapshot --repository Snapshot02 --ignore_unavailable indices --older-than 1 --time-unit days --timestring %Y.%m.%d
2020-04-14 09:37:46,037 INFO Job starting: snapshot indices
2020-04-14 09:37:46,037 WARNING Overriding default connection timeout. New timeout: 21600
2020-04-14 09:37:46,067 INFO Action snapshot will be performed on the following indices: [u'logstash-2019.12.11', u'logstash-2019.12.31', u'logstash-2020.01.01', u'logstash-2020.01.02', u'logstash-2020.01.03', u'logstash-2020.01.04', u'logstash-2020.01.05', u'logstash-2020.01.06', u'logstash-2020.01.07', u'logstash-2020.01.08', u'logstash-2020.01.09', u'logstash-2020.01.10', u'logstash-2020.01.11', u'logstash-2020.01.12', u'logstash-2020.01.13', u'logstash-2020.01.14', u'logstash-2020.01.15', u'logstash-2020.01.16', u'logstash-2020.01.17', u'logstash-2020.01.18', u'logstash-2020.01.19', u'logstash-2020.01.20', u'logstash-2020.01.21', u'logstash-2020.01.22', u'logstash-2020.01.23', u'logstash-2020.01.24', u'logstash-2020.01.25', u'logstash-2020.01.26', u'logstash-2020.01.27', u'logstash-2020.01.28', u'logstash-2020.01.29', u'logstash-2020.01.30', u'logstash-2020.01.31', u'logstash-2020.02.01', u'logstash-2020.02.02', u'logstash-2020.02.03', u'logstash-2020.02.04', u'logstash-2020.02.05', u'logstash-2020.02.06', u'logstash-2020.02.07', u'logstash-2020.02.08', u'logstash-2020.02.09', u'logstash-2020.02.10', u'logstash-2020.02.11', u'logstash-2020.02.12', u'logstash-2020.02.13', u'logstash-2020.02.14', u'logstash-2020.02.15', u'logstash-2020.02.16', u'logstash-2020.02.17', u'logstash-2020.02.18', u'logstash-2020.02.19', u'logstash-2020.02.20', u'logstash-2020.02.21', u'logstash-2020.02.22', u'logstash-2020.02.23', u'logstash-2020.02.24', u'logstash-2020.02.25', u'logstash-2020.02.26', u'logstash-2020.02.27', u'logstash-2020.02.28', u'logstash-2020.02.29', u'logstash-2020.03.04', u'logstash-2020.03.09', u'logstash-2020.03.10', u'logstash-2020.03.31', u'logstash-2020.04.01', u'logstash-2020.04.02', u'logstash-2020.04.04', u'logstash-2020.04.05', u'logstash-2020.04.06', u'logstash-2020.04.07', u'logstash-2020.04.08', u'logstash-2020.04.09', u'logstash-2020.04.10', u'logstash-2020.04.11', u'logstash-2020.04.12', u'logstash-2020.04.13']
2020-04-14 09:37:52,639 ERROR Failed to verify all nodes have repository access.
2020-04-14 09:37:52,639 WARNING Job did not complete successfully.

snapshot mount point still have 2.6TB free and its a single node log server

[root@nxlog03 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 28M 7.8G 1% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/mapper/centos-root 4.0T 3.8T 33G 100% /
/dev/sda1 976M 197M 713M 22% /boot
10.10.xxx.xxx:/volume4/nagioslog01 11T 8.3T 2.6T 76% /usr/local/nagioslogserver/snapshots02
tmpfs 1.6G 0 1.6G 0% /run/user/1000
tmpfs 1.6G 0 1.6G 0% /run/user/0

tried 'touch 123.txt' at /usr/local/nagioslogserver/snapshots02 will no error

here is ls -hal at snapshot folder

root@nxlog03:/usr/local/nagioslogserver/snapshots02[root@nxlog03 snapshots02]# ls -hal
total 1.4M
drwxrwxr-x 4 nagios nagios 20K Apr 14 15:18 [0m[01;34m.[0m
drwxrwxr-x. 11 nagios nagios 4.0K Feb 18 16:09 [01;34m..[0m
---------- 1 root root 0 Apr 14 09:30 123.txt
drwxrwxr-x 3 nagios nagios 4.0K Mar 10 18:04 [01;34m@eaDir[0m
-rw-r--r-- 1 nagios users 1.3K Mar 26 14:09 index
drwxrwxr-x 127 nagios nagios 4.0K Mar 26 13:08 [01;34mindices[0m
-rwxrwxr-x 1 nagios nagios 520 Nov 23 03:05 [01;32mmetadata-curator-20191122190425[0m
-rw-r--r-- 1 nagios users 520 Nov 28 17:20 metadata-curator-20191128091857
-rw-r--r-- 1 nagios users 532 Nov 28 18:01 metadata-curator-20191128100018
-rw-r--r-- 1 nagios users 532 Nov 28 18:51 metadata-curator-20191128105005
-rw-r--r-- 1 nagios users 520 Nov 28 18:52 metadata-curator-20191128105053
-rw-r--r-- 1 nagios users 520 Nov 28 18:52 metadata-curator-20191128105122
-rw-r--r-- 1 nagios users 520 Nov 28 18:53 metadata-curator-20191128105157
-rw-r--r-- 1 nagios users 521 Nov 28 18:54 metadata-curator-20191128105248
-rw-r--r-- 1 nagios users 521 Nov 28 18:54 metadata-curator-20191128105307
-rw-r--r-- 1 nagios users 521 Nov 28 18:54 metadata-curator-20191128105330
-rw-r--r-- 1 nagios users 521 Nov 28 18:55 metadata-curator-20191128105403
-rw-r--r-- 1 nagios users 521 Nov 28 18:56 metadata-curator-20191128105438
-rw-r--r-- 1 nagios users 521 Nov 28 18:56 metadata-curator-20191128105510
-rw-r--r-- 1 nagios users 521 Nov 28 18:57 metadata-curator-20191128105553
-rw-r--r-- 1 nagios users 521 Nov 28 18:57 metadata-curator-20191128105612
-rw-r--r-- 1 nagios users 533 Nov 28 18:58 metadata-curator-20191128105653
-rw-r--r-- 1 nagios users 521 Nov 28 18:58 metadata-curator-20191128105711
-rw-r--r-- 1 nagios users 521 Nov 28 18:59 metadata-curator-20191128105744
-rw-r--r-- 1 nagios users 521 Nov 28 18:59 metadata-curator-20191128105814
-rw-r--r-- 1 nagios users 521 Nov 28 19:00 metadata-curator-20191128105842
-rw-r--r-- 1 nagios users 521 Nov 28 19:00 metadata-curator-20191128105906
-rw-r--r-- 1 nagios users 521 Nov 28 19:01 metadata-curator-20191128105939
-rw-r--r-- 1 nagios users 521 Nov 28 19:01 metadata-curator-20191128110021
-rw-r--r-- 1 nagios users 533 Nov 28 19:01 metadata-curator-20191128110034
-rw-r--r-- 1 nagios users 521 Nov 28 19:02 metadata-curator-20191128110100
-rw-r--r-- 1 nagios users 521 Nov 28 19:02 metadata-curator-20191128110135
-rw-r--r-- 1 nagios users 521 Nov 28 19:03 metadata-curator-20191128110159
-rw-r--r-- 1 nagios users 521 Nov 28 19:03 metadata-curator-20191128110222
-rw-r--r-- 1 nagios users 521 Nov 28 19:04 metadata-curator-20191128110249
-rw-r--r-- 1 nagios users 521 Nov 28 19:04 metadata-curator-20191128110316
-rw-r--r-- 1 nagios users 521 Nov 28 19:04 metadata-curator-20191128110335
-rw-r--r-- 1 nagios users 521 Nov 28 19:05 metadata-curator-20191128110406
-rw-r--r-- 1 nagios users 521 Nov 28 19:05 metadata-curator-20191128110425
-rw-r--r-- 1 nagios users 521 Nov 28 19:06 metadata-curator-20191128110448
-rw-r--r-- 1 nagios users 521 Nov 28 19:06 metadata-curator-20191128110510
-rw-r--r-- 1 nagios users 521 Nov 28 19:06 metadata-curator-20191128110532
-rw-r--r-- 1 nagios users 521 Nov 28 19:07 metadata-curator-20191128110556
-rw-r--r-- 1 nagios users 521 Nov 28 19:07 metadata-curator-20191128110628
-rw-r--r-- 1 nagios users 521 Nov 28 19:08 metadata-curator-20191128110640
-rw-r--r-- 1 nagios users 521 Nov 28 19:08 metadata-curator-20191128110702
-rw-r--r-- 1 nagios users 521 Nov 28 19:08 metadata-curator-20191128110717
-rw-r--r-- 1 nagios users 521 Nov 28 19:09 metadata-curator-20191128110744
-rw-r--r-- 1 nagios users 521 Nov 28 19:09 metadata-curator-20191128110759
-rw-r--r-- 1 nagios users 521 Nov 28 19:09 metadata-curator-20191128110821
-rw-r--r-- 1 nagios users 521 Nov 28 19:10 metadata-curator-20191128110839
-rw-r--r-- 1 nagios users 521 Nov 28 19:10 metadata-curator-20191128110859
-rw-r--r-- 1 nagios users 521 Nov 28 19:10 metadata-curator-20191128110918
-rw-r--r-- 1 nagios users 521 Nov 28 19:11 metadata-curator-20191128110937
-rw-r--r-- 1 nagios users 521 Nov 28 19:11 metadata-curator-20191128110954
-rw-r--r-- 1 nagios users 533 Nov 28 19:11 metadata-curator-20191128111014
-rw-r--r-- 1 nagios users 521 Nov 28 19:11 metadata-curator-20191128111029
-rw-r--r-- 1 nagios users 521 Nov 28 19:12 metadata-curator-20191128111048
-rw-r--r-- 1 nagios users 521 Nov 28 19:12 metadata-curator-20191128111113
-rw-r--r-- 1 nagios users 521 Nov 28 19:12 metadata-curator-20191128111119
-rw-r--r-- 1 nagios users 521 Nov 28 19:13 metadata-curator-20191128111138
-rw-r--r-- 1 nagios users 521 Nov 28 19:13 metadata-curator-20191128111154
-rw-r--r-- 1 nagios users 521 Nov 28 19:13 metadata-curator-20191128111208
-rw-r--r-- 1 nagios users 521 Nov 28 19:13 metadata-curator-20191128111224
-rw-r--r-- 1 nagios users 521 Nov 28 19:14 metadata-curator-20191128111239
-rw-r--r-- 1 nagios users 533 Nov 28 19:14 metadata-curator-20191128111253
-rw-r--r-- 1 nagios users 521 Nov 28 19:14 metadata-curator-20191128111310
-rw-r--r-- 1 nagios users 521 Nov 28 19:14 metadata-curator-20191128111323
-rw-r--r-- 1 nagios users 521 Nov 28 19:14 metadata-curator-20191128111335
-rw-r--r-- 1 nagios users 521 Nov 28 19:15 metadata-curator-20191128111349
-rw-r--r-- 1 nagios users 521 Nov 28 19:15 metadata-curator-20191128111402
-rw-r--r-- 1 nagios users 521 Nov 28 19:15 metadata-curator-20191128111416
-rw-r--r-- 1 nagios users 521 Nov 28 19:15 metadata-curator-20191128111428
-rw-r--r-- 1 nagios users 521 Nov 28 19:16 metadata-curator-20191128111447
-rw-r--r-- 1 nagios users 521 Nov 28 19:16 metadata-curator-20191128111450
-rw-r--r-- 1 nagios users 521 Nov 28 19:16 metadata-curator-20191128111503
-rw-r--r-- 1 nagios users 521 Nov 28 19:16 metadata-curator-20191128111515
-rw-r--r-- 1 nagios users 521 Nov 28 19:16 metadata-curator-20191128111524
-rw-r--r-- 1 nagios users 521 Nov 28 19:16 metadata-curator-20191128111535
-rw-r--r-- 1 nagios users 521 Nov 28 19:17 metadata-curator-20191128111545
-rw-r--r-- 1 nagios users 521 Nov 28 19:17 metadata-curator-20191128111558
-rw-r--r-- 1 nagios users 521 Nov 28 19:17 metadata-curator-20191128111605
-rw-r--r-- 1 nagios users 521 Nov 28 19:17 metadata-curator-20191128111613
-rw-r--r-- 1 nagios users 521 Nov 28 19:17 metadata-curator-20191128111622
-rw-r--r-- 1 nagios users 521 Nov 28 19:17 metadata-curator-20191128111631
-rw-r--r-- 1 nagios users 521 Nov 28 19:18 metadata-curator-20191128111640
-rw-r--r-- 1 nagios users 521 Nov 28 19:18 metadata-curator-20191128111647
-rw-r--r-- 1 nagios users 521 Nov 28 19:18 metadata-curator-20191128111654
-rw-r--r-- 1 nagios users 533 Nov 28 19:18 metadata-curator-20191128111707
-rw-r--r-- 1 nagios users 521 Nov 28 19:18 metadata-curator-20191128111709
-rw-r--r-- 1 nagios users 533 Nov 28 19:18 metadata-curator-20191128111716
-rw-r--r-- 1 nagios users 521 Nov 28 19:18 metadata-curator-20191128111724
-rw-r--r-- 1 nagios users 533 Nov 28 19:18 metadata-curator-20191128111735
-rw-r--r-- 1 nagios users 521 Nov 28 19:19 metadata-curator-20191128111740
-rw-r--r-- 1 nagios users 521 Nov 28 19:19 metadata-curator-20191128111747
-rw-r--r-- 1 nagios users 521 Nov 28 19:19 metadata-curator-20191128111753
-rw-r--r-- 1 nagios users 521 Nov 28 19:19 metadata-curator-20191128111758
-rw-r--r-- 1 nagios users 521 Nov 28 19:19 metadata-curator-20191128111805
-rw-r--r-- 1 nagios users 521 Nov 28 19:19 metadata-curator-20191128111809
-rw-r--r-- 1 nagios users 521 Nov 28 19:19 metadata-curator-20191128111817
-rw-r--r-- 1 nagios users 521 Nov 28 19:19 metadata-curator-20191128111823
-rw-r--r-- 1 nagios users 533 Nov 28 19:19 metadata-curator-20191128111827
-rw-r--r-- 1 nagios users 521 Nov 28 19:19 metadata-curator-20191128111830
-rw-r--r-- 1 nagios users 521 Nov 28 19:20 metadata-curator-20191128111834
-rw-r--r-- 1 nagios users 533 Nov 28 19:20 metadata-curator-20191128111842
-rw-r--r-- 1 nagios users 521 Nov 29 17:19 metadata-curator-20191129091801
-rw-r--r-- 1 nagios users 521 Nov 30 23:19 metadata-curator-20191130151801
-rw-r--r-- 1 nagios users 521 Dec 1 18:24 metadata-curator-20191201102307
-rw-r--r-- 1 nagios users 521 Dec 2 19:33 metadata-curator-20191202113208
-rw-r--r-- 1 nagios users 534 Dec 3 19:53 metadata-curator-20191203115219
-rw-r--r-- 1 nagios users 522 Dec 4 18:13 metadata-curator-20191204101140
-rw-r--r-- 1 nagios users 522 Dec 5 18:22 metadata-curator-20191205102040
-rw-r--r-- 1 nagios users 522 Dec 6 17:57 metadata-curator-20191206095539
-rw-r--r-- 1 nagios users 522 Dec 7 18:09 metadata-curator-20191207100751
-rw-r--r-- 1 nagios users 522 Dec 8 18:24 metadata-curator-20191208102238
-rw-r--r-- 1 nagios users 522 Dec 9 19:39 metadata-curator-20191209113812
-rw-r--r-- 1 nagios users 522 Dec 10 18:10 metadata-curator-20191210100828
-rw-r--r-- 1 nagios users 534 Dec 11 18:21 metadata-curator-20191211101943
-rw-r--r-- 1 nagios users 522 Dec 12 18:25 metadata-curator-20191212102321
-rw-r--r-- 1 nagios users 534 Dec 13 23:41 metadata-curator-20191213153921
-rw-r--r-- 1 nagios users 522 Dec 15 05:40 metadata-curator-20191214213813
-rw-r--r-- 1 nagios users 522 Dec 16 11:33 metadata-curator-20191216033157
-rw-r--r-- 1 nagios users 522 Dec 17 11:50 metadata-curator-20191217034840
-rw-r--r-- 1 nagios users 520 Jan 13 15:44 metadata-curator-20200113074223
-rw-r--r-- 1 nagios users 532 Jan 13 15:44 metadata-curator-20200113074224
-rw-r--r-- 1 nagios users 520 Jan 13 15:44 metadata-curator-20200113074235
-rw-r--r-- 1 nagios users 520 Jan 13 18:22 metadata-curator-20200113102023
-rw-r--r-- 1 nagios users 520 Jan 19 20:47 metadata-curator-20200119124533
-rw-r--r-- 1 nagios users 520 Jan 20 21:07 metadata-curator-20200120130457
-rw-r--r-- 1 nagios users 520 Jan 21 20:48 metadata-curator-20200121124611
-rw-r--r-- 1 nagios users 520 Jan 22 20:48 metadata-curator-20200122124610
-rw-r--r-- 1 nagios users 521 Feb 26 12:38 metadata-curator-20200226043530
-rw-r--r-- 1 nagios users 533 Feb 27 01:17 metadata-curator-20200226171421
-rw-r--r-- 1 nagios users 521 Feb 28 02:40 metadata-curator-20200227183749
-rw-r--r-- 1 nagios users 521 Feb 29 02:46 metadata-curator-20200228184303
-rw-r--r-- 1 nagios users 521 Mar 1 03:11 metadata-curator-20200229190853
-rw-r--r-- 1 nagios users 521 Mar 2 03:15 metadata-curator-20200301191212
-rw-r--r-- 1 nagios users 521 Mar 3 04:08 metadata-curator-20200302200519
-rw-r--r-- 1 nagios users 521 Mar 4 02:26 metadata-curator-20200303182258
-rw-r--r-- 1 nagios users 521 Mar 5 02:05 metadata-curator-20200304180235
-rw-r--r-- 1 nagios users 521 Mar 6 02:20 metadata-curator-20200305181740
-rw-r--r-- 1 nagios users 521 Mar 7 02:09 metadata-curator-20200306180604
-rw-r--r-- 1 nagios users 521 Mar 8 02:44 metadata-curator-20200307184059
-rw-r--r-- 1 nagios users 521 Mar 9 02:30 metadata-curator-20200308182725
-rw-r--r-- 1 nagios users 521 Mar 10 03:48 metadata-curator-20200309194452
-rw-r--r-- 1 nagios users 521 Mar 11 02:37 metadata-curator-20200310183402
-rw-r--r-- 1 nagios users 533 Mar 12 02:35 metadata-curator-20200311183214
-rw-r--r-- 1 nagios users 521 Mar 13 02:13 metadata-curator-20200312181025
-rw-r--r-- 1 nagios users 521 Mar 14 01:58 metadata-curator-20200313175544
-rw-r--r-- 1 nagios users 521 Mar 15 02:21 metadata-curator-20200314181755
-rw-r--r-- 1 nagios users 521 Mar 16 02:33 metadata-curator-20200315182958
-rw-r--r-- 1 nagios users 521 Mar 16 14:09 metadata-curator-20200316060622
-rw-r--r-- 1 nagios users 521 Mar 17 12:22 metadata-curator-20200317041918
-rw-r--r-- 1 nagios users 521 Mar 18 12:15 metadata-curator-20200318041216
-rw-r--r-- 1 nagios users 521 Mar 19 12:27 metadata-curator-20200319042353
-rw-r--r-- 1 nagios users 521 Mar 20 12:13 metadata-curator-20200320041029
-rw-r--r-- 1 nagios users 521 Mar 21 13:28 metadata-curator-20200321052523
-rw-r--r-- 1 nagios users 521 Mar 22 12:42 metadata-curator-20200322043920
-rw-r--r-- 1 nagios users 521 Mar 23 14:23 metadata-curator-20200323061939
-rw-r--r-- 1 nagios users 521 Mar 24 12:43 metadata-curator-20200324044034
-rw-r--r-- 1 nagios users 521 Mar 25 18:00 metadata-curator-20200325095757
-rw-r--r-- 1 nagios users 521 Mar 26 13:08 metadata-curator-20200326050521
-rwxrwxr-x 1 nagios nagios 350 Nov 23 03:05 [01;32msnapshot-curator-20191122190425[0m
-rw-r--r-- 1 nagios users 291 Nov 28 17:33 snapshot-curator-20191128091857
-rw-r--r-- 1 nagios users 287 Nov 28 18:01 snapshot-curator-20191128100018
-rw-r--r-- 1 nagios users 286 Nov 28 18:51 snapshot-curator-20191128105005
-rw-r--r-- 1 nagios users 288 Nov 28 18:52 snapshot-curator-20191128105053
-rw-r--r-- 1 nagios users 286 Nov 28 18:52 snapshot-curator-20191128105122
-rw-r--r-- 1 nagios users 287 Nov 28 18:53 snapshot-curator-20191128105157
-rw-r--r-- 1 nagios users 287 Nov 28 18:54 snapshot-curator-20191128105248
-rw-r--r-- 1 nagios users 289 Nov 28 18:54 snapshot-curator-20191128105307
-rw-r--r-- 1 nagios users 289 Nov 28 18:54 snapshot-curator-20191128105330
-rw-r--r-- 1 nagios users 288 Nov 28 18:55 snapshot-curator-20191128105403
-rw-r--r-- 1 nagios users 289 Nov 28 18:56 snapshot-curator-20191128105438
-rw-r--r-- 1 nagios users 291 Nov 28 18:56 snapshot-curator-20191128105510
-rw-r--r-- 1 nagios users 288 Nov 28 18:57 snapshot-curator-20191128105553
-rw-r--r-- 1 nagios users 288 Nov 28 18:57 snapshot-curator-20191128105612
-rw-r--r-- 1 nagios users 288 Nov 28 18:58 snapshot-curator-20191128105653
-rw-r--r-- 1 nagios users 288 Nov 28 18:58 snapshot-curator-20191128105711
-rw-r--r-- 1 nagios users 288 Nov 28 18:59 snapshot-curator-20191128105744
-rw-r--r-- 1 nagios users 288 Nov 28 18:59 snapshot-curator-20191128105814
-rw-r--r-- 1 nagios users 288 Nov 28 19:00 snapshot-curator-20191128105842
-rw-r--r-- 1 nagios users 288 Nov 28 19:00 snapshot-curator-20191128105906
-rw-r--r-- 1 nagios users 291 Nov 28 19:01 snapshot-curator-20191128105939
-rw-r--r-- 1 nagios users 288 Nov 28 19:01 snapshot-curator-20191128110021
-rw-r--r-- 1 nagios users 288 Nov 28 19:02 snapshot-curator-20191128110034
-rw-r--r-- 1 nagios users 291 Nov 28 19:02 snapshot-curator-20191128110100
-rw-r--r-- 1 nagios users 288 Nov 28 19:03 snapshot-curator-20191128110135
-rw-r--r-- 1 nagios users 290 Nov 28 19:03 snapshot-curator-20191128110159
-rw-r--r-- 1 nagios users 286 Nov 28 19:03 snapshot-curator-20191128110222
-rw-r--r-- 1 nagios users 290 Nov 28 19:04 snapshot-curator-20191128110249
-rw-r--r-- 1 nagios users 288 Nov 28 19:04 snapshot-curator-20191128110316
-rw-r--r-- 1 nagios users 288 Nov 28 19:05 snapshot-curator-20191128110335
-rw-r--r-- 1 nagios users 289 Nov 28 19:05 snapshot-curator-20191128110406
-rw-r--r-- 1 nagios users 288 Nov 28 19:05 snapshot-curator-20191128110425
-rw-r--r-- 1 nagios users 289 Nov 28 19:06 snapshot-curator-20191128110448
-rw-r--r-- 1 nagios users 291 Nov 28 19:06 snapshot-curator-20191128110510
-rw-r--r-- 1 nagios users 288 Nov 28 19:06 snapshot-curator-20191128110532
-rw-r--r-- 1 nagios users 288 Nov 28 19:07 snapshot-curator-20191128110556
-rw-r--r-- 1 nagios users 289 Nov 28 19:07 snapshot-curator-20191128110628
-rw-r--r-- 1 nagios users 291 Nov 28 19:08 snapshot-curator-20191128110640
-rw-r--r-- 1 nagios users 286 Nov 28 19:08 snapshot-curator-20191128110702
-rw-r--r-- 1 nagios users 288 Nov 28 19:08 snapshot-curator-20191128110717
-rw-r--r-- 1 nagios users 287 Nov 28 19:09 snapshot-curator-20191128110744
-rw-r--r-- 1 nagios users 290 Nov 28 19:09 snapshot-curator-20191128110759
-rw-r--r-- 1 nagios users 288 Nov 28 19:09 snapshot-curator-20191128110821
-rw-r--r-- 1 nagios users 290 Nov 28 19:10 snapshot-curator-20191128110839
-rw-r--r-- 1 nagios users 290 Nov 28 19:10 snapshot-curator-20191128110859
-rw-r--r-- 1 nagios users 289 Nov 28 19:10 snapshot-curator-20191128110918
-rw-r--r-- 1 nagios users 289 Nov 28 19:11 snapshot-curator-20191128110937
-rw-r--r-- 1 nagios users 288 Nov 28 19:11 snapshot-curator-20191128110954
-rw-r--r-- 1 nagios users 288 Nov 28 19:11 snapshot-curator-20191128111014
-rw-r--r-- 1 nagios users 291 Nov 28 19:11 snapshot-curator-20191128111029
-rw-r--r-- 1 nagios users 289 Nov 28 19:12 snapshot-curator-20191128111048
-rw-r--r-- 1 nagios users 287 Nov 28 19:12 snapshot-curator-20191128111113
-rw-r--r-- 1 nagios users 289 Nov 28 19:12 snapshot-curator-20191128111119
-rw-r--r-- 1 nagios users 290 Nov 28 19:13 snapshot-curator-20191128111138
-rw-r--r-- 1 nagios users 288 Nov 28 19:13 snapshot-curator-20191128111154
-rw-r--r-- 1 nagios users 289 Nov 28 19:13 snapshot-curator-20191128111208
-rw-r--r-- 1 nagios users 288 Nov 28 19:13 snapshot-curator-20191128111224
-rw-r--r-- 1 nagios users 290 Nov 28 19:14 snapshot-curator-20191128111239
-rw-r--r-- 1 nagios users 288 Nov 28 19:14 snapshot-curator-20191128111253
-rw-r--r-- 1 nagios users 291 Nov 28 19:14 snapshot-curator-20191128111310
-rw-r--r-- 1 nagios users 288 Nov 28 19:14 snapshot-curator-20191128111323
-rw-r--r-- 1 nagios users 288 Nov 28 19:15 snapshot-curator-20191128111335
-rw-r--r-- 1 nagios users 291 Nov 28 19:15 snapshot-curator-20191128111349
-rw-r--r-- 1 nagios users 288 Nov 28 19:15 snapshot-curator-20191128111402
-rw-r--r-- 1 nagios users 288 Nov 28 19:15 snapshot-curator-20191128111416
-rw-r--r-- 1 nagios users 289 Nov 28 19:15 snapshot-curator-20191128111428
-rw-r--r-- 1 nagios users 289 Nov 28 19:16 snapshot-curator-20191128111447
-rw-r--r-- 1 nagios users 291 Nov 28 19:16 snapshot-curator-20191128111450
-rw-r--r-- 1 nagios users 288 Nov 28 19:16 snapshot-curator-20191128111503
-rw-r--r-- 1 nagios users 288 Nov 28 19:16 snapshot-curator-20191128111515
-rw-r--r-- 1 nagios users 288 Nov 28 19:16 snapshot-curator-20191128111524
-rw-r--r-- 1 nagios users 289 Nov 28 19:17 snapshot-curator-20191128111535
-rw-r--r-- 1 nagios users 288 Nov 28 19:17 snapshot-curator-20191128111545
-rw-r--r-- 1 nagios users 289 Nov 28 19:17 snapshot-curator-20191128111558
-rw-r--r-- 1 nagios users 289 Nov 28 19:17 snapshot-curator-20191128111605
-rw-r--r-- 1 nagios users 288 Nov 28 19:17 snapshot-curator-20191128111613
-rw-r--r-- 1 nagios users 288 Nov 28 19:17 snapshot-curator-20191128111622
-rw-r--r-- 1 nagios users 288 Nov 28 19:17 snapshot-curator-20191128111631
-rw-r--r-- 1 nagios users 291 Nov 28 19:18 snapshot-curator-20191128111640
-rw-r--r-- 1 nagios users 289 Nov 28 19:18 snapshot-curator-20191128111647
-rw-r--r-- 1 nagios users 288 Nov 28 19:18 snapshot-curator-20191128111654
-rw-r--r-- 1 nagios users 289 Nov 28 19:18 snapshot-curator-20191128111707
-rw-r--r-- 1 nagios users 290 Nov 28 19:18 snapshot-curator-20191128111709
-rw-r--r-- 1 nagios users 289 Nov 28 19:18 snapshot-curator-20191128111716
-rw-r--r-- 1 nagios users 289 Nov 28 19:18 snapshot-curator-20191128111724
-rw-r--r-- 1 nagios users 288 Nov 28 19:19 snapshot-curator-20191128111735
-rw-r--r-- 1 nagios users 291 Nov 28 19:19 snapshot-curator-20191128111740
-rw-r--r-- 1 nagios users 289 Nov 28 19:19 snapshot-curator-20191128111747
-rw-r--r-- 1 nagios users 287 Nov 28 19:19 snapshot-curator-20191128111753
-rw-r--r-- 1 nagios users 289 Nov 28 19:19 snapshot-curator-20191128111758
-rw-r--r-- 1 nagios users 289 Nov 28 19:19 snapshot-curator-20191128111805
-rw-r--r-- 1 nagios users 290 Nov 28 19:19 snapshot-curator-20191128111809
-rw-r--r-- 1 nagios users 290 Nov 28 19:19 snapshot-curator-20191128111817
-rw-r--r-- 1 nagios users 288 Nov 28 19:19 snapshot-curator-20191128111823
-rw-r--r-- 1 nagios users 288 Nov 28 19:19 snapshot-curator-20191128111827
-rw-r--r-- 1 nagios users 291 Nov 28 19:20 snapshot-curator-20191128111830
-rw-r--r-- 1 nagios users 289 Nov 28 19:20 snapshot-curator-20191128111834
-rw-r--r-- 1 nagios users 288 Nov 28 19:20 snapshot-curator-20191128111842
-rw-r--r-- 1 nagios users 297 Nov 29 18:38 snapshot-curator-20191129091801
-rw-r--r-- 1 nagios users 302 Dec 1 01:03 snapshot-curator-20191130151801
-rw-r--r-- 1 nagios users 309 Dec 1 20:10 snapshot-curator-20191201102307
-rw-r--r-- 1 nagios users 314 Dec 2 20:36 snapshot-curator-20191202113208
-rw-r--r-- 1 nagios users 322 Dec 3 20:52 snapshot-curator-20191203115219
-rw-r--r-- 1 nagios users 328 Dec 4 18:51 snapshot-curator-20191204101140
-rw-r--r-- 1 nagios users 331 Dec 5 18:58 snapshot-curator-20191205102040
-rw-r--r-- 1 nagios users 337 Dec 6 18:31 snapshot-curator-20191206095539
-rw-r--r-- 1 nagios users 340 Dec 7 18:50 snapshot-curator-20191207100751
-rw-r--r-- 1 nagios users 344 Dec 8 19:50 snapshot-curator-20191208102238
-rw-r--r-- 1 nagios users 350 Dec 9 20:45 snapshot-curator-20191209113812
-rw-r--r-- 1 nagios users 355 Dec 10 18:47 snapshot-curator-20191210100828
-rw-r--r-- 1 nagios users 358 Dec 11 19:00 snapshot-curator-20191211101943
-rw-r--r-- 1 nagios users 358 Dec 12 19:02 snapshot-curator-20191212102321
-rw-r--r-- 1 nagios users 359 Dec 14 03:08 snapshot-curator-20191213153921
-rw-r--r-- 1 nagios users 358 Dec 15 08:16 snapshot-curator-20191214213813
-rw-r--r-- 1 nagios users 357 Dec 16 14:37 snapshot-curator-20191216033157
-rw-r--r-- 1 nagios users 360 Dec 17 15:09 snapshot-curator-20191217034840
-rw-r--r-- 1 nagios users 380 Jan 13 15:44 snapshot-curator-20200113074223
-rw-r--r-- 1 nagios users 381 Jan 13 15:44 snapshot-curator-20200113074224
-rw-r--r-- 1 nagios users 380 Jan 13 15:44 snapshot-curator-20200113074235
-rw-r--r-- 1 nagios users 380 Jan 13 18:22 snapshot-curator-20200113102023
-rw-r--r-- 1 nagios users 356 Jan 19 20:47 snapshot-curator-20200119124533
-rw-r--r-- 1 nagios users 375 Jan 20 21:07 snapshot-curator-20200120130457
-rw-r--r-- 1 nagios users 375 Jan 21 20:48 snapshot-curator-20200121124611
-rw-r--r-- 1 nagios users 373 Jan 22 20:48 snapshot-curator-20200122124610
-rw-r--r-- 1 nagios users 281 Feb 26 20:27 snapshot-curator-20200226043530
-rw-r--r-- 1 nagios users 276 Feb 27 01:17 snapshot-curator-20200226171421
-rw-r--r-- 1 nagios users 279 Feb 28 03:38 snapshot-curator-20200227183749
-rw-r--r-- 1 nagios users 279 Feb 29 03:42 snapshot-curator-20200228184303
-rw-r--r-- 1 nagios users 279 Mar 1 05:13 snapshot-curator-20200229190853
-rw-r--r-- 1 nagios users 279 Mar 2 04:45 snapshot-curator-20200301191212
-rw-r--r-- 1 nagios users 281 Mar 3 05:24 snapshot-curator-20200302200519
-rw-r--r-- 1 nagios users 282 Mar 4 03:10 snapshot-curator-20200303182258
-rw-r--r-- 1 nagios users 280 Mar 5 02:47 snapshot-curator-20200304180235
-rw-r--r-- 1 nagios users 282 Mar 6 02:59 snapshot-curator-20200305181740
-rw-r--r-- 1 nagios users 255 Mar 7 02:45 snapshot-curator-20200306180604
-rw-r--r-- 1 nagios users 253 Mar 8 03:49 snapshot-curator-20200307184059
-rw-r--r-- 1 nagios users 254 Mar 9 03:45 snapshot-curator-20200308182725
-rw-r--r-- 1 nagios users 252 Mar 10 05:03 snapshot-curator-20200309194452
-rw-r--r-- 1 nagios users 251 Mar 11 03:29 snapshot-curator-20200310183402
-rw-r--r-- 1 nagios users 253 Mar 12 03:20 snapshot-curator-20200311183214
-rw-r--r-- 1 nagios users 258 Mar 13 02:51 snapshot-curator-20200312181025
-rw-r--r-- 1 nagios users 263 Mar 14 02:33 snapshot-curator-20200313175544
-rw-r--r-- 1 nagios users 268 Mar 15 03:42 snapshot-curator-20200314181755
-rw-r--r-- 1 nagios users 274 Mar 16 03:42 snapshot-curator-20200315182958
-rw-r--r-- 1 nagios users 249 Mar 16 15:21 snapshot-curator-20200316060622
-rw-r--r-- 1 nagios users 255 Mar 17 13:06 snapshot-curator-20200317041918
-rw-r--r-- 1 nagios users 259 Mar 18 12:55 snapshot-curator-20200318041216
-rw-r--r-- 1 nagios users 264 Mar 19 13:05 snapshot-curator-20200319042353
-rw-r--r-- 1 nagios users 268 Mar 20 12:50 snapshot-curator-20200320041029
-rw-r--r-- 1 nagios users 274 Mar 21 14:12 snapshot-curator-20200321052523
-rw-r--r-- 1 nagios users 279 Mar 22 14:52 snapshot-curator-20200322043920
-rw-r--r-- 1 nagios users 244 Mar 23 15:36 snapshot-curator-20200323061939
-rw-r--r-- 1 nagios users 248 Mar 24 13:42 snapshot-curator-20200324044034
-rw-r--r-- 1 nagios users 277 Mar 25 20:16 snapshot-curator-20200325095757
-rw-r--r-- 1 nagios users 280 Mar 26 14:09 snapshot-curator-20200326050521
---------- 1 nagios users 22 Apr 8 13:10 tests-0aPUV8F0Ri-GW9PZefZApA-master
---------- 1 nagios users 22 Apr 2 13:34 tests-1d4X0xNERUOl0rMixfeVEQ-master
---------- 1 nagios users 22 Apr 10 12:49 tests-_4lWJk2ORtm1pCKyFW6lrA-master
---------- 1 nagios users 22 Apr 4 14:19 tests-4QXQwVDQSFymjOyF2x_iPA-master
---------- 1 nagios users 22 Apr 14 09:29 tests-7U0idgFeRiifodDmxU6i5w-master
---------- 1 nagios users 22 Apr 5 11:27 tests-AUYYqNusSOq1cI5TbeG5mQ-master
---------- 1 nagios users 22 Mar 29 13:29 tests-BQGzIghsRpSU1hUXJqeYwA-master
---------- 1 nagios users 22 Apr 9 13:24 tests-Ck_noquWTGeE6DM3OUEPkQ-master
---------- 1 nagios users 22 Apr 14 15:16 tests-DEs9wJFOTouJQaPU78iQtw-master
---------- 1 nagios users 22 Apr 12 13:28 tests-I3XcDS13STePhXbWAEY7fA-master
---------- 1 nagios users 22 Apr 7 13:47 tests-IaiccE23QOq7Z6KqUTLvsA-master
---------- 1 nagios users 22 Apr 11 14:31 tests-iBMMkaDDQgW-GKPhliD4Ew-master
---------- 1 nagios users 22 Apr 14 09:41 tests-luz1nO0QSFicU6vw89uZJw-master
---------- 1 nagios users 22 Apr 14 13:24 tests-MdFtRWaURSiawmE0I4jzAw-master
---------- 1 nagios users 22 Apr 14 09:28 tests--MSf0FsEQgq3IPp0-rY4CQ-master
---------- 1 nagios users 22 Apr 14 09:30 tests-_Ng3vEUOQQmyRIttsErOVw-master
---------- 1 nagios users 22 Mar 27 12:48 tests-NOoGYHENSjqerdaUYx0KAQ-master
---------- 1 nagios users 22 Apr 14 15:18 tests-Ram5KTHhTS6qB-W2cwubHA-master
---------- 1 nagios users 22 Apr 3 12:54 tests-rMN14MRaQHySuml0vJX0uA-master
---------- 1 nagios users 22 Apr 1 13:12 tests-t_KclR_MTOKJcGKiV5qWug-master
---------- 1 nagios users 22 Apr 13 15:56 tests-v9t6fZkqSXOVDDUBEZPaLw-master
---------- 1 nagios users 22 Mar 30 15:46 tests-WjXoH6lOQpqQs1Mj0QEzPw-master
---------- 1 nagios users 22 Mar 31 13:40 tests-WkcGyJrbTkuIHG1SMXh_tA-master
---------- 1 nagios users 22 Apr 6 15:49 tests-WzjwfvyQQrK0Sz2CyJuWGw-master
---------- 1 nagios users 22 Mar 28 14:24 tests-yy5RcCvoRUe8MZgDrXrJpA-master
]0;root@nxlog03:/usr/local/nagioslogserver/snapshots02[root@nxlog03 snapshots02]#

I'm seeing results for nxlog03 but is there other nodes like nxlog01 or nxlog02 ? Check /usr/local/nagioslogserver/var/cluster_hosts and make sure that all the other nodes in the cluster have the same remote filesystem mounted.

[root@nxlog03 var]# cat cluster_hosts
localhost
10.10.xxx.xxx[root@nxlog03 var]#

only 1 ip and localhost in cluster_hosts file

is it the snapshot too full ? still have 2.6TB left

Please PM me a profile from the system. It can be gathered under Admin > System > System Status > Download System Profile or from the command line with:
This will create /tmp/system-profile.tar.gz.

Note that this file can be very large and may not be able to be uploaded through the ticketing system. This is usually due to the logs in the Logstash and/or Elasticsearch directories found in it. If it is too large, please open the profile, extract these directories/files and send them separately.

I'd also like to get a copy of the current settings index. This can be gathered by running:
The file it creates and that we'd like to see is /tmp/nagioslogserver.tar.gz.