Log Server does not create backups

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
vmesquita
Posts: 315
Joined: Fri Aug 10, 2012 12:52 pm

Log Server does not create backups

Post by vmesquita »

Hello!

I created a mounted folder on both instances, poiting to the same server for backup. But still log server is not populating the backups. What could be happening?
backup_log_server.png
You do not have the required permissions to view the files attached to this post.
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Log Server does not create backups

Post by mcapra »

Can you run the following form the Nagios Log Server CLI and share the output:

Code: Select all

curator --dry-run snapshot --repository Backup indices --older-than 1 --time-unit days --timestring %Y.%m.%d
Former Nagios employee
https://www.mcapra.com/
Top
vmesquita
Posts: 315
Joined: Fri Aug 10, 2012 12:52 pm

Re: Log Server does not create backups

Post by vmesquita »

Here's the output:

Code: Select all

2016-08-23 12:35:20,303 INFO      Job starting: snapshot indices
2016-08-23 12:35:20,303 WARNING   Overriding default connection timeout.  New timeout: 21600
2016-08-23 12:35:20,364 INFO      Action snapshot will be performed on the following indices: [u'logstash-2015.01.01', u'logstash-2015.01.02', u'logstash-2015.01.03', u'logstash-2015.01.04', u'logstash-2015.01.05', u'logstash-2015.01.06', u'logstash-2015.01.07', u'logstash-2015.08.06', u'logstash-2015.11.16', u'logstash-2015.11.17', u'logstash-2015.11.18', u'logstash-2015.11.19', u'logstash-2015.11.20', u'logstash-2015.11.21', u'logstash-2015.11.22', u'logstash-2015.11.23', u'logstash-2015.11.24', u'logstash-2015.11.25', u'logstash-2015.11.26', u'logstash-2015.11.27', u'logstash-2015.11.28', u'logstash-2015.11.29', u'logstash-2015.11.30', u'logstash-2015.12.01', u'logstash-2015.12.02', u'logstash-2015.12.03', u'logstash-2015.12.04', u'logstash-2015.12.05', u'logstash-2015.12.06', u'logstash-2015.12.07', u'logstash-2015.12.08', u'logstash-2015.12.09', u'logstash-2015.12.10', u'logstash-2015.12.11', u'logstash-2015.12.12', u'logstash-2015.12.13', u'logstash-2015.12.14', u'logstash-2015.12.15', u'logstash-2015.12.16', u'logstash-2015.12.17', u'logstash-2015.12.18', u'logstash-2015.12.19', u'logstash-2015.12.20', u'logstash-2015.12.21', u'logstash-2015.12.22', u'logstash-2015.12.23', u'logstash-2015.12.24', u'logstash-2015.12.25', u'logstash-2015.12.26', u'logstash-2015.12.27', u'logstash-2015.12.28', u'logstash-2015.12.29', u'logstash-2015.12.30', u'logstash-2015.12.31', u'logstash-2016.01.01', u'logstash-2016.01.02', u'logstash-2016.01.03', u'logstash-2016.01.04', u'logstash-2016.01.05', u'logstash-2016.01.06', u'logstash-2016.01.07', u'logstash-2016.01.08', u'logstash-2016.01.09', u'logstash-2016.01.10', u'logstash-2016.01.11', u'logstash-2016.01.12', u'logstash-2016.01.13', u'logstash-2016.01.14', u'logstash-2016.01.15', u'logstash-2016.01.16', u'logstash-2016.01.17', u'logstash-2016.01.18', u'logstash-2016.01.19', u'logstash-2016.01.20', u'logstash-2016.01.21', u'logstash-2016.01.22', u'logstash-2016.01.23', u'logstash-2016.01.24', u'logstash-2016.01.25', u'logstash-2016.01.26', u'logstash-2016.01.27', u'logstash-2016.01.28', u'logstash-2016.01.29', u'logstash-2016.01.30', u'logstash-2016.01.31', u'logstash-2016.02.01', u'logstash-2016.02.02', u'logstash-2016.02.03', u'logstash-2016.02.04', u'logstash-2016.02.05', u'logstash-2016.02.06', u'logstash-2016.02.07', u'logstash-2016.02.08', u'logstash-2016.02.09', u'logstash-2016.02.10', u'logstash-2016.02.11', u'logstash-2016.02.12', u'logstash-2016.02.13', u'logstash-2016.02.14', u'logstash-2016.02.15', u'logstash-2016.02.16', u'logstash-2016.02.17', u'logstash-2016.02.18', u'logstash-2016.02.19', u'logstash-2016.02.20', u'logstash-2016.02.21', u'logstash-2016.02.22', u'logstash-2016.02.23', u'logstash-2016.02.24', u'logstash-2016.02.25', u'logstash-2016.02.26', u'logstash-2016.02.27', u'logstash-2016.02.28', u'logstash-2016.02.29', u'logstash-2016.03.01', u'logstash-2016.03.02', u'logstash-2016.03.03', u'logstash-2016.03.04', u'logstash-2016.03.05', u'logstash-2016.03.06', u'logstash-2016.03.07', u'logstash-2016.03.08', u'logstash-2016.03.09', u'logstash-2016.03.10', u'logstash-2016.03.11', u'logstash-2016.03.12', u'logstash-2016.03.13', u'logstash-2016.03.14', u'logstash-2016.03.15', u'logstash-2016.03.16', u'logstash-2016.03.17', u'logstash-2016.03.18', u'logstash-2016.03.19', u'logstash-2016.03.20', u'logstash-2016.03.21', u'logstash-2016.03.22', u'logstash-2016.03.23', u'logstash-2016.03.24', u'logstash-2016.03.25', u'logstash-2016.03.26', u'logstash-2016.03.27', u'logstash-2016.03.28', u'logstash-2016.03.29', u'logstash-2016.03.30', u'logstash-2016.03.31', u'logstash-2016.04.01', u'logstash-2016.04.02', u'logstash-2016.04.03', u'logstash-2016.04.04', u'logstash-2016.04.05', u'logstash-2016.04.06', u'logstash-2016.04.07', u'logstash-2016.04.08', u'logstash-2016.04.09', u'logstash-2016.04.10', u'logstash-2016.04.11', u'logstash-2016.04.12', u'logstash-2016.04.13', u'logstash-2016.04.14', u'logstash-2016.04.15', u'logstash-2016.04.16', u'logstash-2016.04.17', u'logstash-2016.04.18', u'logstash-2016.04.19', u'logstash-2016.04.20', u'logstash-2016.04.21', u'logstash-2016.04.22', u'logstash-2016.04.23', u'logstash-2016.04.24', u'logstash-2016.04.25', u'logstash-2016.04.26', u'logstash-2016.04.27', u'logstash-2016.04.28', u'logstash-2016.04.29', u'logstash-2016.04.30', u'logstash-2016.05.01', u'logstash-2016.05.02', u'logstash-2016.05.03', u'logstash-2016.05.04', u'logstash-2016.05.05', u'logstash-2016.05.06', u'logstash-2016.05.07', u'logstash-2016.05.08', u'logstash-2016.05.09', u'logstash-2016.05.10', u'logstash-2016.05.11', u'logstash-2016.05.12', u'logstash-2016.05.13', u'logstash-2016.05.14', u'logstash-2016.05.15', u'logstash-2016.05.16', u'logstash-2016.05.17', u'logstash-2016.05.18', u'logstash-2016.05.19', u'logstash-2016.05.20', u'logstash-2016.05.21', u'logstash-2016.05.22', u'logstash-2016.05.23', u'logstash-2016.05.24', u'logstash-2016.05.25', u'logstash-2016.05.26', u'logstash-2016.05.27', u'logstash-2016.05.28', u'logstash-2016.05.29', u'logstash-2016.05.30', u'logstash-2016.05.31', u'logstash-2016.06.01', u'logstash-2016.06.02', u'logstash-2016.06.03', u'logstash-2016.06.04', u'logstash-2016.06.05', u'logstash-2016.06.06', u'logstash-2016.06.07', u'logstash-2016.06.08', u'logstash-2016.06.09', u'logstash-2016.06.10', u'logstash-2016.06.11', u'logstash-2016.06.12', u'logstash-2016.06.13', u'logstash-2016.06.14', u'logstash-2016.06.15', u'logstash-2016.06.16', u'logstash-2016.06.17', u'logstash-2016.06.18', u'logstash-2016.06.19', u'logstash-2016.06.20', u'logstash-2016.06.21', u'logstash-2016.06.22', u'logstash-2016.06.23', u'logstash-2016.06.24', u'logstash-2016.06.25', u'logstash-2016.06.26', u'logstash-2016.06.27', u'logstash-2016.06.28', u'logstash-2016.06.29', u'logstash-2016.06.30', u'logstash-2016.07.01', u'logstash-2016.07.02', u'logstash-2016.07.03', u'logstash-2016.07.04', u'logstash-2016.07.05', u'logstash-2016.07.06', u'logstash-2016.07.07', u'logstash-2016.07.08', u'logstash-2016.07.09', u'logstash-2016.07.10', u'logstash-2016.07.11', u'logstash-2016.07.12', u'logstash-2016.07.13', u'logstash-2016.07.14', u'logstash-2016.07.15', u'logstash-2016.07.16', u'logstash-2016.07.17', u'logstash-2016.07.18', u'logstash-2016.07.19', u'logstash-2016.07.20', u'logstash-2016.07.21', u'logstash-2016.07.22', u'logstash-2016.07.23', u'logstash-2016.07.24', u'logstash-2016.07.25', u'logstash-2016.07.26', u'logstash-2016.07.27', u'logstash-2016.07.28', u'logstash-2016.07.29', u'logstash-2016.07.30', u'logstash-2016.07.31', u'logstash-2016.08.01', u'logstash-2016.08.02', u'logstash-2016.08.03', u'logstash-2016.08.04', u'logstash-2016.08.05', u'logstash-2016.08.06', u'logstash-2016.08.07', u'logstash-2016.08.08', u'logstash-2016.08.09', u'logstash-2016.08.10', u'logstash-2016.08.11', u'logstash-2016.08.12', u'logstash-2016.08.13', u'logstash-2016.08.14', u'logstash-2016.08.15', u'logstash-2016.08.16', u'logstash-2016.08.17', u'logstash-2016.08.18', u'logstash-2016.08.19', u'logstash-2016.08.20', u'logstash-2016.08.21', u'logstash-2016.08.22']
2016-08-23 12:35:20,364 INFO      DRY RUN MODE.  No changes will be made.
2016-08-23 12:35:20,511 INFO      DRY RUN: snapshot: logstash-2015.01.01
2016-08-23 12:35:20,683 INFO      DRY RUN: snapshot: logstash-2015.01.02
2016-08-23 12:35:20,841 INFO      DRY RUN: snapshot: logstash-2015.01.03
2016-08-23 12:35:20,999 INFO      DRY RUN: snapshot: logstash-2015.01.04
2016-08-23 12:35:21,150 INFO      DRY RUN: snapshot: logstash-2015.01.05
2016-08-23 12:35:21,297 INFO      DRY RUN: snapshot: logstash-2015.01.06
2016-08-23 12:35:21,471 INFO      DRY RUN: snapshot: logstash-2015.01.07
2016-08-23 12:35:21,618 INFO      DRY RUN: snapshot: logstash-2015.08.06
2016-08-23 12:35:21,775 INFO      DRY RUN: snapshot: logstash-2015.11.16
2016-08-23 12:35:21,923 INFO      DRY RUN: snapshot: logstash-2015.11.17
2016-08-23 12:35:22,067 INFO      DRY RUN: snapshot: logstash-2015.11.18
2016-08-23 12:35:22,253 INFO      DRY RUN: snapshot: logstash-2015.11.19
2016-08-23 12:35:22,405 INFO      DRY RUN: snapshot: logstash-2015.11.20
2016-08-23 12:35:22,578 INFO      DRY RUN: snapshot: logstash-2015.11.21
2016-08-23 12:35:22,728 INFO      DRY RUN: snapshot: logstash-2015.11.22
2016-08-23 12:35:22,908 INFO      DRY RUN: snapshot: logstash-2015.11.23
2016-08-23 12:35:23,071 INFO      DRY RUN: snapshot: logstash-2015.11.24
2016-08-23 12:35:23,215 INFO      DRY RUN: snapshot: logstash-2015.11.25
2016-08-23 12:35:23,378 INFO      DRY RUN: snapshot: logstash-2015.11.26
2016-08-23 12:35:23,523 INFO      DRY RUN: snapshot: logstash-2015.11.27
2016-08-23 12:35:23,680 INFO      DRY RUN: snapshot: logstash-2015.11.28
2016-08-23 12:35:23,846 INFO      DRY RUN: snapshot: logstash-2015.11.29
2016-08-23 12:35:23,992 INFO      DRY RUN: snapshot: logstash-2015.11.30
2016-08-23 12:35:24,148 INFO      DRY RUN: snapshot: logstash-2015.12.01
2016-08-23 12:35:24,294 INFO      DRY RUN: snapshot: logstash-2015.12.02
2016-08-23 12:35:24,452 INFO      DRY RUN: snapshot: logstash-2015.12.03
2016-08-23 12:35:24,596 INFO      DRY RUN: snapshot: logstash-2015.12.04
2016-08-23 12:35:24,757 INFO      DRY RUN: snapshot: logstash-2015.12.05
2016-08-23 12:35:24,913 INFO      DRY RUN: snapshot: logstash-2015.12.06
2016-08-23 12:35:25,056 INFO      DRY RUN: snapshot: logstash-2015.12.07
2016-08-23 12:35:25,225 INFO      DRY RUN: snapshot: logstash-2015.12.08
2016-08-23 12:35:25,371 INFO      DRY RUN: snapshot: logstash-2015.12.09
2016-08-23 12:35:25,533 INFO      DRY RUN: snapshot: logstash-2015.12.10
2016-08-23 12:35:25,686 INFO      DRY RUN: snapshot: logstash-2015.12.11
2016-08-23 12:35:25,832 INFO      DRY RUN: snapshot: logstash-2015.12.12
2016-08-23 12:35:25,979 INFO      DRY RUN: snapshot: logstash-2015.12.13
2016-08-23 12:35:26,140 INFO      DRY RUN: snapshot: logstash-2015.12.14
2016-08-23 12:35:26,308 INFO      DRY RUN: snapshot: logstash-2015.12.15
2016-08-23 12:35:26,460 INFO      DRY RUN: snapshot: logstash-2015.12.16
2016-08-23 12:35:26,611 INFO      DRY RUN: snapshot: logstash-2015.12.17
2016-08-23 12:35:26,802 INFO      DRY RUN: snapshot: logstash-2015.12.18
2016-08-23 12:35:27,004 INFO      DRY RUN: snapshot: logstash-2015.12.19
2016-08-23 12:35:27,199 INFO      DRY RUN: snapshot: logstash-2015.12.20
2016-08-23 12:35:27,381 INFO      DRY RUN: snapshot: logstash-2015.12.21
2016-08-23 12:35:27,531 INFO      DRY RUN: snapshot: logstash-2015.12.22
2016-08-23 12:35:27,688 INFO      DRY RUN: snapshot: logstash-2015.12.23
2016-08-23 12:35:27,853 INFO      DRY RUN: snapshot: logstash-2015.12.24
2016-08-23 12:35:28,023 INFO      DRY RUN: snapshot: logstash-2015.12.25
2016-08-23 12:35:28,222 INFO      DRY RUN: snapshot: logstash-2015.12.26
2016-08-23 12:35:28,407 INFO      DRY RUN: snapshot: logstash-2015.12.27
2016-08-23 12:35:28,568 INFO      DRY RUN: snapshot: logstash-2015.12.28
2016-08-23 12:35:28,714 INFO      DRY RUN: snapshot: logstash-2015.12.29
2016-08-23 12:35:28,879 INFO      DRY RUN: snapshot: logstash-2015.12.30
2016-08-23 12:35:29,047 INFO      DRY RUN: snapshot: logstash-2015.12.31
2016-08-23 12:35:29,196 INFO      DRY RUN: snapshot: logstash-2016.01.01
2016-08-23 12:35:29,354 INFO      DRY RUN: snapshot: logstash-2016.01.02
2016-08-23 12:35:29,500 INFO      DRY RUN: snapshot: logstash-2016.01.03
2016-08-23 12:35:29,647 INFO      DRY RUN: snapshot: logstash-2016.01.04
2016-08-23 12:35:29,808 INFO      DRY RUN: snapshot: logstash-2016.01.05
2016-08-23 12:35:29,950 INFO      DRY RUN: snapshot: logstash-2016.01.06
2016-08-23 12:35:30,104 INFO      DRY RUN: snapshot: logstash-2016.01.07
2016-08-23 12:35:30,248 INFO      DRY RUN: snapshot: logstash-2016.01.08
2016-08-23 12:35:30,392 INFO      DRY RUN: snapshot: logstash-2016.01.09
2016-08-23 12:35:30,554 INFO      DRY RUN: snapshot: logstash-2016.01.10
2016-08-23 12:35:30,697 INFO      DRY RUN: snapshot: logstash-2016.01.11
2016-08-23 12:35:30,856 INFO      DRY RUN: snapshot: logstash-2016.01.12
2016-08-23 12:35:31,002 INFO      DRY RUN: snapshot: logstash-2016.01.13
2016-08-23 12:35:31,149 INFO      DRY RUN: snapshot: logstash-2016.01.14
2016-08-23 12:35:31,339 INFO      DRY RUN: snapshot: logstash-2016.01.15
2016-08-23 12:35:31,484 INFO      DRY RUN: snapshot: logstash-2016.01.16
2016-08-23 12:35:31,640 INFO      DRY RUN: snapshot: logstash-2016.01.17
2016-08-23 12:35:31,783 INFO      DRY RUN: snapshot: logstash-2016.01.18
2016-08-23 12:35:31,934 INFO      DRY RUN: snapshot: logstash-2016.01.19
2016-08-23 12:35:32,084 INFO      DRY RUN: snapshot: logstash-2016.01.20
2016-08-23 12:35:32,248 INFO      DRY RUN: snapshot: logstash-2016.01.21
2016-08-23 12:35:32,429 INFO      DRY RUN: snapshot: logstash-2016.01.22
2016-08-23 12:35:32,591 INFO      DRY RUN: snapshot: logstash-2016.01.23
2016-08-23 12:35:32,733 INFO      DRY RUN: snapshot: logstash-2016.01.24
2016-08-23 12:35:32,892 INFO      DRY RUN: snapshot: logstash-2016.01.25
2016-08-23 12:35:33,050 INFO      DRY RUN: snapshot: logstash-2016.01.26
2016-08-23 12:35:33,208 INFO      DRY RUN: snapshot: logstash-2016.01.27
2016-08-23 12:35:33,352 INFO      DRY RUN: snapshot: logstash-2016.01.28
2016-08-23 12:35:33,499 INFO      DRY RUN: snapshot: logstash-2016.01.29
2016-08-23 12:35:33,656 INFO      DRY RUN: snapshot: logstash-2016.01.30
2016-08-23 12:35:33,817 INFO      DRY RUN: snapshot: logstash-2016.01.31
2016-08-23 12:35:33,963 INFO      DRY RUN: snapshot: logstash-2016.02.01
2016-08-23 12:35:34,121 INFO      DRY RUN: snapshot: logstash-2016.02.02
2016-08-23 12:35:34,271 INFO      DRY RUN: snapshot: logstash-2016.02.03
2016-08-23 12:35:34,438 INFO      DRY RUN: snapshot: logstash-2016.02.04
2016-08-23 12:35:34,593 INFO      DRY RUN: snapshot: logstash-2016.02.05
2016-08-23 12:35:34,738 INFO      DRY RUN: snapshot: logstash-2016.02.06
2016-08-23 12:35:34,900 INFO      DRY RUN: snapshot: logstash-2016.02.07
2016-08-23 12:35:35,048 INFO      DRY RUN: snapshot: logstash-2016.02.08
2016-08-23 12:35:35,204 INFO      DRY RUN: snapshot: logstash-2016.02.09
2016-08-23 12:35:35,350 INFO      DRY RUN: snapshot: logstash-2016.02.10
2016-08-23 12:35:35,513 INFO      DRY RUN: snapshot: logstash-2016.02.11
2016-08-23 12:35:35,673 INFO      DRY RUN: snapshot: logstash-2016.02.12
2016-08-23 12:35:35,818 INFO      DRY RUN: snapshot: logstash-2016.02.13
2016-08-23 12:35:35,978 INFO      DRY RUN: snapshot: logstash-2016.02.14
2016-08-23 12:35:36,123 INFO      DRY RUN: snapshot: logstash-2016.02.15
2016-08-23 12:35:36,285 INFO      DRY RUN: snapshot: logstash-2016.02.16
2016-08-23 12:35:36,441 INFO      DRY RUN: snapshot: logstash-2016.02.17
2016-08-23 12:35:36,584 INFO      DRY RUN: snapshot: logstash-2016.02.18
2016-08-23 12:35:36,748 INFO      DRY RUN: snapshot: logstash-2016.02.19
2016-08-23 12:35:36,926 INFO      DRY RUN: snapshot: logstash-2016.02.20
2016-08-23 12:35:37,125 INFO      DRY RUN: snapshot: logstash-2016.02.21
2016-08-23 12:35:37,349 INFO      DRY RUN: snapshot: logstash-2016.02.22
2016-08-23 12:35:37,529 INFO      DRY RUN: snapshot: logstash-2016.02.23
2016-08-23 12:35:37,690 INFO      DRY RUN: snapshot: logstash-2016.02.24
2016-08-23 12:35:37,835 INFO      DRY RUN: snapshot: logstash-2016.02.25
2016-08-23 12:35:38,003 INFO      DRY RUN: snapshot: logstash-2016.02.26
2016-08-23 12:35:38,194 INFO      DRY RUN: snapshot: logstash-2016.02.27
2016-08-23 12:35:38,370 INFO      DRY RUN: snapshot: logstash-2016.02.28
2016-08-23 12:35:38,553 INFO      DRY RUN: snapshot: logstash-2016.02.29
2016-08-23 12:35:38,707 INFO      DRY RUN: snapshot: logstash-2016.03.01
2016-08-23 12:35:38,859 INFO      DRY RUN: snapshot: logstash-2016.03.02
2016-08-23 12:35:39,021 INFO      DRY RUN: snapshot: logstash-2016.03.03
2016-08-23 12:35:39,168 INFO      DRY RUN: snapshot: logstash-2016.03.04
2016-08-23 12:35:39,328 INFO      DRY RUN: snapshot: logstash-2016.03.05
2016-08-23 12:35:39,475 INFO      DRY RUN: snapshot: logstash-2016.03.06
2016-08-23 12:35:39,631 INFO      DRY RUN: snapshot: logstash-2016.03.07
2016-08-23 12:35:39,792 INFO      DRY RUN: snapshot: logstash-2016.03.08
2016-08-23 12:35:39,934 INFO      DRY RUN: snapshot: logstash-2016.03.09
2016-08-23 12:35:40,094 INFO      DRY RUN: snapshot: logstash-2016.03.10
2016-08-23 12:35:40,256 INFO      DRY RUN: snapshot: logstash-2016.03.11
2016-08-23 12:35:40,415 INFO      DRY RUN: snapshot: logstash-2016.03.12
2016-08-23 12:35:40,574 INFO      DRY RUN: snapshot: logstash-2016.03.13
2016-08-23 12:35:40,715 INFO      DRY RUN: snapshot: logstash-2016.03.14
2016-08-23 12:35:40,867 INFO      DRY RUN: snapshot: logstash-2016.03.15
2016-08-23 12:35:41,014 INFO      DRY RUN: snapshot: logstash-2016.03.16
2016-08-23 12:35:41,177 INFO      DRY RUN: snapshot: logstash-2016.03.17
2016-08-23 12:35:41,322 INFO      DRY RUN: snapshot: logstash-2016.03.18
2016-08-23 12:35:41,479 INFO      DRY RUN: snapshot: logstash-2016.03.19
2016-08-23 12:35:41,630 INFO      DRY RUN: snapshot: logstash-2016.03.20
2016-08-23 12:35:41,777 INFO      DRY RUN: snapshot: logstash-2016.03.21
2016-08-23 12:35:41,921 INFO      DRY RUN: snapshot: logstash-2016.03.22
2016-08-23 12:35:42,088 INFO      DRY RUN: snapshot: logstash-2016.03.23
2016-08-23 12:35:42,251 INFO      DRY RUN: snapshot: logstash-2016.03.24
2016-08-23 12:35:42,404 INFO      DRY RUN: snapshot: logstash-2016.03.25
2016-08-23 12:35:42,554 INFO      DRY RUN: snapshot: logstash-2016.03.26
2016-08-23 12:35:42,697 INFO      DRY RUN: snapshot: logstash-2016.03.27
2016-08-23 12:35:42,859 INFO      DRY RUN: snapshot: logstash-2016.03.28
2016-08-23 12:35:43,034 INFO      DRY RUN: snapshot: logstash-2016.03.29
2016-08-23 12:35:43,192 INFO      DRY RUN: snapshot: logstash-2016.03.30
2016-08-23 12:35:43,340 INFO      DRY RUN: snapshot: logstash-2016.03.31
2016-08-23 12:35:43,508 INFO      DRY RUN: snapshot: logstash-2016.04.01
2016-08-23 12:35:43,662 INFO      DRY RUN: snapshot: logstash-2016.04.02
2016-08-23 12:35:43,812 INFO      DRY RUN: snapshot: logstash-2016.04.03
2016-08-23 12:35:43,974 INFO      DRY RUN: snapshot: logstash-2016.04.04
2016-08-23 12:35:44,121 INFO      DRY RUN: snapshot: logstash-2016.04.05
2016-08-23 12:35:44,266 INFO      DRY RUN: snapshot: logstash-2016.04.06
2016-08-23 12:35:44,446 INFO      DRY RUN: snapshot: logstash-2016.04.07
2016-08-23 12:35:44,593 INFO      DRY RUN: snapshot: logstash-2016.04.08
2016-08-23 12:35:44,760 INFO      DRY RUN: snapshot: logstash-2016.04.09
2016-08-23 12:35:44,907 INFO      DRY RUN: snapshot: logstash-2016.04.10
2016-08-23 12:35:45,053 INFO      DRY RUN: snapshot: logstash-2016.04.11
2016-08-23 12:35:45,217 INFO      DRY RUN: snapshot: logstash-2016.04.12
2016-08-23 12:35:45,359 INFO      DRY RUN: snapshot: logstash-2016.04.13
2016-08-23 12:35:45,528 INFO      DRY RUN: snapshot: logstash-2016.04.14
2016-08-23 12:35:45,677 INFO      DRY RUN: snapshot: logstash-2016.04.15
2016-08-23 12:35:45,825 INFO      DRY RUN: snapshot: logstash-2016.04.16
2016-08-23 12:35:45,980 INFO      DRY RUN: snapshot: logstash-2016.04.17
2016-08-23 12:35:46,128 INFO      DRY RUN: snapshot: logstash-2016.04.18
2016-08-23 12:35:46,287 INFO      DRY RUN: snapshot: logstash-2016.04.19
2016-08-23 12:35:46,453 INFO      DRY RUN: snapshot: logstash-2016.04.20
2016-08-23 12:35:46,597 INFO      DRY RUN: snapshot: logstash-2016.04.21
2016-08-23 12:35:46,764 INFO      DRY RUN: snapshot: logstash-2016.04.22
2016-08-23 12:35:46,943 INFO      DRY RUN: snapshot: logstash-2016.04.23
2016-08-23 12:35:47,153 INFO      DRY RUN: snapshot: logstash-2016.04.24
2016-08-23 12:35:47,321 INFO      DRY RUN: snapshot: logstash-2016.04.25
2016-08-23 12:35:47,518 INFO      DRY RUN: snapshot: logstash-2016.04.26
2016-08-23 12:35:47,709 INFO      DRY RUN: snapshot: logstash-2016.04.27
2016-08-23 12:35:47,876 INFO      DRY RUN: snapshot: logstash-2016.04.28
2016-08-23 12:35:48,047 INFO      DRY RUN: snapshot: logstash-2016.04.29
2016-08-23 12:35:48,239 INFO      DRY RUN: snapshot: logstash-2016.04.30
2016-08-23 12:35:48,418 INFO      DRY RUN: snapshot: logstash-2016.05.01
2016-08-23 12:35:48,610 INFO      DRY RUN: snapshot: logstash-2016.05.02
2016-08-23 12:35:48,764 INFO      DRY RUN: snapshot: logstash-2016.05.03
2016-08-23 12:35:48,925 INFO      DRY RUN: snapshot: logstash-2016.05.04
2016-08-23 12:35:49,073 INFO      DRY RUN: snapshot: logstash-2016.05.05
2016-08-23 12:35:49,235 INFO      DRY RUN: snapshot: logstash-2016.05.06
2016-08-23 12:35:49,379 INFO      DRY RUN: snapshot: logstash-2016.05.07
2016-08-23 12:35:49,528 INFO      DRY RUN: snapshot: logstash-2016.05.08
2016-08-23 12:35:49,683 INFO      DRY RUN: snapshot: logstash-2016.05.09
2016-08-23 12:35:49,844 INFO      DRY RUN: snapshot: logstash-2016.05.10
2016-08-23 12:35:50,002 INFO      DRY RUN: snapshot: logstash-2016.05.11
2016-08-23 12:35:50,146 INFO      DRY RUN: snapshot: logstash-2016.05.12
2016-08-23 12:35:50,290 INFO      DRY RUN: snapshot: logstash-2016.05.13
2016-08-23 12:35:50,449 INFO      DRY RUN: snapshot: logstash-2016.05.14
2016-08-23 12:35:50,608 INFO      DRY RUN: snapshot: logstash-2016.05.15
2016-08-23 12:35:50,754 INFO      DRY RUN: snapshot: logstash-2016.05.16
2016-08-23 12:35:50,919 INFO      DRY RUN: snapshot: logstash-2016.05.17
2016-08-23 12:35:51,059 INFO      DRY RUN: snapshot: logstash-2016.05.18
2016-08-23 12:35:51,221 INFO      DRY RUN: snapshot: logstash-2016.05.19
2016-08-23 12:35:51,384 INFO      DRY RUN: snapshot: logstash-2016.05.20
2016-08-23 12:35:51,533 INFO      DRY RUN: snapshot: logstash-2016.05.21
2016-08-23 12:35:51,692 INFO      DRY RUN: snapshot: logstash-2016.05.22
2016-08-23 12:35:51,837 INFO      DRY RUN: snapshot: logstash-2016.05.23
2016-08-23 12:35:51,992 INFO      DRY RUN: snapshot: logstash-2016.05.24
2016-08-23 12:35:52,152 INFO      DRY RUN: snapshot: logstash-2016.05.25
2016-08-23 12:35:52,303 INFO      DRY RUN: snapshot: logstash-2016.05.26
2016-08-23 12:35:52,460 INFO      DRY RUN: snapshot: logstash-2016.05.27
2016-08-23 12:35:52,614 INFO      DRY RUN: snapshot: logstash-2016.05.28
2016-08-23 12:35:52,783 INFO      DRY RUN: snapshot: logstash-2016.05.29
2016-08-23 12:35:52,995 INFO      DRY RUN: snapshot: logstash-2016.05.30
2016-08-23 12:35:53,148 INFO      DRY RUN: snapshot: logstash-2016.05.31
2016-08-23 12:35:53,297 INFO      DRY RUN: snapshot: logstash-2016.06.01
2016-08-23 12:35:53,462 INFO      DRY RUN: snapshot: logstash-2016.06.02
2016-08-23 12:35:53,610 INFO      DRY RUN: snapshot: logstash-2016.06.03
2016-08-23 12:35:53,767 INFO      DRY RUN: snapshot: logstash-2016.06.04
2016-08-23 12:35:53,925 INFO      DRY RUN: snapshot: logstash-2016.06.05
2016-08-23 12:35:54,076 INFO      DRY RUN: snapshot: logstash-2016.06.06
2016-08-23 12:35:54,236 INFO      DRY RUN: snapshot: logstash-2016.06.07
2016-08-23 12:35:54,386 INFO      DRY RUN: snapshot: logstash-2016.06.08
2016-08-23 12:35:54,545 INFO      DRY RUN: snapshot: logstash-2016.06.09
2016-08-23 12:35:54,707 INFO      DRY RUN: snapshot: logstash-2016.06.10
2016-08-23 12:35:54,854 INFO      DRY RUN: snapshot: logstash-2016.06.11
2016-08-23 12:35:54,997 INFO      DRY RUN: snapshot: logstash-2016.06.12
2016-08-23 12:35:55,150 INFO      DRY RUN: snapshot: logstash-2016.06.13
2016-08-23 12:35:55,308 INFO      DRY RUN: snapshot: logstash-2016.06.14
2016-08-23 12:35:55,465 INFO      DRY RUN: snapshot: logstash-2016.06.15
2016-08-23 12:35:55,612 INFO      DRY RUN: snapshot: logstash-2016.06.16
2016-08-23 12:35:55,759 INFO      DRY RUN: snapshot: logstash-2016.06.17
2016-08-23 12:35:55,917 INFO      DRY RUN: snapshot: logstash-2016.06.18
2016-08-23 12:35:56,063 INFO      DRY RUN: snapshot: logstash-2016.06.19
2016-08-23 12:35:56,233 INFO      DRY RUN: snapshot: logstash-2016.06.20
2016-08-23 12:35:56,386 INFO      DRY RUN: snapshot: logstash-2016.06.21
2016-08-23 12:35:56,531 INFO      DRY RUN: snapshot: logstash-2016.06.22
2016-08-23 12:35:56,688 INFO      DRY RUN: snapshot: logstash-2016.06.23
2016-08-23 12:35:56,843 INFO      DRY RUN: snapshot: logstash-2016.06.24
2016-08-23 12:35:57,033 INFO      DRY RUN: snapshot: logstash-2016.06.25
2016-08-23 12:35:57,247 INFO      DRY RUN: snapshot: logstash-2016.06.26
2016-08-23 12:35:57,440 INFO      DRY RUN: snapshot: logstash-2016.06.27
2016-08-23 12:35:57,632 INFO      DRY RUN: snapshot: logstash-2016.06.28
2016-08-23 12:35:57,822 INFO      DRY RUN: snapshot: logstash-2016.06.29
2016-08-23 12:35:57,984 INFO      DRY RUN: snapshot: logstash-2016.06.30
2016-08-23 12:35:58,138 INFO      DRY RUN: snapshot: logstash-2016.07.01
2016-08-23 12:35:58,311 INFO      DRY RUN: snapshot: logstash-2016.07.02
2016-08-23 12:35:58,487 INFO      DRY RUN: snapshot: logstash-2016.07.03
2016-08-23 12:35:58,679 INFO      DRY RUN: snapshot: logstash-2016.07.04
2016-08-23 12:35:58,856 INFO      DRY RUN: snapshot: logstash-2016.07.05
2016-08-23 12:35:59,009 INFO      DRY RUN: snapshot: logstash-2016.07.06
2016-08-23 12:35:59,160 INFO      DRY RUN: snapshot: logstash-2016.07.07
2016-08-23 12:35:59,302 INFO      DRY RUN: snapshot: logstash-2016.07.08
2016-08-23 12:35:59,451 INFO      DRY RUN: snapshot: logstash-2016.07.09
2016-08-23 12:35:59,639 INFO      DRY RUN: snapshot: logstash-2016.07.10
2016-08-23 12:35:59,784 INFO      DRY RUN: snapshot: logstash-2016.07.11
2016-08-23 12:35:59,948 INFO      DRY RUN: snapshot: logstash-2016.07.12
2016-08-23 12:36:00,102 INFO      DRY RUN: snapshot: logstash-2016.07.13
2016-08-23 12:36:00,248 INFO      DRY RUN: snapshot: logstash-2016.07.14
2016-08-23 12:36:00,414 INFO      DRY RUN: snapshot: logstash-2016.07.15
2016-08-23 12:36:00,570 INFO      DRY RUN: snapshot: logstash-2016.07.16
2016-08-23 12:36:00,725 INFO      DRY RUN: snapshot: logstash-2016.07.17
2016-08-23 12:36:00,872 INFO      DRY RUN: snapshot: logstash-2016.07.18
2016-08-23 12:36:01,018 INFO      DRY RUN: snapshot: logstash-2016.07.19
2016-08-23 12:36:01,174 INFO      DRY RUN: snapshot: logstash-2016.07.20
2016-08-23 12:36:01,347 INFO      DRY RUN: snapshot: logstash-2016.07.21
2016-08-23 12:36:01,536 INFO      DRY RUN: snapshot: logstash-2016.07.22
2016-08-23 12:36:01,697 INFO      DRY RUN: snapshot: logstash-2016.07.23
2016-08-23 12:36:01,850 INFO      DRY RUN: snapshot: logstash-2016.07.24
2016-08-23 12:36:02,015 INFO      DRY RUN: snapshot: logstash-2016.07.25
2016-08-23 12:36:02,173 INFO      DRY RUN: snapshot: logstash-2016.07.26
2016-08-23 12:36:02,326 INFO      DRY RUN: snapshot: logstash-2016.07.27
2016-08-23 12:36:02,472 INFO      DRY RUN: snapshot: logstash-2016.07.28
2016-08-23 12:36:02,616 INFO      DRY RUN: snapshot: logstash-2016.07.29
2016-08-23 12:36:02,773 INFO      DRY RUN: snapshot: logstash-2016.07.30
2016-08-23 12:36:02,933 INFO      DRY RUN: snapshot: logstash-2016.07.31
2016-08-23 12:36:03,086 INFO      DRY RUN: snapshot: logstash-2016.08.01
2016-08-23 12:36:03,232 INFO      DRY RUN: snapshot: logstash-2016.08.02
2016-08-23 12:36:03,376 INFO      DRY RUN: snapshot: logstash-2016.08.03
2016-08-23 12:36:03,541 INFO      DRY RUN: snapshot: logstash-2016.08.04
2016-08-23 12:36:03,687 INFO      DRY RUN: snapshot: logstash-2016.08.05
2016-08-23 12:36:03,859 INFO      DRY RUN: snapshot: logstash-2016.08.06
2016-08-23 12:36:04,021 INFO      DRY RUN: snapshot: logstash-2016.08.07
2016-08-23 12:36:04,163 INFO      DRY RUN: snapshot: logstash-2016.08.08
2016-08-23 12:36:04,325 INFO      DRY RUN: snapshot: logstash-2016.08.09
2016-08-23 12:36:04,469 INFO      DRY RUN: snapshot: logstash-2016.08.10
2016-08-23 12:36:04,633 INFO      DRY RUN: snapshot: logstash-2016.08.11
2016-08-23 12:36:04,792 INFO      DRY RUN: snapshot: logstash-2016.08.12
2016-08-23 12:36:04,937 INFO      DRY RUN: snapshot: logstash-2016.08.13
2016-08-23 12:36:05,096 INFO      DRY RUN: snapshot: logstash-2016.08.14
2016-08-23 12:36:05,241 INFO      DRY RUN: snapshot: logstash-2016.08.15
2016-08-23 12:36:05,403 INFO      DRY RUN: snapshot: logstash-2016.08.16
2016-08-23 12:36:05,560 INFO      DRY RUN: snapshot: logstash-2016.08.17
2016-08-23 12:36:06,066 INFO      DRY RUN: snapshot: logstash-2016.08.18
2016-08-23 12:36:06,224 INFO      DRY RUN: snapshot: logstash-2016.08.19
2016-08-23 12:36:06,370 INFO      DRY RUN: snapshot: logstash-2016.08.20
2016-08-23 12:36:06,516 INFO      DRY RUN: snapshot: logstash-2016.08.21
2016-08-23 12:36:06,683 INFO      DRY RUN: snapshot: logstash-2016.08.22
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Log Server does not create backups

Post by mcapra »

Currently, it looks as if you have 289 indexes open. That is an awful lot of indexes to have open. Setting the "close indexes older than" value to 0 is particularly dangerous since every open index is stored in RAM. This essentially means your machine is trying to keep all 289 indexes in RAM. I'm guessing we're going to need to close many of those indexes.

Can you share the output of top?

Can you share some of your index sizes? You should be able to view them from the (Administration -> Index Status) page.

How much RAM is allocated to this machine?
Former Nagios employee
https://www.mcapra.com/
Top
vmesquita
Posts: 315
Joined: Fri Aug 10, 2012 12:52 pm

Re: Log Server does not create backups

Post by vmesquita »

top output:

Code: Select all

top - 12:59:01 up 5 days, 23:18,  1 user,  load average: 0.36, 0.16, 0.12
Tasks: 176 total,   1 running, 175 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.7%us,  0.6%sy,  2.4%ni, 95.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8043104k total,  7812128k used,   230976k free,   187584k buffers
Swap:  4194300k total,   147676k used,  4046624k free,  2017220k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4602 nagios    39  19 3344m 542m  11m S 13.0  6.9 693:08.87 java
 1991 nagios    20   0 52.3g 4.4g 142m S  9.0 57.6   1295:10 java
 1278 root      20   0 15032 1272  920 R  0.3  0.0   0:00.01 top
    1 root      20   0 23500 1264 1096 S  0.0  0.0   0:00.93 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.75 migration/0
    4 root      20   0     0    0    0 S  0.0  0.0   0:06.87 ksoftirqd/0
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.67 watchdog/0
    7 root      RT   0     0    0    0 S  0.0  0.0   0:01.24 migration/1
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/1
    9 root      20   0     0    0    0 S  0.0  0.0   0:06.35 ksoftirqd/1
   10 root      RT   0     0    0    0 S  0.0  0.0   0:00.34 watchdog/1
   11 root      RT   0     0    0    0 S  0.0  0.0   0:01.36 migration/2
   12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/2
   13 root      20   0     0    0    0 S  0.0  0.0   0:06.07 ksoftirqd/2
   14 root      RT   0     0    0    0 S  0.0  0.0   0:00.36 watchdog/2
   15 root      RT   0     0    0    0 S  0.0  0.0   0:00.83 migration/3
   16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/3
   17 root      20   0     0    0    0 S  0.0  0.0   0:02.79 ksoftirqd/3
   18 root      RT   0     0    0    0 S  0.0  0.0   0:00.36 watchdog/3
   19 root      20   0     0    0    0 S  0.0  0.0   0:20.59 events/0
   20 root      20   0     0    0    0 S  0.0  0.0   0:16.87 events/1
   21 root      20   0     0    0    0 S  0.0  0.0   0:18.94 events/2
   22 root      20   0     0    0    0 S  0.0  0.0   0:27.12 events/3
   23 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events/0
   24 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events/1
   25 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events/2
   26 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events/3
   27 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events_long/0
   28 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events_long/1
   29 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events_long/2
   30 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events_long/3
   31 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events_power_ef
   32 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events_power_ef
   33 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events_power_ef
   34 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events_power_ef
   35 root      20   0     0    0    0 S  0.0  0.0   0:00.00 cgroup
   36 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khelper
   37 root      20   0     0    0    0 S  0.0  0.0   0:00.00 netns
Index sizes:

Code: Select all

      logstash-2016.08.23 	3,163,638	857.3MB	5	1	 close   delete
	logstash-2016.08.22	5,292,377	1.3GB	5	1	 close   delete
	logstash-2016.08.21	10,808,076	2.4GB	5	1	 close   delete
	logstash-2016.08.20	4,432,048	1GB	5	1	 close   delete
	logstash-2016.08.19	4,636,484	1.1GB	5	1	 close   delete
	logstash-2016.08.18	1,500,306	428.1MB	5	1
The server has 8 Gb.

I actually never changed the default values and we have been implementing this server for a while now. I'll change the "close indexes older than" parameter to 60 days,
Last edited by dwhitfield on Fri Oct 07, 2016 12:08 pm, edited 3 times in total.
Reason: still cleaning up a code block...possibly good enough now.
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Log Server does not create backups

Post by mcapra »

The whole 0 defaults thing has been tossed around a lot internally. On the one hand, we don't want people's use of the software to be limited by default settings. On the other, it allows users to create potentially damaging environments.

With 8GB available on this machine, I would actually recommend keeping no more than 7 indexes open at a time or increasing the RAM this server has available. 60 is probably still way too much with an average index size around 800MB.

If you need to search through older data, you can always open additional indexes via the (Administration -> Index Status) page.
Former Nagios employee
https://www.mcapra.com/
Top
vmesquita
Posts: 315
Joined: Fri Aug 10, 2012 12:52 pm

Re: Log Server does not create backups

Post by vmesquita »

Ok, I'll change it to 7. Should I manually close the indexes already older than 7 days?
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Log Server does not create backups

Post by mcapra »

The command subsystem should get around to doing that eventually, but just in case it doesn't this should save you a lot of time over doing it by hand:

Code: Select all

curator close indices --older-than 7 --time-unit days --timestring %Y.%m.%d
service elasticsearch restart
Former Nagios employee
https://www.mcapra.com/
Top
vmesquita
Posts: 315
Joined: Fri Aug 10, 2012 12:52 pm

Re: Log Server does not create backups

Post by vmesquita »

I tried the command suggested, but I got this error:

Code: Select all

# curator close indices --older-than 7 --time-unit days --timestring %Y.%m.%d
2016-08-23 16:40:23,265 INFO      Job starting: close indices
2016-08-23 16:40:23,324 INFO      Action close will be performed on the following indices: [u'logstash-2015.01.01', u'logstash-2015.01.02', u'logstash-2015.01.03', u'logstash-2015.01.04', u'logstash-2015.01.05', u'logstash-2015.01.06', u'logstash-2015.01.07', u'logstash-2015.08.06', u'logstash-2015.11.16', u'logstash-2015.11.17', u'logstash-2015.11.18', u'logstash-2015.11.19', u'logstash-2015.11.20', u'logstash-2015.11.21', u'logstash-2015.11.22', u'logstash-2015.11.23', u'logstash-2015.11.24', u'logstash-2015.11.25', u'logstash-2015.11.26', u'logstash-2015.11.27', u'logstash-2015.11.28', u'logstash-2015.11.29', u'logstash-2015.11.30', u'logstash-2015.12.01', u'logstash-2015.12.02', u'logstash-2015.12.03', u'logstash-2015.12.04', u'logstash-2015.12.05', u'logstash-2015.12.06', u'logstash-2015.12.07', u'logstash-2015.12.08', u'logstash-2015.12.09', u'logstash-2015.12.10', u'logstash-2015.12.11', u'logstash-2015.12.12', u'logstash-2015.12.13', u'logstash-2015.12.14', u'logstash-2015.12.15', u'logstash-2015.12.16', u'logstash-2015.12.17', u'logstash-2015.12.18', u'logstash-2015.12.19', u'logstash-2015.12.20', u'logstash-2015.12.21', u'logstash-2015.12.22', u'logstash-2015.12.23', u'logstash-2015.12.24', u'logstash-2015.12.25', u'logstash-2015.12.26', u'logstash-2015.12.27', u'logstash-2015.12.28', u'logstash-2015.12.29', u'logstash-2015.12.30', u'logstash-2015.12.31', u'logstash-2016.01.01', u'logstash-2016.01.02', u'logstash-2016.01.03', u'logstash-2016.01.04', u'logstash-2016.01.05', u'logstash-2016.01.06', u'logstash-2016.01.07', u'logstash-2016.01.08', u'logstash-2016.01.09', u'logstash-2016.01.10', u'logstash-2016.01.11', u'logstash-2016.01.12', u'logstash-2016.01.13', u'logstash-2016.01.14', u'logstash-2016.01.15', u'logstash-2016.01.16', u'logstash-2016.01.17', u'logstash-2016.01.18', u'logstash-2016.01.19', u'logstash-2016.01.20', u'logstash-2016.01.21', u'logstash-2016.01.22', u'logstash-2016.01.23', u'logstash-2016.01.24', u'logstash-2016.01.25', u'logstash-2016.01.26', u'logstash-2016.01.27', u'logstash-2016.01.28', u'logstash-2016.01.29', u'logstash-2016.01.30', u'logstash-2016.01.31', u'logstash-2016.02.01', u'logstash-2016.02.02', u'logstash-2016.02.03', u'logstash-2016.02.04', u'logstash-2016.02.05', u'logstash-2016.02.06', u'logstash-2016.02.07', u'logstash-2016.02.08', u'logstash-2016.02.09', u'logstash-2016.02.10', u'logstash-2016.02.11', u'logstash-2016.02.12', u'logstash-2016.02.13', u'logstash-2016.02.14', u'logstash-2016.02.15', u'logstash-2016.02.16', u'logstash-2016.02.17', u'logstash-2016.02.18', u'logstash-2016.02.19', u'logstash-2016.02.20', u'logstash-2016.02.21', u'logstash-2016.02.22', u'logstash-2016.02.23', u'logstash-2016.02.24', u'logstash-2016.02.25', u'logstash-2016.02.26', u'logstash-2016.02.27', u'logstash-2016.02.28', u'logstash-2016.02.29', u'logstash-2016.03.01', u'logstash-2016.03.02', u'logstash-2016.03.03', u'logstash-2016.03.04', u'logstash-2016.03.05', u'logstash-2016.03.06', u'logstash-2016.03.07', u'logstash-2016.03.08', u'logstash-2016.03.09', u'logstash-2016.03.10', u'logstash-2016.03.11', u'logstash-2016.03.12', u'logstash-2016.03.13', u'logstash-2016.03.14', u'logstash-2016.03.15', u'logstash-2016.03.16', u'logstash-2016.03.17', u'logstash-2016.03.18', u'logstash-2016.03.19', u'logstash-2016.03.20', u'logstash-2016.03.21', u'logstash-2016.03.22', u'logstash-2016.03.23', u'logstash-2016.03.24', u'logstash-2016.03.25', u'logstash-2016.03.26', u'logstash-2016.03.27', u'logstash-2016.03.28', u'logstash-2016.03.29', u'logstash-2016.03.30', u'logstash-2016.03.31', u'logstash-2016.04.01', u'logstash-2016.04.02', u'logstash-2016.04.03', u'logstash-2016.04.04', u'logstash-2016.04.05', u'logstash-2016.04.06', u'logstash-2016.04.07', u'logstash-2016.04.08', u'logstash-2016.04.09', u'logstash-2016.04.10', u'logstash-2016.04.11', u'logstash-2016.04.12', u'logstash-2016.04.13', u'logstash-2016.04.14', u'logstash-2016.04.15', u'logstash-2016.04.16', u'logstash-2016.04.17', u'logstash-2016.04.18', u'logstash-2016.04.19', u'logstash-2016.04.20', u'logstash-2016.04.21', u'logstash-2016.04.22', u'logstash-2016.04.23', u'logstash-2016.04.24', u'logstash-2016.04.25', u'logstash-2016.04.26', u'logstash-2016.04.27', u'logstash-2016.04.28', u'logstash-2016.04.29', u'logstash-2016.04.30', u'logstash-2016.05.01', u'logstash-2016.05.02', u'logstash-2016.05.03', u'logstash-2016.05.04', u'logstash-2016.05.05', u'logstash-2016.05.06', u'logstash-2016.05.07', u'logstash-2016.05.08', u'logstash-2016.05.09', u'logstash-2016.05.10', u'logstash-2016.05.11', u'logstash-2016.05.12', u'logstash-2016.05.13', u'logstash-2016.05.14', u'logstash-2016.05.15', u'logstash-2016.05.16', u'logstash-2016.05.17', u'logstash-2016.05.18', u'logstash-2016.05.19', u'logstash-2016.05.20', u'logstash-2016.05.21', u'logstash-2016.05.22', u'logstash-2016.05.23', u'logstash-2016.05.24', u'logstash-2016.05.25', u'logstash-2016.05.26', u'logstash-2016.05.27', u'logstash-2016.05.28', u'logstash-2016.05.29', u'logstash-2016.05.30', u'logstash-2016.05.31', u'logstash-2016.06.01', u'logstash-2016.06.02', u'logstash-2016.06.03', u'logstash-2016.06.04', u'logstash-2016.06.05', u'logstash-2016.06.06', u'logstash-2016.06.07', u'logstash-2016.06.08', u'logstash-2016.06.09', u'logstash-2016.06.10', u'logstash-2016.06.11', u'logstash-2016.06.12', u'logstash-2016.06.13', u'logstash-2016.06.14', u'logstash-2016.06.15', u'logstash-2016.06.16', u'logstash-2016.06.17', u'logstash-2016.06.18', u'logstash-2016.06.19', u'logstash-2016.06.20', u'logstash-2016.06.21', u'logstash-2016.06.22', u'logstash-2016.06.23', u'logstash-2016.06.24', u'logstash-2016.06.25', u'logstash-2016.06.26', u'logstash-2016.06.27', u'logstash-2016.06.28', u'logstash-2016.06.29', u'logstash-2016.06.30', u'logstash-2016.07.01', u'logstash-2016.07.02', u'logstash-2016.07.03', u'logstash-2016.07.04', u'logstash-2016.07.05', u'logstash-2016.07.06', u'logstash-2016.07.07', u'logstash-2016.07.08', u'logstash-2016.07.09', u'logstash-2016.07.10', u'logstash-2016.07.11', u'logstash-2016.07.12', u'logstash-2016.07.13', u'logstash-2016.07.14', u'logstash-2016.07.15', u'logstash-2016.07.16', u'logstash-2016.07.17', u'logstash-2016.07.18', u'logstash-2016.07.19', u'logstash-2016.07.20', u'logstash-2016.07.21', u'logstash-2016.07.22', u'logstash-2016.07.23', u'logstash-2016.07.24', u'logstash-2016.07.25', u'logstash-2016.07.26', u'logstash-2016.07.27', u'logstash-2016.07.28', u'logstash-2016.07.29', u'logstash-2016.07.30', u'logstash-2016.07.31', u'logstash-2016.08.01', u'logstash-2016.08.02', u'logstash-2016.08.03', u'logstash-2016.08.04', u'logstash-2016.08.05', u'logstash-2016.08.06', u'logstash-2016.08.07', u'logstash-2016.08.08', u'logstash-2016.08.09', u'logstash-2016.08.10', u'logstash-2016.08.11', u'logstash-2016.08.12', u'logstash-2016.08.13', u'logstash-2016.08.14', u'logstash-2016.08.15', u'logstash-2016.08.16']
2016-08-23 16:40:23,325 WARNING   Very large list of indices.  Breaking it up into smaller chunks.
2016-08-23 16:41:17,958 WARNING   Non-fatal error encountered.
Traceback (most recent call last):
  File "/usr/bin/curator", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/curator/curator.py", line 5, in main
    cli( obj={ "filters": [] } )
  File "/usr/lib/python2.6/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.6/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.6/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 86, in augment_usage_errors
    yield
  File "/usr/lib/python2.6/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/curator/cli/index_selection.py", line 161, in indices
    retval = do_command(client, ctx.parent.info_name, l, ctx.parent.params, master_timeout)
  File "/usr/lib/python2.6/site-packages/curator/cli/utils.py", line 227, in do_command
    return close(client, indices)
  File "/usr/lib/python2.6/site-packages/curator/api/close.py", line 38, in close
    return close_indices(client, indices)
  File "/usr/lib/python2.6/site-packages/curator/api/close.py", line 19, in close_indices
    seal_indices(client, indices)
  File "/usr/lib/python2.6/site-packages/curator/api/seal.py", line 37, in seal_indices
    fails = [ i for i in sorted(results) if results[i]['failed'] > 0 ]
TypeError: sequence index must be integer, not 'str'
I also tried to close 1 index (the oldest) via the web interface and it seems to have worked.
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Log Server does not create backups

Post by mcapra »

The output will be fairly long, so you may want to direct it to a file. Share the output of the following:

Code: Select all

curator --debug --timeout 120 close indices --older-than 7 --time-unit days --timestring %Y.%m.%d
There might be an index that was corrupted.
Former Nagios employee
https://www.mcapra.com/
Top
Locked

Return to “Nagios Log Server”