Nagios Support Forum

Posted: **Wed Mar 18, 2015 1:23 pm**

Did anything show up in your backup repository? When you submit a job from the Command Subsystem, a random node in the cluster will run the job. Be sure that all nodes have access to the repository and that the permissions on the repo are setup properly. Again, did you tail all of the logs or just logs from one node? Thank you for your patience.

Jesse

Posted: **Wed Mar 18, 2015 1:33 pm**

stecino wrote:
jolson wrote:Please click the 'reset all jobs' button. This will set all jobs to their default values (and names). After this has been done, click 'Edit' next to 'backup_maintenance' and set the job to run 'Now'. This is a new feature that we implemented that makes it easier to run jobs. Let me know if that works for you!

backup_maintenance Waiting SUCCESS 03/18/2015 11:10:12 1 day 03/19/2015 11:10:12 System Edit

nothing in the logs

Tried one more time

Code: Select all

Running command do_maintenance with args ' ' for job id: backup_maintenance
2015-03-18 11:30:06,734 INFO      Job starting...
2015-03-18 11:30:06,738 INFO      Beginning BLOOM operations...
2015-03-18 11:30:06,758 INFO      Attempting to disable bloom filter for index logstash-2015.02.17.
2015-03-18 11:30:07,110 INFO      logstash-2015.02.17: Successfully bloom filter disabled.
2015-03-18 11:30:07,110 INFO      Attempting to disable bloom filter for index logstash-2015.02.18.
2015-03-18 11:30:07,520 INFO      logstash-2015.02.18: Successfully bloom filter disabled.
2015-03-18 11:30:07,521 INFO      Attempting to disable bloom filter for index logstash-2015.02.19.
2015-03-18 11:30:07,905 INFO      logstash-2015.02.19: Successfully bloom filter disabled.
2015-03-18 11:30:07,906 INFO      Attempting to disable bloom filter for index logstash-2015.02.20.
2015-03-18 11:30:08,303 INFO      logstash-2015.02.20: Successfully bloom filter disabled.
2015-03-18 11:30:08,303 INFO      Attempting to disable bloom filter for index logstash-2015.02.21.
2015-03-18 11:30:08,681 INFO      logstash-2015.02.21: Successfully bloom filter disabled.
2015-03-18 11:30:08,682 INFO      Attempting to disable bloom filter for index logstash-2015.02.22.
2015-03-18 11:30:09,064 INFO      logstash-2015.02.22: Successfully bloom filter disabled.
2015-03-18 11:30:09,064 INFO      Attempting to disable bloom filter for index logstash-2015.02.23.
2015-03-18 11:30:09,468 INFO      logstash-2015.02.23: Successfully bloom filter disabled.
2015-03-18 11:30:09,469 INFO      Attempting to disable bloom filter for index logstash-2015.02.24.
2015-03-18 11:30:09,837 INFO      logstash-2015.02.24: Successfully bloom filter disabled.
2015-03-18 11:30:09,838 INFO      Attempting to disable bloom filter for index logstash-2015.02.25.
2015-03-18 11:30:09,992 INFO      logstash-2015.02.25: Successfully bloom filter disabled.
2015-03-18 11:30:09,993 INFO      Attempting to disable bloom filter for index logstash-2015.02.26.
2015-03-18 11:30:10,205 INFO      logstash-2015.02.26: Successfully bloom filter disabled.
2015-03-18 11:30:10,206 INFO      Attempting to disable bloom filter for index logstash-2015.02.27.
2015-03-18 11:30:10,544 INFO      logstash-2015.02.27: Successfully bloom filter disabled.
2015-03-18 11:30:10,544 INFO      Attempting to disable bloom filter for index logstash-2015.02.28.
2015-03-18 11:30:10,813 INFO      logstash-2015.02.28: Successfully bloom filter disabled.
2015-03-18 11:30:10,814 INFO      Attempting to disable bloom filter for index logstash-2015.03.01.
2015-03-18 11:30:11,148 INFO      logstash-2015.03.01: Successfully bloom filter disabled.
2015-03-18 11:30:11,149 INFO      Attempting to disable bloom filter for index logstash-2015.03.02.
2015-03-18 11:30:11,541 INFO      logstash-2015.03.02: Successfully bloom filter disabled.
2015-03-18 11:30:11,542 INFO      Attempting to disable bloom filter for index logstash-2015.03.03.
2015-03-18 11:30:11,920 INFO      logstash-2015.03.03: Successfully bloom filter disabled.
2015-03-18 11:30:11,920 INFO      Attempting to disable bloom filter for index logstash-2015.03.04.
2015-03-18 11:30:12,343 INFO      logstash-2015.03.04: Successfully bloom filter disabled.
2015-03-18 11:30:12,343 INFO      Attempting to disable bloom filter for index logstash-2015.03.05.
2015-03-18 11:30:12,774 INFO      logstash-2015.03.05: Successfully bloom filter disabled.
2015-03-18 11:30:12,774 INFO      Attempting to disable bloom filter for index logstash-2015.03.06.
2015-03-18 11:30:13,171 INFO      logstash-2015.03.06: Successfully bloom filter disabled.
2015-03-18 11:30:13,171 INFO      Attempting to disable bloom filter for index logstash-2015.03.07.
2015-03-18 11:30:13,531 INFO      logstash-2015.03.07: Successfully bloom filter disabled.
2015-03-18 11:30:13,531 INFO      Attempting to disable bloom filter for index logstash-2015.03.08.
2015-03-18 11:30:13,852 INFO      logstash-2015.03.08: Successfully bloom filter disabled.
2015-03-18 11:30:13,853 INFO      Attempting to disable bloom filter for index logstash-2015.03.09.
2015-03-18 11:30:14,085 INFO      logstash-2015.03.09: Successfully bloom filter disabled.
2015-03-18 11:30:14,085 INFO      Attempting to disable bloom filter for index logstash-2015.03.10.
2015-03-18 11:30:14,314 INFO      logstash-2015.03.10: Successfully bloom filter disabled.
2015-03-18 11:30:14,314 INFO      Attempting to disable bloom filter for index logstash-2015.03.11.
2015-03-18 11:30:14,693 INFO      logstash-2015.03.11: Successfully bloom filter disabled.
2015-03-18 11:30:14,693 INFO      Attempting to disable bloom filter for index logstash-2015.03.12.
2015-03-18 11:30:15,151 INFO      logstash-2015.03.12: Successfully bloom filter disabled.
2015-03-18 11:30:15,151 INFO      Attempting to disable bloom filter for index logstash-2015.03.13.
2015-03-18 11:30:15,611 INFO      logstash-2015.03.13: Successfully bloom filter disabled.
2015-03-18 11:30:15,612 INFO      Attempting to disable bloom filter for index logstash-2015.03.14.
2015-03-18 11:30:16,045 INFO      logstash-2015.03.14: Successfully bloom filter disabled.
2015-03-18 11:30:16,046 INFO      Attempting to disable bloom filter for index logstash-2015.03.15.
2015-03-18 11:30:16,487 INFO      logstash-2015.03.15: Successfully bloom filter disabled.
2015-03-18 11:30:16,487 INFO      Attempting to disable bloom filter for index logstash-2015.03.16.
2015-03-18 11:30:16,892 INFO      logstash-2015.03.16: Successfully bloom filter disabled.
2015-03-18 11:30:16,892 INFO      Attempting to disable bloom filter for index logstash-2015.03.17.
2015-03-18 11:30:17,327 INFO      logstash-2015.03.17: Successfully bloom filter disabled.
2015-03-18 11:30:17,328 INFO      logstash-2015.03.18 is within the threshold period (1 days).
2015-03-18 11:30:17,328 INFO      logstash-2015.03.19 is within the threshold period (1 days).
2015-03-18 11:30:17,328 INFO      DISABLE BLOOM FILTER index operations completed.
2015-03-18 11:30:17,329 INFO      Done in 0:00:10.605645.
2015-03-18 11:30:17,662 INFO      Job starting...
2015-03-18 11:30:17,663 INFO      Default timeout of 30 seconds is too low for command OPTIMIZE.  Overriding to 21,600 seconds (6 hours).
2015-03-18 11:30:17,670 INFO      Beginning OPTIMIZE operations...
2015-03-18 11:30:17,698 INFO      Attempting to optimize index logstash-2015.02.17.
2015-03-18 11:30:17,736 INFO      Job starting...
2015-03-18 11:30:17,743 INFO      Beginning CLOSE operations...
2015-03-18 11:30:17,773 INFO      logstash-2015.02.17 is within the threshold period (30 days).
2015-03-18 11:30:17,774 INFO      logstash-2015.02.18 is within the threshold period (30 days).
2015-03-18 11:30:17,774 INFO      logstash-2015.02.19 is within the threshold period (30 days).
2015-03-18 11:30:17,775 INFO      logstash-2015.02.20 is within the threshold period (30 days).
2015-03-18 11:30:17,775 INFO      logstash-2015.02.21 is within the threshold period (30 days).
2015-03-18 11:30:17,776 INFO      logstash-2015.02.22 is within the threshold period (30 days).
2015-03-18 11:30:17,776 INFO      logstash-2015.02.23 is within the threshold period (30 days).
2015-03-18 11:30:17,776 INFO      logstash-2015.02.24 is within the threshold period (30 days).
2015-03-18 11:30:17,777 INFO      logstash-2015.02.25 is within the threshold period (30 days).
2015-03-18 11:30:17,777 INFO      logstash-2015.02.26 is within the threshold period (30 days).
2015-03-18 11:30:17,778 INFO      logstash-2015.02.27 is within the threshold period (30 days).
2015-03-18 11:30:17,778 INFO      logstash-2015.02.28 is within the threshold period (30 days).
2015-03-18 11:30:17,778 INFO      logstash-2015.03.01 is within the threshold period (30 days).
2015-03-18 11:30:17,779 INFO      logstash-2015.03.02 is within the threshold period (30 days).
2015-03-18 11:30:17,779 INFO      logstash-2015.03.03 is within the threshold period (30 days).
2015-03-18 11:30:17,779 INFO      logstash-2015.03.04 is within the threshold period (30 days).
2015-03-18 11:30:17,780 INFO      logstash-2015.03.05 is within the threshold period (30 days).
2015-03-18 11:30:17,780 INFO      logstash-2015.03.06 is within the threshold period (30 days).
2015-03-18 11:30:17,780 INFO      logstash-2015.03.07 is within the threshold period (30 days).
2015-03-18 11:30:17,781 INFO      logstash-2015.03.08 is within the threshold period (30 days).
2015-03-18 11:30:17,781 INFO      logstash-2015.03.09 is within the threshold period (30 days).
2015-03-18 11:30:17,781 INFO      logstash-2015.03.10 is within the threshold period (30 days).
2015-03-18 11:30:17,782 INFO      logstash-2015.03.11 is within the threshold period (30 days).
2015-03-18 11:30:17,782 INFO      logstash-2015.03.12 is within the threshold period (30 days).
2015-03-18 11:30:17,782 INFO      logstash-2015.03.13 is within the threshold period (30 days).
2015-03-18 11:30:17,783 INFO      logstash-2015.03.14 is within the threshold period (30 days).
2015-03-18 11:30:17,783 INFO      logstash-2015.03.15 is within the threshold period (30 days).
2015-03-18 11:30:17,783 INFO      logstash-2015.03.16 is within the threshold period (30 days).
2015-03-18 11:30:17,784 INFO      logstash-2015.03.17 is within the threshold period (30 days).
2015-03-18 11:30:17,784 INFO      logstash-2015.03.18 is within the threshold period (30 days).
2015-03-18 11:30:17,784 INFO      logstash-2015.03.19 is within the threshold period (30 days).
operations completed.
2015-03-18 11:30:17,785 INFO      Done in 0:00:00.072569.
2015-03-18 11:30:17,791 INFO      Beginning DELETE operations...
SUCCESS
2015-03-18 11:30:17,819 INFO      logstash-2015.02.17 is within the threshold period (30 days).
2015-03-18 11:30:17,819 INFO      logstash-2015.02.18 is within the threshold period (30 days).
2015-03-18 11:30:17,820 INFO      logstash-2015.02.19 is within the threshold period (30 days).
2015-03-18 11:30:17,820 INFO      logstash-2015.02.20 is within the threshold period (30 days).
2015-03-18 11:30:17,821 INFO      logstash-2015.02.21 is within the threshold period (30 days).
2015-03-18 11:30:17,821 INFO      logstash-2015.02.22 is within the threshold period (30 days).
2015-03-18 11:30:17,821 INFO      logstash-2015.02.23 is within the threshold period (30 days).
2015-03-18 11:30:17,822 INFO      logstash-2015.02.24 is within the threshold period (30 days).
2015-03-18 11:30:17,822 INFO      logstash-2015.02.25 is within the threshold period (30 days).
2015-03-18 11:30:17,822 INFO      logstash-2015.02.26 is within the threshold period (30 days).
2015-03-18 11:30:17,823 INFO      logstash-2015.02.27 is within the threshold period (30 days).
2015-03-18 11:30:17,823 INFO      logstash-2015.02.28 is within the threshold period (30 days).
2015-03-18 11:30:17,823 INFO      logstash-2015.03.01 is within the threshold period (30 days).
2015-03-18 11:30:17,824 INFO      logstash-2015.03.02 is within the threshold period (30 days).
2015-03-18 11:30:17,824 INFO      logstash-2015.03.03 is within the threshold period (30 days).
2015-03-18 11:30:17,824 INFO      logstash-2015.03.04 is within the threshold period (30 days).
2015-03-18 11:30:17,825 INFO      logstash-2015.03.05 is within the threshold period (30 days).
2015-03-18 11:30:17,825 INFO      logstash-2015.03.06 is within the threshold period (30 days).
2015-03-18 11:30:17,826 INFO      logstash-2015.03.07 is within the threshold period (30 days).
2015-03-18 11:30:17,826 INFO      logstash-2015.03.08 is within the threshold period (30 days).
2015-03-18 11:30:17,826 INFO      logstash-2015.03.09 is within the threshold period (30 days).
2015-03-18 11:30:17,827 INFO      logstash-2015.03.10 is within the threshold period (30 days).
2015-03-18 11:30:17,827 INFO      logstash-2015.03.11 is within the threshold period (30 days).
2015-03-18 11:30:17,827 INFO      logstash-2015.03.12 is within the threshold period (30 days).
2015-03-18 11:30:17,828 INFO      logstash-2015.03.13 is within the threshold period (30 days).
2015-03-18 11:30:17,828 INFO      logstash-2015.03.14 is within the threshold period (30 days).
2015-03-18 11:30:17,828 INFO      logstash-2015.03.15 is within the threshold period (30 days).
2015-03-18 11:30:17,828 INFO      logstash-2015.03.16 is within the threshold period (30 days).
2015-03-18 11:30:17,829 INFO      logstash-2015.03.17 is within the threshold period (30 days).
2015-03-18 11:30:17,829 INFO      logstash-2015.03.18 is within the threshold period (30 days).
2015-03-18 11:30:17,830 INFO      logstash-2015.03.19 is within the threshold period (30 days).
2015-03-18 11:30:17,830 INFO      DELETE index operations completed.
2015-03-18 11:30:17,830 INFO      Done in 0:00:00.069883.
2015-03-18 11:30:17,835 INFO      Job starting...
2015-03-18 11:30:17,842 INFO      Beginning SNAPSHOT operations...
2015-03-18 11:30:17,903 INFO      Job starting...
2015-03-18 11:30:17,903 INFO      Default timeout of 30 seconds is too low for command SNAPSHOT.  Overriding to 21,600 seconds (6 hours).
2015-03-18 11:30:17,910 INFO      Beginning SNAPSHOT operations...
2015-03-18 11:30:17,940 INFO      Attempting to create snapshot for index logstash-2015.02.17.
2015-03-18 11:30:18,010 INFO      Skipping index logstash-2015.02.17: Already optimized.
2015-03-18 11:30:18,011 INFO      Attempting to optimize index logstash-2015.02.18.
2015-03-18 11:30:18,345 INFO      Skipping index logstash-2015.02.18: Already optimized.
2015-03-18 11:30:18,345 INFO      Attempting to optimize index logstash-2015.02.19.
2015-03-18 11:30:18,668 INFO      Skipping index logstash-2015.02.19: Already optimized.
2015-03-18 11:30:18,669 INFO      Attempting to optimize index logstash-2015.02.20.
2015-03-18 11:30:18,992 INFO      Skipping index logstash-2015.02.20: Already optimized.
2015-03-18 11:30:18,992 INFO      Attempting to optimize index logstash-2015.02.21.
2015-03-18 11:30:19,300 INFO      Skipping index logstash-2015.02.21: Already optimized.
2015-03-18 11:30:19,301 INFO      Attempting to optimize index logstash-2015.02.22.
2015-03-18 11:30:19,611 INFO      Skipping index logstash-2015.02.22: Already optimized.
2015-03-18 11:30:19,611 INFO      Attempting to optimize index logstash-2015.02.23.
2015-03-18 11:30:19,929 INFO      Skipping index logstash-2015.02.23: Already optimized.
2015-03-18 11:30:19,930 INFO      Attempting to optimize index logstash-2015.02.24.
2015-03-18 11:30:20,215 INFO      Skipping index logstash-2015.02.24: Already optimized.
2015-03-18 11:30:20,215 INFO      Attempting to optimize index logstash-2015.02.25.
2015-03-18 11:30:20,301 INFO      Skipping index logstash-2015.02.25: Already optimized.
2015-03-18 11:30:20,301 INFO      Attempting to optimize index logstash-2015.02.26.
2015-03-18 11:30:20,453 INFO      Skipping index logstash-2015.02.26: Already optimized.
2015-03-18 11:30:20,454 INFO      Attempting to optimize index logstash-2015.02.27.
2015-03-18 11:30:20,682 INFO      logstash-2015.02.17 is within the threshold period (30 days).
2015-03-18 11:30:20,682 INFO      logstash-2015.02.18 is within the threshold period (30 days).
2015-03-18 11:30:20,682 INFO      logstash-2015.02.19 is within the threshold period (30 days).
2015-03-18 11:30:20,683 INFO      logstash-2015.02.20 is within the threshold period (30 days).
2015-03-18 11:30:20,683 INFO      logstash-2015.02.21 is within the threshold period (30 days).
2015-03-18 11:30:20,684 INFO      logstash-2015.02.22 is within the threshold period (30 days).
2015-03-18 11:30:20,684 INFO      logstash-2015.02.23 is within the threshold period (30 days).
2015-03-18 11:30:20,684 INFO      logstash-2015.02.24 is within the threshold period (30 days).
2015-03-18 11:30:20,685 INFO      logstash-2015.02.25 is within the threshold period (30 days).
2015-03-18 11:30:20,685 INFO      logstash-2015.02.26 is within the threshold period (30 days).
2015-03-18 11:30:20,685 INFO      logstash-2015.02.28 is within the threshold period (30 days).
2015-03-18 11:30:20,686 INFO      logstash-2015.02.27 is within the threshold period (30 days).
2015-03-18 11:30:20,686 INFO      logstash-2015.03.01 is within the threshold period (30 days).
2015-03-18 11:30:20,686 INFO      logstash-2015.03.02 is within the threshold period (30 days).
2015-03-18 11:30:20,687 INFO      DELETE SNAPSHOT index operations completed.
2015-03-18 11:30:20,687 INFO      Done in 0:00:02.875199.
2015-03-18 11:30:20,788 INFO      Skipping index logstash-2015.02.27: Already optimized.
2015-03-18 11:30:20,789 INFO      Attempting to optimize index logstash-2015.02.28.
2015-03-18 11:30:21,114 INFO      Skipping index logstash-2015.02.28: Already optimized.
2015-03-18 11:30:21,114 INFO      Attempting to optimize index logstash-2015.03.01.
2015-03-18 11:30:21,282 INFO      Skipping: A snapshot with name 'logstash-2015.02.17' already exists.
2015-03-18 11:30:21,282 INFO      Attempting to create snapshot for index logstash-2015.02.18.
2015-03-18 11:30:21,401 INFO      Skipping index logstash-2015.03.01: Already optimized.
2015-03-18 11:30:21,401 INFO      Attempting to optimize index logstash-2015.03.02.
2015-03-18 11:30:21,511 INFO      Skipping: A snapshot with name 'logstash-2015.02.18' already exists.
2015-03-18 11:30:21,511 INFO      Attempting to create snapshot for index logstash-2015.02.19.
2015-03-18 11:30:21,571 INFO      Skipping index logstash-2015.03.02: Already optimized.
2015-03-18 11:30:21,571 INFO      Attempting to optimize index logstash-2015.03.03.
2015-03-18 11:30:21,729 INFO      Skipping: A snapshot with name 'logstash-2015.02.19' already exists.
2015-03-18 11:30:21,729 INFO      Attempting to create snapshot for index logstash-2015.02.20.
2015-03-18 11:30:21,733 INFO      Skipping index logstash-2015.03.03: Already optimized.
2015-03-18 11:30:21,734 INFO      Attempting to optimize index logstash-2015.03.04.
2015-03-18 11:30:21,937 INFO      Skipping index logstash-2015.03.04: Already optimized.
2015-03-18 11:30:21,937 INFO      Attempting to optimize index logstash-2015.03.05.
2015-03-18 11:30:22,070 INFO      Skipping: A snapshot with name 'logstash-2015.02.20' already exists.
2015-03-18 11:30:22,070 INFO      Attempting to create snapshot for index logstash-2015.02.21.
2015-03-18 11:30:22,260 INFO      Skipping index logstash-2015.03.05: Already optimized.
2015-03-18 11:30:22,260 INFO      Attempting to optimize index logstash-2015.03.06.
2015-03-18 11:30:22,431 INFO      Skipping: A snapshot with name 'logstash-2015.02.21' already exists.
2015-03-18 11:30:22,431 INFO      Attempting to create snapshot for index logstash-2015.02.22.
2015-03-18 11:30:22,612 INFO      Skipping index logstash-2015.03.06: Already optimized.
2015-03-18 11:30:22,612 INFO      Attempting to optimize index logstash-2015.03.07.
2015-03-18 11:30:22,795 INFO      Skipping: A snapshot with name 'logstash-2015.02.22' already exists.
2015-03-18 11:30:22,796 INFO      Attempting to create snapshot for index logstash-2015.02.23.
2015-03-18 11:30:22,858 INFO      Skipping index logstash-2015.03.07: Already optimized.
2015-03-18 11:30:22,859 INFO      Attempting to optimize index logstash-2015.03.08.
2015-03-18 11:30:23,132 INFO      Skipping index logstash-2015.03.08: Already optimized.
2015-03-18 11:30:23,132 INFO      Attempting to optimize index logstash-2015.03.09.
2015-03-18 11:30:23,149 INFO      Skipping: A snapshot with name 'logstash-2015.02.23' already exists.
2015-03-18 11:30:23,149 INFO      Attempting to create snapshot for index logstash-2015.02.24.
2015-03-18 11:30:23,433 INFO      Skipping index logstash-2015.03.09: Already optimized.
2015-03-18 11:30:23,434 INFO      Attempting to optimize index logstash-2015.03.10.
2015-03-18 11:30:23,494 INFO      Skipping: A snapshot with name 'logstash-2015.02.24' already exists.
2015-03-18 11:30:23,494 INFO      Attempting to create snapshot for index logstash-2015.02.25.
2015-03-18 11:30:23,635 INFO      Skipping: A snapshot with name 'logstash-2015.02.25' already exists.
2015-03-18 11:30:23,635 INFO      Attempting to create snapshot for index logstash-2015.02.26.
2015-03-18 11:30:23,741 INFO      Skipping index logstash-2015.03.10: Already optimized.
2015-03-18 11:30:23,741 INFO      Attempting to optimize index logstash-2015.03.11.
2015-03-18 11:30:23,830 INFO      Skipping: A snapshot with name 'logstash-2015.02.26' already exists.
2015-03-18 11:30:23,831 INFO      Attempting to create snapshot for index logstash-2015.02.27.
2015-03-18 11:30:24,118 INFO      Skipping index logstash-2015.03.11: Already optimized.
2015-03-18 11:30:24,118 INFO      Attempting to optimize index logstash-2015.03.12.
2015-03-18 11:30:24,206 INFO      Skipping: A snapshot with name 'logstash-2015.02.27' already exists.
2015-03-18 11:30:24,206 INFO      Attempting to create snapshot for index logstash-2015.02.28.
2015-03-18 11:30:24,494 INFO      Skipping index logstash-2015.03.12: Already optimized.
2015-03-18 11:30:24,494 INFO      Attempting to optimize index logstash-2015.03.13.
2015-03-18 11:30:24,543 INFO      Skipping: A snapshot with name 'logstash-2015.02.28' already exists.
2015-03-18 11:30:24,544 INFO      Attempting to create snapshot for index logstash-2015.03.01.
2015-03-18 11:30:24,784 INFO      Skipping index logstash-2015.03.13: Already optimized.
2015-03-18 11:30:24,784 INFO      Attempting to optimize index logstash-2015.03.14.
2015-03-18 11:30:24,862 WARNING   Previous snapshot was unsuccessful.  Deleting snapshot logstash-2015.03.01 and trying again.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/curator/curator.py", line 736, in <module>
    main()
  File "/usr/lib/python2.6/site-packages/curator/curator.py", line 731, in main
    arguments.func(client, **argdict)
  File "/usr/lib/python2.6/site-packages/curator/curator.py", line 585, in command_loop
    skipped = op(client, index_name, **kwargs)
  File "/usr/lib/python2.6/site-packages/curator/curator.py", line 409, in _create_snapshot
    _delete_snapshot(client, repository, snap_name)
TypeError: _delete_snapshot() takes exactly 2 arguments (3 given)
2015-03-18 11:30:25,049 INFO      Skipping index logstash-2015.03.14: Already optimized.
2015-03-18 11:30:25,049 INFO      Attempting to optimize index logstash-2015.03.15.
2015-03-18 11:30:25,329 INFO      Skipping index logstash-2015.03.15: Already optimized.
2015-03-18 11:30:25,330 INFO      Attempting to optimize index logstash-2015.03.16.
2015-03-18 11:30:25,628 INFO      Skipping index logstash-2015.03.16: Already optimized.
2015-03-18 11:30:25,629 INFO      logstash-2015.03.17 is within the threshold period (2 days).
2015-03-18 11:30:25,629 INFO      logstash-2015.03.18 is within the threshold period (2 days).
2015-03-18 11:30:25,630 INFO      logstash-2015.03.19 is within the threshold period (2 days).
2015-03-18 11:30:25,630 INFO      OPTIMIZE index operations completed.
2015-03-18 11:30:25,630 INFO      Done in 0:00:07.991523.
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 2 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command do_backups with args ' ' for job id: backups
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 4 global jobs.

nothing still generated

Posted: **Wed Mar 18, 2015 1:36 pm**

jolson wrote:Did anything show up in your backup repository? When you submit a job from the Command Subsystem, a random node in the cluster will run the job. Be sure that all nodes have access to the repository and that the permissions on the repo are setup properly. Again, did you tail all of the logs or just logs from one node? Thank you for your patience.

Jesse

Yes I verified that I can create/delete file on the mount from all nodes

Posted: **Wed Mar 18, 2015 1:49 pm**

I deleted one index logstash-2015.03.01 that said was partially backed up. And kicked off the job again.

SUCCESS
2015-03-18 11:43:23,972 INFO logstash-2015.03.01: Successfully created snapshot.
2015-03-18 11:43:23,972 INFO Attempting to create snapshot for index logstash-2015.03.02.
2015-03-18 11:43:24,646 INFO Skipping: A snapshot with name 'logstash-2015.03.02' already exists.
2015-03-18 11:43:24,647 INFO Attempting to create snapshot for index logstash-2015.03.03.

2015-03-18 11:47:15,856 INFO logstash-2015.03.03: Successfully created snapshot.
2015-03-18 11:47:15,856 INFO Attempting to create snapshot for index logstash-2015.03.04.

Looking good so far, I will keep you updated. What does partial backup mean, and why would it cause a problem?

Posted: **Wed Mar 18, 2015 2:00 pm**

Hello,

It looks like the deletion of your partial index failed for the following reason:

Code: Select all

      File "/usr/lib/python2.6/site-packages/curator/curator.py", line 409, in _create_snapshot
        _delete_snapshot(client, repository, snap_name)
TypeError: _delete_snapshot() takes exactly 2 arguments (3 given)

I will bring this up with our dev team. In the meantime, please let us know the progress of this case. Thanks!

Posted: **Wed Mar 18, 2015 4:43 pm**

stecino wrote:I deleted one index logstash-2015.03.01 that said was partially backed up. And kicked off the job again.

SUCCESS
2015-03-18 11:43:23,972 INFO logstash-2015.03.01: Successfully created snapshot.
2015-03-18 11:43:23,972 INFO Attempting to create snapshot for index logstash-2015.03.02.
2015-03-18 11:43:24,646 INFO Skipping: A snapshot with name 'logstash-2015.03.02' already exists.
2015-03-18 11:43:24,647 INFO Attempting to create snapshot for index logstash-2015.03.03.

2015-03-18 11:47:15,856 INFO logstash-2015.03.03: Successfully created snapshot.
2015-03-18 11:47:15,856 INFO Attempting to create snapshot for index logstash-2015.03.04.

Looking good so far, I will keep you updated. What does partial backup mean, and why would it cause a problem?

So it went up to creating snapshot for logstash-2015.03.05 and stopped. I am going to issue another backup to see if it will pick up things.

Posted: **Wed Mar 18, 2015 4:52 pm**

Sounds good to me - let us know how the backup process goes. As of right now, I'm not sure what happened to make your index stop the backups from occuring.

Posted: **Wed Mar 18, 2015 5:31 pm**

jolson wrote:Sounds good to me - let us know how the backup process goes. As of right now, I'm not sure what happened to make your index stop the backups from occuring.

It only picked up from March 5-March 12, it didn't see latest index files. But even then it failed

2015-03-18 14:46:35,147 INFO Attempting to create snapshot for index logstash-2015.03.04.
2015-03-18 14:46:35,822 INFO Skipping: A snapshot with name 'logstash-2015.03.04' already exists.
2015-03-18 14:46:35,823 INFO Attempting to create snapshot for index logstash-2015.03.05.
2015-03-18 14:46:36,479 INFO Skipping: A snapshot with name 'logstash-2015.03.05' already exists.
2015-03-18 14:46:36,480 INFO Attempting to create snapshot for index logstash-2015.03.06.
2015-03-18 14:46:40,105 INFO logstash-2015.03.06: Successfully created snapshot.
2015-03-18 14:46:40,105 INFO Attempting to create snapshot for index logstash-2015.03.07.
2015-03-18 14:46:43,940 INFO logstash-2015.03.07: Successfully created snapshot.
2015-03-18 14:46:43,940 INFO Attempting to create snapshot for index logstash-2015.03.08.
2015-03-18 14:46:47,968 INFO logstash-2015.03.08: Successfully created snapshot.
2015-03-18 14:46:47,968 INFO Attempting to create snapshot for index logstash-2015.03.09.
2015-03-18 14:46:52,199 INFO logstash-2015.03.09: Successfully created snapshot.
2015-03-18 14:46:52,199 INFO Attempting to create snapshot for index logstash-2015.03.10.
2015-03-18 14:46:56,652 INFO logstash-2015.03.10: Successfully created snapshot.
2015-03-18 14:46:56,652 INFO Attempting to create snapshot for index logstash-2015.03.11.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
2015-03-18 14:47:01,410 INFO logstash-2015.03.11: Successfully created snapshot.
2015-03-18 14:47:01,410 INFO Attempting to create snapshot for index logstash-2015.03.12.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Processed 0 node jobs.
Processed 0 global jobs.

logstash-2015.03.12 PARTIAL logstash-2015.03.12
logstash-2015.03.11 FAILED logstash-2015.03.11
logstash-2015.03.10 FAILED logstash-2015.03.10
logstash-2015.03.09 FAILED logstash-2015.03.09
logstash-2015.03.08 FAILED logstash-2015.03.08
logstash-2015.03.07 FAILED logstash-2015.03.07
logstash-2015.03.06 FAILED logstash-2015.03.06

Posted: **Thu Mar 19, 2015 10:19 am**

As for why the partial snapshot caused your backup to fail, it looks like curator.py had issues deleting it. I am not sure why, and I am still in discussion about that particular part of this issue.

As for why your backups are currently failing, it looks like a disk is being filled somewhere. Is your backup repository being filled up? This is the most likely cause of a PARTIAL backup.

Posted: **Fri Mar 20, 2015 4:15 pm**

jolson wrote:As for why the partial snapshot caused your backup to fail, it looks like curator.py had issues deleting it. I am not sure why, and I am still in discussion about that particular part of this issue.

As for why your backups are currently failing, it looks like a disk is being filled somewhere. Is your backup repository being filled up? This is the most likely cause of a PARTIAL backup.

No I have plenty of space, on the nodes, as well as on the backup mount. I deleted all the failed the backups, then scheduled one for the next day, it all worked. So I am not sure why these random errors happen

Nagios Support Forum

Backup Repository

Re: Backup Repository

Re: Backup Repository

Re: Backup Repository

Re: Backup Repository

Re: Backup Repository

Re: Backup Repository

Re: Backup Repository

Re: Backup Repository

Re: Backup Repository

Re: Backup Repository