PARTIAL backups

[stecino]

I should have clarified - when you run the following command:

Code: Select all

curator snapshot --repository nlsback --older-than 1

Be certain that 'nlsback' is replaced with the name of your repository (which could be something like 'NLS Backups').

e.g.

Code: Select all

curator snapshot --repository 'NLS Backup Repo' --older-than 1

curator snapshot --repository NLS_BACKUP --older-than 1
2015-08-26 17:16:33,158 INFO Job starting...
2015-08-26 17:16:33,159 INFO Default timeout of 30 seconds is too low for command SNAPSHOT. Overriding to 21,600 seconds (6 hours).
2015-08-26 17:16:33,166 INFO Beginning SNAPSHOT operations...
2015-08-26 17:16:33,227 INFO Attempting to create snapshot for index logstash-2015.07.28.
2015-08-26 17:16:33,768 INFO Skipping: A snapshot with name 'logstash-2015.07.28' already exists.
2015-08-26 17:16:33,768 INFO Attempting to create snapshot for index logstash-2015.07.29.
2015-08-26 17:16:34,227 INFO Skipping: A snapshot with name 'logstash-2015.07.29' already exists.
2015-08-26 17:16:34,227 INFO Attempting to create snapshot for index logstash-2015.07.30.
2015-08-26 17:16:34,697 INFO Skipping: A snapshot with name 'logstash-2015.07.30' already exists.
2015-08-26 17:16:34,698 INFO Attempting to create snapshot for index logstash-2015.07.31.
2015-08-26 17:16:35,153 INFO Skipping: A snapshot with name 'logstash-2015.07.31' already exists.
2015-08-26 17:16:35,153 INFO Attempting to create snapshot for index logstash-2015.08.01.
2015-08-26 17:16:35,605 INFO Skipping: A snapshot with name 'logstash-2015.08.01' already exists.
2015-08-26 17:16:35,606 INFO Attempting to create snapshot for index logstash-2015.08.02.
2015-08-26 17:16:36,051 INFO Skipping: A snapshot with name 'logstash-2015.08.02' already exists.
2015-08-26 17:16:36,051 INFO Attempting to create snapshot for index logstash-2015.08.03.
2015-08-26 17:16:36,513 INFO Skipping: A snapshot with name 'logstash-2015.08.03' already exists.
2015-08-26 17:16:36,513 INFO Attempting to create snapshot for index logstash-2015.08.04.
2015-08-26 17:16:36,982 INFO Skipping: A snapshot with name 'logstash-2015.08.04' already exists.
2015-08-26 17:16:36,983 INFO Attempting to create snapshot for index logstash-2015.08.05.
2015-08-26 17:16:37,281 WARNING Previous snapshot was unsuccessful. Deleting snapshot logstash-2015.08.05 and trying again.
Traceback (most recent call last):
File "/usr/bin/curator", line 9, in <module>
load_entry_point('elasticsearch-curator==1.2.2', 'console_scripts', 'curator')()
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 731, in main
arguments.func(client, **argdict)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 585, in command_loop
skipped = op(client, index_name, **kwargs)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 409, in _create_snapshot
client.snapshot.delete(repository=repository, snapshot=snap_name)
File "/usr/lib/python2.6/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.6/site-packages/elasticsearch/client/snapshot.py", line 34, in delete
_make_path('_snapshot', repository, snapshot), params=params)
File "/usr/lib/python2.6/site-packages/elasticsearch/transport.py", line 284, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.6/site-packages/elasticsearch/connection/http_urllib3.py", line 55, in perform_request
self._raise_error(response.status, raw_data)
File "/usr/lib/python2.6/site-packages/elasticsearch/connection/base.py", line 97, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.TransportError: TransportError(503, u'RemoteTransportException[[78a3bc74-18f9-46c6-a763-267d4860c047][inet[/xx.xx.xx.246:9300]][cluster:admin/snapshot/delete]]; nested: ConcurrentSnapshotExecutionException[[NLS_BACKUP:logstash-2015.08.05] another snapshot is currently running cannot delete]; ')

[jolson]

Let's take a look at your cluster state. I'm particularly interested in seeing if you have any hanging snapshots - which could explain why further snapshots fail. I'm also looking for the md5 sum of your curator.py file.

Cluster state:

Code: Select all

curl -XGET 'http://localhost:9200/_cluster/state?pretty'

Md5sum:

Code: Select all

md5sum /usr/lib/python2.6/site-packages/curator/curator.py

I'm also interested in your knapsack states:

Code: Select all

curl -XPOST 'http://localhost:9200/_export/state'

[stecino]

Let's take a look at your cluster state. I'm particularly interested in seeing if you have any hanging snapshots - which could explain why further snapshots fail. I'm also looking for the md5 sum of your curator.py file.

Cluster state:

Code: Select all

curl -XGET 'http://localhost:9200/_cluster/state?pretty'

Md5sum:

Code: Select all

md5sum /usr/lib/python2.6/site-packages/curator/curator.py

I'm also interested in your knapsack states:

Code: Select all

curl -XPOST 'http://localhost:9200/_export/state'

Which particular section of the cluster json would you like to see?
There are no snapshots:

"snapshots" : {
"snapshots" : [ ]



Md5Sum: md5sum /usr/lib/python2.6/site-packages/curator/curator.py
2098632ed35e30491a0a95fc1895d382 /usr/lib/python2.6/site-packages/curator/curator.py

curl -XPOST 'http://localhost:9200/_export/state'
{"count":0,"states":[]}

[jolson]

I have attached a known working version of curator.py (you will need to unzip it). Your md5sum is incorrect, I think that curator.py must not have been overwritten during your upgrade when we moved to the new version.

Try replacing your current version of curator.py with this new one.

Move the old one to your home directory for safety:

Code: Select all

mv /usr/lib/python2.6/site-packages/curator/curator.py ~/curator.py.backup

Put the new one in its place with the same permissions:

Code: Select all

mv curator.py /usr/lib/python2.6/site-packages/curator/curator.py
chown root.root /usr/lib/python2.6/site-packages/curator/curator.py
chmod 644 /usr/lib/python2.6/site-packages/curator/curator.py

Do the above on all of your nodes. The proper md5sum is 9d19626b8486f05156c77a0dacc93343.

[stecino]

I have attached a known working version of curator.py (you will need to unzip it). Your md5sum is incorrect, I think that curator.py must not have been overwritten during your upgrade when we moved to the new version.

Try replacing your current version of curator.py with this new one.

Move the old one to your home directory for safety:

Code: Select all

mv /usr/lib/python2.6/site-packages/curator/curator.py ~/curator.py.backup

Put the new one in its place with the same permissions:

Code: Select all

mv curator.py /usr/lib/python2.6/site-packages/curator/curator.py
chown root.root /usr/lib/python2.6/site-packages/curator/curator.py
chmod 644 /usr/lib/python2.6/site-packages/curator/curator.py

Do the above on all of your nodes. The proper md5sum is 9d19626b8486f05156c77a0dacc93343.

md5sum /usr/lib/python2.6/site-packages/curator/curator.py
9d19626b8486f05156c77a0dacc93343 /usr/lib/python2.6/site-packages/curator/curator.py

[stecino]

curator snapshot --repository NLS_BACKUP --older-than 1
2015-08-26 20:52:57,964 INFO Job starting...
2015-08-26 20:52:57,965 INFO Default timeout of 30 seconds is too low for command SNAPSHOT. Overriding to 21,600 seconds (6 hours).
2015-08-26 20:52:57,976 INFO Beginning SNAPSHOT operations...
2015-08-26 20:52:58,063 INFO Attempting to create snapshot for index logstash-2015.07.28.
2015-08-26 20:52:58,677 INFO Skipping: A snapshot with name 'logstash-2015.07.28' already exists.
2015-08-26 20:52:58,678 INFO Attempting to create snapshot for index logstash-2015.07.29.
2015-08-26 20:52:59,270 INFO Skipping: A snapshot with name 'logstash-2015.07.29' already exists.
2015-08-26 20:52:59,271 INFO Attempting to create snapshot for index logstash-2015.07.30.
2015-08-26 20:52:59,876 INFO Skipping: A snapshot with name 'logstash-2015.07.30' already exists.
2015-08-26 20:52:59,876 INFO Attempting to create snapshot for index logstash-2015.07.31.
2015-08-26 20:53:00,470 INFO Skipping: A snapshot with name 'logstash-2015.07.31' already exists.
2015-08-26 20:53:00,471 INFO Attempting to create snapshot for index logstash-2015.08.01.
2015-08-26 20:53:01,131 INFO Skipping: A snapshot with name 'logstash-2015.08.01' already exists.
2015-08-26 20:53:01,132 INFO Attempting to create snapshot for index logstash-2015.08.02.
2015-08-26 20:53:01,752 INFO Skipping: A snapshot with name 'logstash-2015.08.02' already exists.
2015-08-26 20:53:01,753 INFO Attempting to create snapshot for index logstash-2015.08.03.
2015-08-26 20:53:02,332 INFO Skipping: A snapshot with name 'logstash-2015.08.03' already exists.
2015-08-26 20:53:02,332 INFO Attempting to create snapshot for index logstash-2015.08.04.
2015-08-26 20:53:02,944 INFO Skipping: A snapshot with name 'logstash-2015.08.04' already exists.
2015-08-26 20:53:02,944 INFO Attempting to create snapshot for index logstash-2015.08.05.
2015-08-26 20:53:03,417 WARNING Previous snapshot was unsuccessful. Deleting snapshot logstash-2015.08.05 and trying again.
Traceback (most recent call last):
File "/usr/bin/curator", line 9, in <module>
load_entry_point('elasticsearch-curator==1.2.2', 'console_scripts', 'curator')()
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 731, in main
arguments.func(client, **argdict)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 585, in command_loop
skipped = op(client, index_name, **kwargs)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 409, in _create_snapshot
_delete_snapshot(client, repository, snap_name)
TypeError: _delete_snapshot() takes exactly 2 arguments (3 given)

[jolson]

Try deleting all of your PARTIAL backups via the Web GUI, and running the curator command once more.

[stecino]

Try deleting all of your PARTIAL backups via the Web GUI, and running the curator command once more.

Looks like that fixed the issue. I will let the scheduled backup run at night. I don't want to task the system at the moment by the backup

[jolson]

Happy to hear it - looking forward to hearing back from you.

[stecino]

Happy to hear it - looking forward to hearing back from you.

Sorry to disappoint, but I am still having Partial backups

elasticsearch log is full with these errors

[2015-08-27 13:29:01,355][WARN ][repositories.fs ] [00a07fa5-7288-4d96-880c-474285dad017] The specified location [/nls_backup] should start with a repository path specified by the path.repo setting, but the path.repo setting was not set on this node
[2015-08-27 13:29:01,356][WARN ][repositories ] [00a07fa5-7288-4d96-880c-474285dad017] failed to create repository [fs][NLS_BACKUP]
org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, org.elasticsearch.repositories.RepositoryException: [NLS_BACKUP] location [/nls_backup] doesn't match any of the locations specified by path.repo because this setting is empty
at org.elasticsearch.repositories.fs.FsRepository.<init>(Unknown Source)
while locating org.elasticsearch.repositories.fs.FsRepository
while locating org.elasticsearch.repositories.Repository

1 error
at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:344)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:131)
at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:69)
at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:409)
at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:304)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:480)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:188)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:158)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.repositories.RepositoryException: [NLS_BACKUP] location [/nls_backup] doesn't match any of the locations specified by path.repo because this setting is empty
at org.elasticsearch.repositories.fs.FsRepository.<init>(FsRepository.java:85)
at sun.reflect.GeneratedConstructorAccessor102.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
... 11 more
[2015-08-27 13:29:01,357][WARN ][repositories ] [00a07fa5-7288-4d96-880c-474285dad017] failed to create repository [NLS_BACKUP]
org.elasticsearch.repositories.RepositoryException: [NLS_BACKUP] failed to create repository
at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:416)
at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:304)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:480)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:188)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:158)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, org.elasticsearch.repositories.RepositoryException: [NLS_BACKUP] location [/nls_backup] doesn't match any of the locations specified by path.repo because this setting is empty
at org.elasticsearch.repositories.fs.FsRepository.<init>(Unknown Source)
while locating org.elasticsearch.repositories.fs.FsRepository
while locating org.elasticsearch.repositories.Repository

1 error
at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:344)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:131)
at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:69)
at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:409)
... 7 more