Exception: Can't update non dynamic settings

[teirekos]

During backup_maintenace I get the following error on elasticsearch log:

[2015-08-03 14:36:12,573][DEBUG][action.admin.indices.settings.put] [845bc07c-ed91-4920-8e23-747c9cc699f5] failed to update settings on indices [logstash-2015.07.10]
org.elasticsearch.ElasticsearchIllegalArgumentException: Can't update non dynamic settings[[index.codec.bloom.load]] for open indices [[logstash-2015.07.10]]
at org.elasticsearch.cluster.metadata.MetaDataUpdateSettingsService$2.execute(MetaDataUpdateSettingsService.java:243)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:374)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:188)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:158)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


Any ideas? Thanx a lot.

[jolson]

It's possible that the index 2015.07.10 is unhealthy - let's check on the index health by using the following command:

Code: Select all

curl 'localhost:9200/_cluster/health?level=indices&pretty'

The index may need to be deleted and restored.

[teirekos]

I attach you the "prettty" output.
It looks fine. I had the same issue with 2015.07.09 and the I closed it and then the error message pointed to 2015.07.10.
Today since it closed the 19th it complains about 11!

2015-08-04 09:15:37,699 INFO Attempting to disable bloom filter for index logstash-2015.07.10.
2015-08-04 09:15:37,742 INFO Skipping index logstash-2015.07.10: Already closed.
2015-08-04 09:15:37,742 INFO Attempting to disable bloom filter for index logstash-2015.07.11.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 736, in <module>
main()
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 731, in main
arguments.func(client, **argdict)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 585, in command_loop
skipped = op(client, index_name, **kwargs)
File "/usr/lib/python2.6/site-packages/curator/curator.py", line 455, in _bloom_index
client.indices.put_settings(index=index_name, body='index.codec.bloom.load=false')
File "/usr/lib/python2.6/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.6/site-packages/elasticsearch/client/indices.py", line 596, in put_settings
params=params, body=body)
File "/usr/lib/python2.6/site-packages/elasticsearch/transport.py", line 307, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.6/site-packages/elasticsearch/connection/http_urllib3.py", line 86, in perform_request
self._raise_error(response.status, raw_data)
File "/usr/lib/python2.6/site-packages/elasticsearch/connection/base.py", line 102, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, u"RemoteTransportException[[7387f4a4-c097-4c4a-9f83-22ae496b3eac][inet[/10.1.11.10:9300]][indices:admin/settings/update]]; nested: ElasticsearchIllegalArgumentException[Can't update non dynamic settings[[index.codec.bloom.load]] for open indices [[logstash-2015.07.11]]]; ")
SUCCESS

[jolson]

Let's check on the md5sum of your curator package - I'd like to verify that your version of curator was updated properly.

Code: Select all

md5sum /usr/lib/python2.6/site-packages/curator/curator.py

What is your Disable Bloom Filter Cache older than setting? You can find this setting at 'Administration -> Backup & Maintenance'. Try setting it to '0' as a test and see if you have better luck running your backups.

[teirekos]

Correct there is a curator version mismatch.

NagiosLogServer1 (fresh setup with the latest version 2015R2.1 from ova file)

[root@NagiosLogServer1 ~]# md5sum /usr/lib/python2.6/site-packages/curator/curator.py
ba8b1fb0d80fd9fbe3de26d162317444 /usr/lib/python2.6/site-packages/curator/curator.py

NagiosLogServer2 (master node - upgraded with source tar file all the way... to the latest 2015R2.1)

[root@NagiosLogServer2 ~]# md5sum /usr/lib/python2.6/site-packages/curator/curator.py
2098632ed35e30491a0a95fc1895d382 /usr/lib/python2.6/site-packages/curator/curator.py

NagiosLogServer3 (fresh setup with the latest version 2015R2.1 from ova file)

[root@NagiosLogServer3 ~]# md5sum /usr/lib/python2.6/site-packages/curator/curator.py
ba8b1fb0d80fd9fbe3de26d162317444 /usr/lib/python2.6/site-packages/curator/curator.py


And yes by disabling the Bloom Filter Cache there are no exceptions (as seen on the attached txt).

[jolson]

This definitely appears to be a bug - it could very likely be caused by the following change made in more recent versions of elasticsearch:
https://github.com/elastic/elasticsearch/pull/6959

I have opened up a bug report with our development team - the task ID is 6191.

Regarding the version of curator that you have installed - 2098632ed35e30491a0a95fc1895d382 is the correct md5sum. Feel free to copy this file to your other instances with an incorrect sum and replace the old curator.py file.

[teirekos]

Thanx.
Pls close the thread.

[jolson]

teirekos,

In Elasticsearch versions 1.4 and higher, the bloom filter is disabled by default due to the change I mentioned above. In our next release of NLS we will be disabling the bloom filter setting and removing 'BLOOM FILTER' from the backup/maintenance process. Thanks for the bug report!