Issues after upgrade to 2.1.12
Posted: Wed Feb 23, 2022 10:12 am
Today I upgraded our cluster from 2.1.9 to 2.1.12, however the shard allocation broke. Cluster was yellow and the number of unassigned shards didn't decrease.
After several restart and a near mental break down I finally discovered that the elasticsearch.yml config had been overwritten to default, hence removing our changes. So the upgrade left me with the following conclusions:
- /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml was overwritten which never happened in the previous upgrades (to my knowledge)
- Despite my cluster.routing.allocation.disk.watermark.X configs being removed (obviously the disk space was less then XX%), there was no watermark warning in /var/log/elasticsearch/*.log !!
- Still one log4j jar file in the techstack: /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/jruby-kafka-1.5.0-java/lib/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar
Shards are now allocating again with my settings, but heads up when you upgrade! elasticsearch.yml config apparently will be overwritten.
After several restart and a near mental break down I finally discovered that the elasticsearch.yml config had been overwritten to default, hence removing our changes. So the upgrade left me with the following conclusions:
- /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml was overwritten which never happened in the previous upgrades (to my knowledge)
- Despite my cluster.routing.allocation.disk.watermark.X configs being removed (obviously the disk space was less then XX%), there was no watermark warning in /var/log/elasticsearch/*.log !!
- Still one log4j jar file in the techstack: /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/jruby-kafka-1.5.0-java/lib/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar
Shards are now allocating again with my settings, but heads up when you upgrade! elasticsearch.yml config apparently will be overwritten.