Issues after upgrade to 2.1.12

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
CBoekhuis
Posts: 212
Joined: Tue Aug 16, 2011 4:55 am

Issues after upgrade to 2.1.12

Post by CBoekhuis »

Today I upgraded our cluster from 2.1.9 to 2.1.12, however the shard allocation broke. Cluster was yellow and the number of unassigned shards didn't decrease.
After several restart and a near mental break down I finally discovered that the elasticsearch.yml config had been overwritten to default, hence removing our changes. So the upgrade left me with the following conclusions:
- /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml was overwritten which never happened in the previous upgrades (to my knowledge)
- Despite my cluster.routing.allocation.disk.watermark.X configs being removed (obviously the disk space was less then XX%), there was no watermark warning in /var/log/elasticsearch/*.log !!
- Still one log4j jar file in the techstack: /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/jruby-kafka-1.5.0-java/lib/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar

Shards are now allocating again with my settings, but heads up when you upgrade! elasticsearch.yml config apparently will be overwritten.
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Issues after upgrade to 2.1.12

Post by ssax »

I went from 2.1.10 to 2.1.12 and it didn't replace my /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml file.

What OS are you running on the backend?

Code: Select all

uname -a
cat /etc/*release
CBoekhuis
Posts: 212
Joined: Tue Aug 16, 2011 4:55 am

Re: Issues after upgrade to 2.1.12

Post by CBoekhuis »

uname -a:
Linux colog2.boekhuis.nl 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/*release:
CentOS Linux release 7.9.2009 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.9.2009 (Core)
CentOS Linux release 7.9.2009 (Core)

In the mean time I checked the file on our test cluster which I upgraded first. The config file had the time stamp February 11th. That's when I upgraded from 2.1.9 to 2.1.11:
-rwxrwxr-x. 1 nagios nagios 13251 Feb 11 15:23 /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml

On the 23th I upgraded the test cluster from 2.1.11 to 2.1.12 and it didn't change the elasticsearch.yml. The elasticsearch.yml on the test cluster has no custom changes, therefor I didn't run into issues there.

After the test upgrade I upgraded the production cluster from 2.1.9 to 2.1.12 where the elasticsearch.yml got replaced by the default one.
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Issues after upgrade to 2.1.12

Post by ssax »

I just labbed this up as well with 2.1.9 and I was able to replicate it.
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Issues after upgrade to 2.1.12

Post by ssax »

I have submitted a bug report to development:

Code: Select all

NEW TASK ID 15717 created - Nagios Log Server Bug Report: LS - Upgrade from LS 2.1.9 to 2.1.12 replaces the elasticsearch.yml file
Locked