Nagios Support Forum

I've upgraded one instance (from 2) and I get the following (so far):

i) Index Overview from Index Status is empty (screenshot attached)
ii) 62 unassigned shards (what shall I do with them?)
iii) The elasticsearch.yml config file was overwritten so from "discovery.zen.minimum_master_nodes: 2" now I have 1
[nagios@NagiosLogServer2 elasticsearch]$ grep minimum_master_nodes /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml
# discovery.zen.minimum_master_nodes: 1
[nagios@NagiosLogServer2 elasticsearch]$ ls -ltr /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml
-rwxrwxr-x 1 nagios nagios 13251 Jul 20 10:00 /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml

Thanx

Index Overview from Index Status is empty (screenshot attached)

I do not see this behavior on my test cluster - could you try upgrading your other instance as well and see if the behavior is reprimanded?

62 unassigned shards (what shall I do with them?)

This may have to do with the instances being on different versions - I recommend ensuring that both instances are on the latest version before we attempt troubleshooting the unassigned shards.

The elasticsearch.yml config file was overwritten so from "discovery.zen.minimum_master_nodes: 2" now I have 1

Feel free to change this configuration back and restart elasticsearch. The configuration file was re-written as part of the upgrade process.

I am in a bit of trouble here.
I've tried to upgrade my other instance as well and although the upgrade process was successful then I was not able to start elasticsearch. The error I got was:

"root@NagiosLogServer elasticsearch]# service elasticsearch status
elasticsearch dead but subsys locked"

I attach the logs in case you are interested.
Because that was the server I was receiving the logs (my 1st and oldest server of NLS i.e. many upgraded versions) in order to have at least a good instance I successfully restored it to a previous version. Thus now I have a problematic instance in the latest version (see post below) and a "good" one in an older version. In my "good" instance I also have mounted my backup disk.
It seems that with this upgrade where logstash and elasticsearch are upgraded the old but good instance mixes the versions.
What options do I have here?
I've allocated another VMware machine and installed the ova with the latest version of NLS in order to migrate the good server (I need to have the same IP) but how exactly can I safely do this? I was thinking to shutdown the 1st instance and start the new one with the same IP but how is it going to behave with the other problematic instance?
Thanx.

"root@NagiosLogServer elasticsearch]# service elasticsearch status
elasticsearch dead but subsys locked"

I'd like to do a remote session with you to try and get this resolved. Would you be willing to send a ticket request to [email protected]? We can get something schedules and I'll take a look at your cluster.

Hello Jesse,

Thanx indeed for the remote login support offer, but unfortunately this is not possible here.

Is there any other way? Send you all server configuration perhaps?

K

Can you please check your command history and verify that when you upgraded, you ran the './upgrade' command? Occasionally people run the './fullinstall' script by mistake, which has the potential to break the installation.

100% I used "./upgrade"
But anyway, it seems that now I managed to successfully migrate my old version of NLS to a new one and keep the IP as long and the backup disk.
Both instances look ok. Cluster status is Green. Indexes visible from both GUIs. Now I'll wait for the backups and run a few queries to check-test the dashboards and the timestamps...
I've noticed that in the latest version we (users) do not need to touch the memory HEAP SIZE anymore. What about the /etc/sysconfig/logstash and the LS_OPEN_FILES parameter?
I also suggest somehow to check and automatically enable and change to the number of instances the
/usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml

On each instance change this:
# discovery.zen.minimum_master_nodes: 1

because it can cause a lot of trouble if it is forgotten.

Thanx

teirekos,

Thank you for the update, I'm glad to hear that your Nagios Log Server cluster is up and running.

# discovery.zen.minimum_master_nodes: 1

We have discussed this internally, and would like your input - what do you think an appropriate setting would be? We understand that this variable is important for cluster reliability, but setting it automatically can unleash several unintended problems. What would work best for you in your environment? Perhaps a quorum of the total amount of nodes in your cluster?

According to my experience as soon as a second instance is added this value must be enabled (otherwise there is no cluster service) and must be larger to 1.
Since multicast is enabled by default, then the above step is compulsory.
But I quorum of the total amount of nodes is smthing that someone may not want.
So perhaps you can get somehow when a second node is added (perhaps license related info) and then automatically set the value of min master nodes to 3 and leave it there for future customization from each admin.

Also you can provide a "start up document" informing users how to tune their NLSs with critical variables in config files (like the min master nodes, the HEAP memory - which you have fixed it - etc.)

Thanx a lot. Pls close the post.

The default setting of the minimum master nodes is as follows:

# discovery.zen.minimum_master_nodes: 1

That is to say there is no default setting - each instance will elect itself master with this default in place. I have spoken with the development team, and the biggest difficulty is that if setting were handled automatically, the following scenario could take place.

1. Two instances in the cluster, the settings is automatically set to 2 (2 instances in the cluster, cluster doesn't elect a master unless there are two instances present).

2. One instance goes down.

3. The other instance automatically sets the value to 1, because that's how many nodes are currently in the cluster - hence electing itself master.


I will bring this in as a feature request, but we must determine some way for the setting to be automatic without resetting dynamically as described above.

Also you can provide a "start up document" informing users how to tune their NLSs with critical variables in config files (like the min master nodes, the HEAP memory - which you have fixed it - etc.)

This is a great suggestion, I'll include it in the feature request.