Page 1 of 2
Applying config changes
Posted: Tue May 10, 2016 5:09 pm
by Frodo
When applying changes to our global configuration in our cluster I have noticed that all logstash instances will become unavailable.
As we have a load balancer in front of the cluster this is really bad. We will loose logs.
What is the supported way to apply config changes without restarting all logstash instances at the same time?
Re: Applying config changes
Posted: Wed May 11, 2016 9:40 am
by hsmith
There is not one, as the logstash configuration will have to update configuration across the whole cluster. What is the interaction you're experiencing with the load balancer? We have a lot of people that have a load balancer in front of NLS, and I've never seen any issues personally.
Re: Applying config changes
Posted: Wed May 11, 2016 9:58 am
by eloyd
It may also depend on how you're sending data to the log server via the load balancer. If using rsyslog, it will queue up messages if they cannot be immediately sent, but it might be your load balancer that's accepting them, but not able to send them on (and not queuing).
Re: Applying config changes
Posted: Wed May 11, 2016 4:14 pm
by hsmith
Thanks Eric
Re: Applying config changes
Posted: Wed May 11, 2016 4:26 pm
by Frodo
If I do save. Doesn't that distribute the changes on all nodes? If so then I will have to restart each logstash manually one node at a time. We have udp syslog going to this server and they are fire and forget. Some systems can't do anything else
Re: Applying config changes
Posted: Wed May 11, 2016 4:35 pm
by hsmith
What do you mean 'If I do save'? Are you talking about running the apply configuration? You shouldn't have to do anything manually if you run that. If you edit the flat configuration files, the next apply configuration will overwrite them. I might be missing what you're describing though.
Re: Applying config changes
Posted: Thu May 12, 2016 9:28 am
by Frodo
To clarify. If I click the "Save" button in "Administration / Configuration Editor / Global Configuration". Will that write down the new configuration without restarting logstash on all nodes? If so then I will have to restart logstash on one node at a time to prevent data loss.
Re: Applying config changes
Posted: Thu May 12, 2016 10:04 am
by hsmith
Saving it will push it to the DB - but it will not push the configuration to the flat config files. Only the Apply Config will push the configs.
Re: Applying config changes
Posted: Thu May 12, 2016 10:08 am
by eloyd
...and restart the appropriate services across the cluster in an orderly fashion.
Re: Applying config changes
Posted: Thu May 12, 2016 12:59 pm
by tmcdonald
eloyd wrote:...and restart the appropriate services across the cluster in an orderly fashion.
An important clarification.