Applying config changes

[Frodo]

When applying changes to our global configuration in our cluster I have noticed that all logstash instances will become unavailable.
As we have a load balancer in front of the cluster this is really bad. We will loose logs.

What is the supported way to apply config changes without restarting all logstash instances at the same time?

[hsmith]

There is not one, as the logstash configuration will have to update configuration across the whole cluster. What is the interaction you're experiencing with the load balancer? We have a lot of people that have a load balancer in front of NLS, and I've never seen any issues personally.

[eloyd]

It may also depend on how you're sending data to the log server via the load balancer. If using rsyslog, it will queue up messages if they cannot be immediately sent, but it might be your load balancer that's accepting them, but not able to send them on (and not queuing).

[hsmith]

Thanks Eric

[Frodo]

If I do save. Doesn't that distribute the changes on all nodes? If so then I will have to restart each logstash manually one node at a time. We have udp syslog going to this server and they are fire and forget. Some systems can't do anything else

[hsmith]

What do you mean 'If I do save'? Are you talking about running the apply configuration? You shouldn't have to do anything manually if you run that. If you edit the flat configuration files, the next apply configuration will overwrite them. I might be missing what you're describing though.

[Frodo]

To clarify. If I click the "Save" button in "Administration / Configuration Editor / Global Configuration". Will that write down the new configuration without restarting logstash on all nodes? If so then I will have to restart logstash on one node at a time to prevent data loss.

[hsmith]

Saving it will push it to the DB - but it will not push the configuration to the flat config files. Only the Apply Config will push the configs.

[eloyd]

...and restart the appropriate services across the cluster in an orderly fashion.

[tmcdonald]

...and restart the appropriate services across the cluster in an orderly fashion.

An important clarification.