Nagios Support Forum

Have I been helpful? Nominate me for a Nagios MVP award!

tmcdonald wrote:

eloyd wrote:...and restart the appropriate services across the cluster in an orderly fashion.

An important clarification.

However this orderly fashion still leaves the cluster unresponsive for a few minutes.
It really should wait for the first logstash start responding again before it restarts the next node.

If you watch the status overview you will notice logstash on all nodes becoming red at the same time for a few minutes.

Sounds like a great feature request - rolling restarts with configurable delays between restarts across the cluster.

While I see the advantage of a delayed start, so that you can ensure one logstash is running. It raises a question in my mind though - how is your load balancer determining which service to forward packets to?

Say the load balancer sends data in a round robin fashion to 1,2,3 - 1 is starting, while 2,3 are down still, won't your load balancer send the data to a closed port? Each load balancer application is going to handle this differently. Some might recognize that the ports closed and forward elsewhere with probes, where others wont.

The other issue is, by doing a delayed response, is if the load balancer isn't detecting when one is down, it leads the whole cluster being down for much longer time. Say in synchronous it takes 1 minute for each logstash to restart, where as in an asynchronous environment, this would take 3 minutes.

I'm all for adding functionality to NLS, but I'm not sure this would be good.

rkennedy wrote:While I see the advantage of a delayed start, so that you can ensure one logstash is running. It raises a question in my mind though - how is your load balancer determining which service to forward packets to?

Say the load balancer sends data in a round robin fashion to 1,2,3 - 1 is starting, while 2,3 are down still, won't your load balancer send the data to a closed port? Each load balancer application is going to handle this differently. Some might recognize that the ports closed and forward elsewhere with probes, where others wont.

The other issue is, by doing a delayed response, is if the load balancer isn't detecting when one is down, it leads the whole cluster being down for much longer time. Say in synchronous it takes 1 minute for each logstash to restart, where as in an asynchronous environment, this would take 3 minutes.

I'm all for adding functionality to NLS, but I'm not sure this would be good.

I vote for an option to choose async and let us define the delay between nodes. If this is not a possible solution I really would like to have a method to save the new configuration from the UI without restarting logstash. Then we could manually restart logstash on each node. We are using HAproxy.

I've created Feature Request 8526. This is a feature request to have the ability to push the configuration to the flat files without forcing a restart. Please note that a feature request is not a guarantee. I do see a good use care for this though.