Cluster IP or Load balance or ?

[benhank]

BB this topic is a good one.

[eloyd]

You should still learn Squid.

[jolson]

eloyd,

Could you expand on how squid would be useful here? I wasn't aware that it could be used as a load-balancer. The main capabilities that are of interest are:
1. Retain the source IP of the end device through the load-balancing process.
2. Detect when a node has failed and remove it from the pool of balanced IPs.

[eloyd]

Aside from being a PITA to Bandit, my original "squid" response was to his comment:

I was thinking about using HAProxy or something that I can install on the nodes and use a virtual IP between them. You have something that's simple to install as I have never installed any

Squid makes a perfect load balancer (http://parkersamp.com/2010/11/howto-loa ... -and-squid) and a virtual IP with heartbeat makes a great way to ensure that your squid server (across multiple physical hosts) is always running.

[BanditBBS]

Aside from being a PITA to Bandit, my original "squid" response was to his comment:

I was thinking about using HAProxy or something that I can install on the nodes and use a virtual IP between them. You have something that's simple to install as I have never installed any

Squid makes a perfect load balancer (http://parkersamp.com/2010/11/howto-loa ... -and-squid) and a virtual IP with heartbeat makes a great way to ensure that your squid server (across multiple physical hosts) is always running.

Aww, I feel so special

I didn't read your link besides the header...is says how to load balance "HTTP" can you do other ports as well? (syslog and other stuff)

EDIT: HAHAHA, what a bad typo in my first attempt, the "Aww" was something else completely on accident, I swear, but made me LOL regardless and hurry and change it!

[eloyd]

I guess I should pay more attention.

No, it can't proxy / load-balance syslog data. About the only thing that can do anything like that directly is rsyslog. A decent place to start is http://www.rsyslog.com/load-balancing-for-rsyslog

But the point of logstash is to be able to take the data from anywhere and stick it into a cluster-based data repository. So it sort of defeats the purpose if you bypass it with iptables and rsyslog configs.

[ssax]

BanditBBS, do you have any outstanding questions left on the servers being far apart and from jolson's response on that?

[CFT6Server]

Good thread guys and I am running into the same problem. We are using Cisco ACE which we do not have control of. So although this provided us with NLB on the web interface and log collecting, it basically made it useless since all the source IP is changed. I can write a grok filter to extract the server names from the message, but this doesn't guarantee the information (also it can break easily if the message formats are different, which they are). I am really not sure what to do at this point as HA and making sure we capture all logs is vital and can't afford to have nodes down. With host being down, we could use hundreds of thousands of logs.

[jolson]

CFT6Server,

The most promising way forward is in development by Logstash staff: https://github.com/elastic/logstash/issues/2633
Other than that, I can think of a couple of options:

1. Turn off source-IP re-writing on your load-balancer (probably not feasible).
2. Custom-build HAProxy which might or might not work, it's not tested.
3. Use a grok filter to extract server names
4. Not use a load balancer
5. Settle for lesser balancing of round-robin DNS or similar.

Eventually logstash hopes to support load-balancing on its own, which could solve this problem for the majority of our users. You can bet that I'll push to have this implemented if/when the Logstash team finishes development.

[CFT6Server]

Thanks for the information. Looks like it is in the works, but could be far away. We are currently getting this implemented, so I think the two immediate option is to try HAProxy or point the logs directly at the log server nodes (which I am trying to avoid). I will see if I can get a lab setup together to try out HAProxy.