Cluster IP or Load balance or ?

[jolson]

Sounds good - let us know what you find out. Thanks!

[tylerhoadley]

I'm in the process of testing nginx udp/tcp load balancing with keepalived vrrp. seems to fit the requirements for udp/tcp load balancing. Configuration wise seems straight forward too.

https://www.nginx.com/blog/announcing-u ... balancing/

Cheers

[hsmith]

Thank you for the information, and for the link!

[tylerhoadley]

Quick update. Nginx has been very successful in providing even load between log source inputs and https requests. At the moment I have the services running locally on the 2 nagios log server systems, and it isn't as simple as it could be if the LB were on dedicated hosts because of the 0.0.0.0 service bindings. So each host requires some tweaking to ensure elastic-search and Apache are bond to the dedicate host IP, and that nginx is listening and forwarding correct on the VIP (if not you will have port bindings already in use). One other hurdle I had early in this process is that nginx is not transparent like HAProxy could be. This of course was mitigated via a top level grok replacing logsource/Hostname. Also because of binding issues, you have to convert logstash global inputs into (advanced) per node inputs where I bind the iinput listeners to the local IP. Once again, could be even simpler with dedicate nginx LB nodes. I will be moving forward with this setup with more than just Unix/Linux/Windows hosts, and in the event I can not convert a Unique name/ip to Host field, will duplicate entries to both nodes but for the most part, believe this will take care of the majority of nodes in my environment.

oh and finally... because nginx (community) does not include health checks (only Plus) there is a small gap if you restart elasticsearch and get the waiting for message. this is because nginx sees apache still listening although the main site isn't completely available. (I have 2 dedicate physical nodes with lots of power so elastic search takes only a couple shorts seconds to catch up so it isn't a big concern at this moment in time)

Code: Select all

if [type] == 'syslog' { 
  if [logsource] =~ /.+/ {
    mutate {
      replace => [ 'host', "%{logsource}" ]
    }
  }
}
if [type] == 'eventlog' { 
  if [Hostname] =~ /.+/ {
    mutate {
      replace => [ 'host', "%{Hostname}" ]
    }
  }
}

Cheers,

[hsmith]

We appreciate you reporting on the progress

[tylerhoadley]

Been a while since I set this up, so thought I would post an update.

Since this, nginx has released a Transparent proxy version (nginx-plus-r10) in which I have setup, so I can remove the grok rewrites (older syslog on AIX and solaris can't be grok'd the same way as rsyslog). I have also setup 2 new dedicated LB servers to handle all requests so that I can remove the per system input configurations and make them global again.

I found that when I did upgrades with the VIP, elastricsearch would always bind to it. so either you do it while its in backup mode or once the upgrade is done, stop elastricsearch, edit '/usr/local/nagioslogserver/var/cluster_hosts' and start it back up.

Cheers,

[dwhitfield]

Thanks for the update! Let us know if you end up with any questions about the new setup!