Page 3 of 4
Re: Redundancy or load balancing on log server
Posted: Fri Nov 18, 2016 3:20 pm
by gormank
The italicized bit says a 2 instance system is resilient but you say otherwise. We're at the point where you're using doublespeak so a more talk is pointless.
Re: Redundancy or load balancing on log server
Posted: Fri Nov 18, 2016 3:34 pm
by tmcdonald
gormank wrote:The italicized bit says a 2 instance system is resilient but you say otherwise.
I actually never said that a 2-instance system is not resilient. What I did say is that it is not
as resilient as a 4-instance system.
"Reliable" is not a black-and-white quality where something either is or is not reliable. It is definitely a sliding scale, and even a single instance can be reliable. "Redundant" or "resilient" are similar, except that you need two instances in order to have any redundancy/resiliency. As you add instances to the cluster, it becomes more reliable as a function of having more redundancy/resiliency. In a 2-instance cluster, if you lose one instance you can still search your logs but if the other goes down as well then the issue needs to be addressed. Adding a third (or fourth, fifth, etc.) instance increases the redundancy/resiliency and therefore reliability.
Re: Redundancy or load balancing on log server
Posted: Fri Nov 18, 2016 5:04 pm
by gormank
Any idea how/if nxlog can be set up to send to a second (or more) host if the first isn't reachable? The only thing I've found is the feature is planned, but not implemented, but that info is old.
Re: Redundancy or load balancing on log server
Posted: Mon Nov 21, 2016 11:36 am
by mcapra
Not within nxlog. NLog has some
pretty good options for native failover, but it's a huge pain to set up on a Windows machine and we haven't done much testing with it.
Re: Redundancy or load balancing on log server
Posted: Tue Jan 03, 2017 9:29 pm
by gormank
Sorry to bring this back from the dead, but...
Nxlog doesn't support IPv6 so its out.
Consider my 2 node cluster, and a load balancer setup. We can assume NLS1 is primary, and NLS2 is failover.
1. Will this work in theory?
2. What are your suggestions for the load balancer to monitor the aliveness of the NLS1?
The LB checks something on hosts to decide if the traffic should be pointed to the other host.
Re: Redundancy or load balancing on log server
Posted: Wed Jan 04, 2017 10:29 am
by rkennedy
No worries.
1. Yes, it should - however, NLS1 wouldn't serve as the primary, and NLS2 as the failover. They would continue to run at the same time, with one copy of your data available on each machine. (primary / replica shards)
2. This is more of a question for the type of load balancer you decide to go with. Rather then a round-robin type of LB, I would use one that monitors for an active connection + routes accordingly.
Re: Redundancy or load balancing on log server
Posted: Wed Jan 04, 2017 11:48 am
by gormank
From an agent (rsyslog sending logs) standpoint one host can be the receiver at any instant in time. Thus to simplify things conceptually, I used primary/secondary.
These are F5s if that helps. We've already eliminated round robin in this discussion by stating primary and secondary.
So what I'm looking for is specific info on what to monitor as in what active connections and what routes to use what you wrote below.
Re: Redundancy or load balancing on log server
Posted: Wed Jan 04, 2017 1:09 pm
by rkennedy
So what I'm looking for is specific info on what to monitor as in what active connections and what routes to use what you wrote below.
Ah - got it. From the load balancer currently - are you able to see if 9300 is open? (this could vary based on your environment)
The two ports I would monitor a connection for are 9300, and then any port that logstash is listening on. (5544 would be a good one if you're still using default ports, but you'll know the answer to this better then I will since it's your environment.)
9300 ensures that Elasticsearch is up, and 5544 (or any input port) assures that Logstash is still running.
Hope this helps point you in the right direction - let me know if you have any further questions about it.
Re: Redundancy or load balancing on log server
Posted: Wed Jan 04, 2017 1:17 pm
by gormank
The environment isn't built. Its not even built in a lab. All I have is NLS running on a VM on my Windows box.
Default ports are planned right now, so 5544 as a receiver for logs is good. ESXi also wants another port by default, as well as maybe Windows logs on yet another.
Monitor the openness of the ports, much like running nmap on them (no the F5 won't use nmap, but I like a way to test manually)?
Re: Redundancy or load balancing on log server
Posted: Wed Jan 04, 2017 1:45 pm
by rkennedy
Then both of the two mentioned should work fine. Nmap will show open if elasticsearch is running, and closed if it's stopped -
Code: Select all
[root@centos6x64 scripts]# nmap 192.168.3.190 -p 9300
Starting Nmap 6.47 ( http://nmap.org ) at 2017-01-04 12:40 CST
Nmap scan report for 192.168.3.190
Host is up (0.00070s latency).
PORT STATE SERVICE
9300/tcp open vrace
MAC Address: 00:50:56:84:D4:01 (VMware)
Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds
[root@centos6x64 scripts]# nmap 192.168.3.190 -p 9300
Starting Nmap 6.47 ( http://nmap.org ) at 2017-01-04 12:40 CST
Nmap scan report for 192.168.3.190
Host is up (0.00091s latency).
PORT STATE SERVICE
9300/tcp closed vrace
MAC Address: 00:50:56:84:D4:01 (VMware)
Same applies for when logstash is stopped / started -
Code: Select all
[root@centos6x64 scripts]# nmap 192.168.67.4 -p 5544
Starting Nmap 6.47 ( http://nmap.org ) at 2017-01-04 12:44 CST
Nmap scan report for 192.168.67.4
Host is up (0.00053s latency).
PORT STATE SERVICE
5544/tcp open unknown
MAC Address: 00:50:56:84:BE:1B (VMware)
Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds
[root@centos6x64 scripts]# nmap 192.168.67.4 -p 5544
Starting Nmap 6.47 ( http://nmap.org ) at 2017-01-04 12:44 CST
Nmap scan report for 192.168.67.4
Host is up (0.00050s latency).
PORT STATE SERVICE
5544/tcp closed unknown
MAC Address: 00:50:56:84:BE:1B (VMware)
Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds
I've used HA proxy before which does some kind of check-alive for determining where to route, but haven't used an F5 before. I imagine it has a similar 'availability' option though, to probe a port to see if it's in an 'OK' state.