General questions: load balancing, restoring and security

[batzos]

I have a few questions to close the evaluation of NLS.
1. In a cluster of e.g. 4 instances (without having a load balancer) the rsyslog configuration of the source servers points to the IP and port of the primary instance. If this server is offline, how are the logs redirected to the other instances? Is there a mechanism from the cluster or we have to put a load balancer in front of it to achieve it?

2. When we restore an amount of logs from the backup repository in order to make an investigation these logs are again distributed to all instances in shards, primary and replicas? Do they pass initially from the primary instance? I am asking you because if we have to restore a big volume of logs, we have to check if the instances have enough free storage space, right? E.g. I want to restore 10 TBs of logs. We have 4 instances and each instance has 6 TBs of free storage space. Is it possible to restore them or we need at least 10 TBs in the primary instance (or even 20 TBs including the replicas)?

3. Security issue. Apart from the user control access from the interface with username and password do we have to be concerned by other ways accessing the data? E.g. from the CLI or from the browser to a specific port which is not protected?

4. The shards are distributed between the instances no matter if a load balancer is put in place, right? In that case the traffic passes initially from the primary instance? When we put a load balancer, the traffic passes through the primary instance or it goes directly to the instance chosen?

Thank you in advance.

[eloyd]

1. rsyslog buffers when it cannot reach the remote host. Load balancer or other DNS manipulation will solve the unreachable host problem.

2. pass. Don't have enough experience with restore operations to know the answer.

3. If someone has access to the CLI, then they have access to the data as well. Or at least, they potentially do. If they are (or can become) the nagios user or root, they do. The apache user may have access to some of the web files, which means that they may have the ability to alter them and inject malware style things. Also, there is a RESTful API via port 80 (or 443) that allows queries and stuff to be performed without logging into the GUI, but that requires an API key. So unencrypted network traffic may be sniffed and the API key may be obtained for further exploits.

4. Shards are distributed regardless of load balancer, yes. load balancer only affects (or should, if configured properly) a service address, not inter-node communications. They will handle shard replication amongst themselves regardless of the load balancer. In other words, adding a load balancer should have no effect on shard replication; the nodes talk directly to each other.

[rkennedy]

1. @eloyd is right. Any cast should work as well, or a HA DNS setup.

2. Yes, the instances will need enough hard drive space, as well as ram. When you open the logs, they will be stored on your drives, and cached to your ram for to allow you to search through them. I would advise against opening all of your logs, and specifically drill down to the shard corresponding to the day you're looking to open. This way, you don't over load your ram. One thing to note, is opening 10TB of logs will probably crash just about any server once they get into ram. It just can't handle 10TB open at once.

3. @eloyd is right once again here, but this is mostly user control as you mentioned. Make sure to encrypt your logs if it's sensitive. I wrote this document that describes how to setup SSL with NXLog, https://assets.nagios.com/downloads/nag ... th-SSL.pdf. Towards the bottom you can see the clear difference between un-encrypted vs encrypted.

4. v

The shards are distributed between the instances no matter if a load balancer is put in place, right?

Yes.

In that case the traffic passes initially from the primary instance?

I'm unsure what you're referring to when you mention 'primary instance'.

When we put a load balancer, the traffic passes through the primary instance or it goes directly to the instance chosen?

This is dependent on how your load balancer is setup. You could do round robin, geo-location, HA, etc. A simple round robin should work fine though, as it'll distribute everything between them all.

[batzos]

Thank you guys for your immediate responses.
When I say primary, I mean the instance whose IP is used in the rsyslog or nxlog config.
As I understood, all instances have the same role in cluster. There is no instance that is the "brain" of the cluster. How and where can you avoid "split brain"? What do I need to do the configuration? Do I do it in the load balancer?

[rkennedy]

No problem. I pasted #2 from the same response I gave to sales, so if you have anything further to ask, feel free.

Got it, I would put the load balancer in front of all of the NLS servers, and then use that to receive all of your data (so config as the output for rsyslog or nxlog).

A load balancer would help avoid a split brain by distributing things a bit further, but it won't help if a server gets overloaded. There are a few things that can cause the split brain. See this link -
http://blog.trifork.com/2013/10/24/how- ... ticsearch/

[eloyd]

Actually, there is a master. Read https://www.elastic.co/guide/en/elastic ... uster.html for more details. Also, if the mater fails, another one becomes the master. This is detailed in https://www.elastic.co/guide/en/elastic ... ilure.html.

[rkennedy]

True, thanks @eloyd!

@batzos - let us know if you have any further questions.