NLS Cluster Questions - How To

[gimeb]

Currently, I have a single instance of NLS collecting logs from many production Windows and Linux servers. This is same trial instance I have originally installed, configured and applied permanent license for. I am planning to add 3 additional instances and create cluster.

I have reviewed the Webinar on 2015/04/17 ((https://www.youtube.com/watch?v=oQJaHKfaw9c) where on 26min 30sec into presentation Shamas Demoret have mentioned benefits regarding setting up DNS for the cluster (test-cluster.nagios.com) in the Global Settings (vs single instance IP) and use it in either via F5 or in the round robin DNS.

I have the following questions:

1. What are best practices to configure cluster and add instances to it. Please provide step by step instructions. I have reviewed Adding-Another-Instance-to-Nagios-Log-Server-Cluster.pdf and do not know if these are the only steps required. In addition, should I use the same IP address of the original instance (136.133.236.12) while configuring all other instances as additional cluster members?
2. Currently, I have nxlog and rsyslog configured for Windows and Linux servers where the original IP (136.133.236.12) of the NLS single instance is defined. What would be required if I will use cluster DNS name. Do I need to reconfigure all of the production servers and I how do it in Linux (rsyslog).
3. Please let me know how to properly setup cluster DNS so the logs will be distributed either via DNS round robin or hardware load balancer (F5) between instances so we will have a redundancy. As have indicated in the Webinar that F5 load balancer has been successfully accomplished in your environment. I am very much interested to know on how you got this to work with F5 since I saw some articles on support site where people are having issues with F5 load balancer.
4. On another hand, should I just forget about load balancer and keep everything pointing to the same original IP. This way I do not have to change any of the log configuration settings.
5. What would be your recommendation for memory, CPU and disk allocation for the new cluster instances? Should they be configured as the original one (4GB, 2CPU and 60GB)

[mcapra]

1. What are best practices to configure cluster and add instances to it. Please provide step by step instructions. I have reviewed Adding-Another-Instance-to-Nagios-Log-Server-Cluster.pdf and do not know if these are the only steps required. In addition, should I use the same IP address of the original instance (136.133.236.12) while configuring all other instances as additional cluster members?

There are no additional steps as far as Nagios Log Server is concerned. There may be additional configuration to set up load balancing or RRDNS, but that document covers all of the NLS related items. When configuring the additional instances, I would suggest using the IP address of the original instance for the sake of consistency. Strictly speaking, the cluster ID is the only thing that really matters assuming the hostname/ipaddress used is already on the cluster.

2. Currently, I have nxlog and rsyslog configured for Windows and Linux servers where the original IP (136.133.236.12) of the NLS single instance is defined. What would be required if I will use cluster DNS name. Do I need to reconfigure all of the production servers and I how do it in Linux (rsyslog).

Before changing any configurations, I would make sure the load balancer (or RRDNS) is functioning correctly. Assuming it is, you would point your rsyslog and nxlog configurations to the address of the load balancer or RRDNS. This would essentially involve replacing your rsyslog/nxlog configuration references to 136.133.236.12 with whatever the logical address of the loadbalancer or RRDNS is. The idea of this is that you are hitting each Nagios Log Server's logstash (the transport layer) in a balanced fashion rather than bombarding one specific machine/instance.

3. Please let me know how to properly setup cluster DNS so the logs will be distributed either via DNS round robin or hardware load balancer (F5) between instances so we will have a redundancy. As have indicated in the Webinar that F5 load balancer has been successfully accomplished in your environment. I am very much interested to know on how you got this to work with F5 since I saw some articles on support site where people are having issues with F5 load balancer.

There's a multitude of ways to address load balancing. Unfortunately we haven't documented any particular process for handling the load balancing between Nagios Log Server instances. Setting up RRDNS in BIND is a fairly well documented process though (though you shouldn't use a NLS instance as the load balancer itself):

http://www.zytrax.com/books/dns/ch9/rr.html
https://www.digitalocean.com/community/ ... ailability

4. On another hand, should I just forget about load balancer and keep everything pointing to the same original IP. This way I do not have to change any of the log configuration settings.

That is one option. The primary purpose of adding a load balancer is to distribute the workload across the transport layer. If you have all of your sources shipping to the same instance, there isn't particularly anything wrong with that. It's just not taking advantage of the other instance's transport layers in any way. The redundancy and distribution of data will still exist though.

5. What would be your recommendation for memory, CPU and disk allocation for the new cluster instances? Should they be configured as the original one (4GB, 2CPU and 60GB)

This is a very difficult thing to pin down since almost no one's logs look the same or come in at the same frequency. Compression will occur at various levels, different filters being applied to different logs may consume more CPU cycles, long-term retention varies across orgs, etc. I will say that provisioning more than 64GB of RAM to a single Nagios Log Server instance is a bad idea:

https://www.elastic.co/guide/en/elastic ... essed_oops

[gimeb]

I have additional questions:
1. Would it be OK to configure cluster with all its members pointing to the same IP address of the original instance and then later on reconfigure to work with F5. Are there any pros/cons with this scenario
2. Would you suggest as starting point to configure all cluster members the same with respect of memory, CPU and disk space
3. Currently, my original instance is on version 1.4.3 and there is a message on Home page that "New version is available". Should I upgrade it first before adding additional cluster members.
4. The original version of NLS was installed from VM template downloaded from Nagios Support site. Please provide the download link so I can start working with other instances and let me know if templates contain the current version of NLS
5. We have purchased 4 instances. I have already installed the first one - this is my current instance. Should I use the same template (refer to my question #4) for the rest (i.e. 3 additional ones) or I need to download 3 separate templates.

[mcapra]

1. Would it be OK to configure cluster with all its members pointing to the same IP address of the original instance and then later on reconfigure to work with F5. Are there any pros/cons with this scenario

Not entirely sure what your meaning is here. Strictly speaking each instance (or cluster member) should have it's own logical IP address. Assuming this is true, there shouldn't be any reconfiguration of NLS itself when adding the F5. You would just define those IP addresses within your F5 (depending on how the system load is monitored).

2. Would you suggest as starting point to configure all cluster members the same with respect of memory, CPU and disk space

Yes, especially if the instances are going to be behind a load balancer. It makes everything a little bit smoother on the back-end.

3. Currently, my original instance is on version 1.4.3 and there is a message on Home page that "New version is available". Should I upgrade it first before adding additional cluster members.

That may be a bug. 1.4.3 is the latest version currently so you should be good to go.

4. The original version of NLS was installed from VM template downloaded from Nagios Support site. Please provide the download link so I can start working with other instances and let me know if templates contain the current version of NLS

Here is the latest OVA running 1.4.3 (the latest version as of this post):
https://assets.nagios.com/downloads/nag ... 4.3-64.ova

5. We have purchased 4 instances. I have already installed the first one - this is my current instance. Should I use the same template (refer to my question #4) for the rest (i.e. 3 additional ones) or I need to download 3 separate templates.

If you're referring to our OVA template for Nagios Log Server, you should be able to re-use that one without any issues. Otherwise if you're unsure what version the OVA is running you can download a fresh one from the link above.

[gimeb]

Let me rephrase my question #1:
1. As per documentation, in order to add a new cluster member 'You will need to have on hand the IP address or host name of an existing instance in the cluster as well as the cluster ID which can be
found under Administration > Cluster Status". Therefore, can I proceed with creating the cluster this way first and then later on update to the cluster DNS and defined it's VIP in F5

On my question #3 a meant to say that my current version is 1.4.0 and it looks like I should upgrade to 1.4.3 first before adding another instances. Please let me know the steps to upgrade

[mcapra]

can I proceed with creating the cluster this way first and then later on update to the cluster DNS and defined it's VIP in F5

That should be fine. There may be one or two additional files that need to be updated depending on the exact configuration, but in most cases that seems fine.

On my question #3 a meant to say that my current version is 1.4.0 and it looks like I should upgrade to 1.4.3 first before adding another instances. Please let me know the steps to upgrade

Here are the steps for manually upgrading your NLS version(s):
https://assets.nagios.com/downloads/nag ... Server.pdf

[gimeb]

Thank you for the information. You have indicated that "There may be one or two additional files that need to be updated depending on the exact configuration". Please let me know what do you mean. Are you referring to adding instances to the cluster or updating cluster DNS

[rkennedy]

Thank you for the information. You have indicated that "There may be one or two additional files that need to be updated depending on the exact configuration". Please let me know what do you mean. Are you referring to adding instances to the cluster or updating cluster DNS

The file that may need to be updated is /usr/local/nagioslogserver/var/cluster_hosts, but this shouldn't matter as long as the machines are listening on their respective IP's as well as the F5 VIP.

[gimeb]

So, let me confirm the steps:
1. I will update my original instance to the latest 1.4.3 version using instructions you have provided in Upgrade-Instructions-For-Nagios-Log-Server.pdf
2. I will use VM template you have provided to created another 3 instances
3. I will add new instances to the cluster.

The only step I am not clear is #3, i.e. what is required to add these instances to the cluster.

[rkennedy]

So, let me confirm the steps:
1. I will update my original instance to the latest 1.4.3 version using instructions you have provided in Upgrade-Instructions-For-Nagios-Log-Server.pdf
2. I will use VM template you have provided to created another 3 instances
3. I will add new instances to the cluster.

The only step I am not clear is #3, i.e. what is required to add these instances to the cluster.

You will just need your cluster ID, and the IP or hostname of the master machine.