No.of required NLS instances

[technosol]

Hi

Suppose I have 90GB size of data input to NLS daily and the log retention period is 120 days. I am planinig to store the data in a fast SAN space.

Per NLS node/instance, recommended by Nagios; Load is 25GB per day. May I know how manay NLS instances required to cater my requirement?

As I understood, 4 no.of instances are enough to handle the daily load. Correct me if I am wrong and please share with me if you have a calculation method to identify no.of required NLS instances for such scenarios .


Thank you.

[cdienger]

It's difficult to say for as it depends on the type of data as well as how many indices you'll have open at a time. In most cases people only need immediate access to a fraction of what could be made available and should be closing/saving older data. It's best to keep open only recent indices and closing older ones as needed. That said, 4 instances is a good starting point and I would take advantage of NLS' free trial to install more if needed to get a good sizing that works for your environment.

For steps to setup archiving older data: https://assets.nagios.com/downloads/nag ... enance.pdf

[technosol]

Hi

Do we have access to all the features in the NLS free trial version?

[cdienger]

Yes, the trial enables all functionality for 60 days.

[technosol]

Hi,
Thank you for the clarifications. Please help me clear the doubts as we need committed to the customer .

We have got a request from a client to store 120 days of logs each day generates 72GB of logs, based on the recommendation from Nagios for a node not to exceed 25GB per day.

We have proposed a 4 Node cluster of NLS to be deployed, we have couple of questions regarding this hope you can help to clarify.

1) Customer wants to know because this will be deployed on the VM and does not want to allocate too much storage, can we go ahead with a 4 node cluster but have a single data source for storage without replicating the data across the multiple nodes and let the VM handle the storage redundancy? is there any documentation which we can referrer as a guideline?

2) What is the recommended storage size and IOPS for a system this big?

3) Is there anything special we need to consider when it comes to storage size of 8TB +

4) Any other technical items or licensing we need to consider?

[scottwilkerson]

1) Customer wants to know because this will be deployed on the VM and does not want to allocate too much storage, can we go ahead with a 4 node cluster but have a single data source for storage without replicating the data across the multiple nodes and let the VM handle the storage redundancy? is there any documentation which we can referrer as a guideline?

I would highly discourage this as it removes the ability for the cluster to heal itself, however if you must you can run the procedure in this document on each of the Instances in the cluster and it will not make replica shards any longer.

https://assets.nagios.com/downloads/nag ... oyment.pdf

2) What is the recommended storage size and IOPS for a system this big?

Storage size depends on on the size you outlined which would be about 8.6TB + I would add a healthy amount of extra space to accommodate the indexes, and burst. I would say at least 15TB to be save, and because of the quantity of data being searched and indexes the amount of IOPS need to be extremely high, I would be looking for > 10,000 in an optimal setup.

3) Is there anything special we need to consider when it comes to storage size of 8TB +

Just disk speed, the faster the better really.

4) Any other technical items or licensing we need to consider?

It is important to reiterate, that there are many factors that we cannot consider like size of each log, number of filters configured on the inputs, burst (do the 72GB come evenly spread out over 24 hours or in bursts), and it may be possible that 4 Instances is under shooting, and the client may need to increase the number of nodes to accommodate the setup.