Hi,
We are interested in your new product.
We want to you know few things:
1) What is the rate of updating the live dashboard, I mean how fast does it work. Are we talking about seconds until it will be shown on the dashboard or minute+? Considering we are talking about 500GB - 1TB of daily data.
2) Will it be able to deal with 500GB-1TB of data per day? How many instances we will need and what are the hardware requirements that you suggest for such data transmission?
3) Does it stores all the indexed logs locally on the instance machine?
4) Log Server knows how to increase amount of Elastic Search servers(shards) and Logstash servers than traffic grow or we need to do it manually?
Log Server - Abilities
Re: Log Server - Abilities
Let me try to answer these as best I can:
1) Even with that amount of data the dashboard will take between 1-5 seconds to display. There are many things that will impact this including the number of instances you have in your cluster, the number of filters that are being applied to the dashboard and the number of panels you might have in a customized dashboard.
2) A short answer for this is yes, our load testing for large data looked very good. I asked about this and a first look estimate with dealing with that amount of data is at least 5 Instances. When you start adding sophisticated filters to the dashboard with the number of logs you are planning on dealing with the back end will have to look through all the logs to draw the correct dashboard. So depending on many variables you could need up to 10 instances to handle a large amount of data and be able to timely query, filter and display it.
3) The data is broken out over a cluster and all instances share the load and data. There are also backups of each shard of data so that when one instance goes down the backups on the other instances will circumvent the downed server and continue to collect and display logs without it.
Here are a few good resource for understanding clusters and instances of Nagios Log Server: Nagios Log Server – Managing Clusters, Nagios Log Server - Managing Instances
4) This is similar to parts the documentation above will explain. There is also a presentation from the Nagios World Conference by our Development Manager Scott Wilkerson explaining how Nagios Log Server works and that will be available on YouTube as soon as it is finished. It has a nice video in it that explains how the shards distribute the work load and data amongst the available Instances.
I hope these are the answers you are looking for, but please don't hesitate to ask for any information you need. The Documentation is also a good place for information and there are other parts of it that you can browse through.
/Luke
1) Even with that amount of data the dashboard will take between 1-5 seconds to display. There are many things that will impact this including the number of instances you have in your cluster, the number of filters that are being applied to the dashboard and the number of panels you might have in a customized dashboard.
2) A short answer for this is yes, our load testing for large data looked very good. I asked about this and a first look estimate with dealing with that amount of data is at least 5 Instances. When you start adding sophisticated filters to the dashboard with the number of logs you are planning on dealing with the back end will have to look through all the logs to draw the correct dashboard. So depending on many variables you could need up to 10 instances to handle a large amount of data and be able to timely query, filter and display it.
3) The data is broken out over a cluster and all instances share the load and data. There are also backups of each shard of data so that when one instance goes down the backups on the other instances will circumvent the downed server and continue to collect and display logs without it.
Here are a few good resource for understanding clusters and instances of Nagios Log Server: Nagios Log Server – Managing Clusters, Nagios Log Server - Managing Instances
4) This is similar to parts the documentation above will explain. There is also a presentation from the Nagios World Conference by our Development Manager Scott Wilkerson explaining how Nagios Log Server works and that will be available on YouTube as soon as it is finished. It has a nice video in it that explains how the shards distribute the work load and data amongst the available Instances.
I hope these are the answers you are looking for, but please don't hesitate to ask for any information you need. The Documentation is also a good place for information and there are other parts of it that you can browse through.
/Luke
/Luke
Re: Log Server - Abilities
Thanks for the reply, I just didn't understand your answer for my 3rd question, how is the data being stored exactly - it is stored on each of the instances? means I will need a storage of 15TB if I want to keep data of 30 days 
Am I right?
Am I right?-
scottwilkerson
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Log Server - Abilities
There are 5 primary and 5 replica shards for each index. A new index is created every day.
If you have 2 instances, all of the data is replicated, however once you have 3+ instances, the data is distributed between the 3 making sure you still have one primary and one replica of each shard.
If you have 2 instances, all of the data is replicated, however once you have 3+ instances, the data is distributed between the 3 making sure you still have one primary and one replica of each shard.
Re: Log Server - Abilities
Scot and I did a quick run over the numbers and this what we came up with:
1 TB/ day primary shard
1 TB/ day replica shard
x 30 days
----------------
= 60 TB over however many instances you decide is needed.
So, 5 Instances would be 12 TB per instance
10 Instances would be 6 TB per instance
It is also good to note two thing when thinking about storage. You're going to want to give yourself a little leeway in storage so maybe a little extra room for each server. Also, if you are going to import archived logs into your cluster you will not have to calculate room for them because they will be on a file system with separate storage.
Then the process of loading dashboards and applying queries and filters is a question of memory capacity not storage space. So you will want to find how many instances you will need to share the memory workload and then divide the storage between them.
/Luke
1 TB/ day primary shard
1 TB/ day replica shard
x 30 days
----------------
= 60 TB over however many instances you decide is needed.
So, 5 Instances would be 12 TB per instance
10 Instances would be 6 TB per instance
It is also good to note two thing when thinking about storage. You're going to want to give yourself a little leeway in storage so maybe a little extra room for each server. Also, if you are going to import archived logs into your cluster you will not have to calculate room for them because they will be on a file system with separate storage.
Then the process of loading dashboards and applying queries and filters is a question of memory capacity not storage space. So you will want to find how many instances you will need to share the memory workload and then divide the storage between them.
/Luke
/Luke
Re: Log Server - Abilities
Great answer. Please keep this topic open until we decide.
For now we are testing the trial version we want to learn it, see how does it create alerts in Nagios XI etc.
I just don't want to spam your forum with new topics
For now we are testing the trial version we want to learn it, see how does it create alerts in Nagios XI etc.
I just don't want to spam your forum with new topics
Re: Log Server - Abilities
No problem! I'll leave this topic open to keep the discussion going. The trial is 90 days so you will have a lot of time to test all the features you might need.
/Luke