Page 1 of 1
log server storage question
Posted: Thu Jul 27, 2017 11:28 am
by benhank
hey guys is there a measurement we can use to see how much disk space we need if we save 3 months of data for 1,000 window servers?
We are about to add them to our current environment and need to figure out how much disk space we may need.
Re: log server storage question
Posted: Thu Jul 27, 2017 2:07 pm
by cdienger
This can vary depending on the level of logging, activity, etc... on the Windows machines. The advice I give is to configure a few sources to send data to NLS, let it run a few days(the longer the better) and check the size of the indices(Administration > Cluster Status > Indices) to get a rough idea of "typical" log volume, and then multiple accordingly. That said, I'll give this a go in my lab and let you know what kind of numbers I find. I'm out until Tuesday after today so I'll plan to follow up then. In the meantime maybe someone else can chime in with real world numbers
Re: log server storage question
Posted: Tue Aug 01, 2017 10:59 am
by benhank
The issue I have is that I cant add servers to NLS, until we get the numbers I am talking about lol.talk about catch 22 eh lol.
I was hoping you guys would already have a few in your dev environment and could pull the info from there.
Is there a way that NLS could show me data/storage usage by device or device type?
Re: log server storage question
Posted: Tue Aug 01, 2017 1:33 pm
by cdienger
Hey Ben,
Here are the results from my lab test with a couple of lab machines:
logstash-2017.08.01 17,482 events 31.8MB
logstash-2017.07.31 24,119 events 21.2MB
logstash-2017.07.30 24,252 events 21.3MB
logstash-2017.07.29 24,050 events 21.3MB
logstash-2017.07.28 24,766 events 22.1MB
Luckily dev was able to provide a larger sample. See attached. It is the output of the command:
Code: Select all
curl -XGET 'http://localhost:9200/_stats?pretty'
Of interest is the primary_size_in_bytes and num_docs. For example:
Code: Select all
"logstash-2017.07.26": {
"index": {
"primary_size_in_bytes": 1649725355,
"size_in_bytes": 3297816935
},
"translog": {
"operations": 0
},
"docs": {
"num_docs": 1976744,
"max_doc": 1976744,
"deleted_docs": 0
The index size is ~1.6gb for ~2million events from 37 Windows machines. The size_in_bytes is the primary size plus a replicate for redundancy.
https://www.youtube.com/watch?v=wORXARN ... u.be&t=280 illustrates how multiple instance of NLS will distribute shards between themselves.
Re: log server storage question
Posted: Tue Aug 01, 2017 2:37 pm
by benhank
THANKS MAN! THANKS!
Re: log server storage question
Posted: Tue Aug 01, 2017 2:55 pm
by cdienger
Let us know if you have any further questions.