log server storage question

[benhank]

hey guys is there a measurement we can use to see how much disk space we need if we save 3 months of data for 1,000 window servers?
We are about to add them to our current environment and need to figure out how much disk space we may need.

[cdienger]

This can vary depending on the level of logging, activity, etc... on the Windows machines. The advice I give is to configure a few sources to send data to NLS, let it run a few days(the longer the better) and check the size of the indices(Administration > Cluster Status > Indices) to get a rough idea of "typical" log volume, and then multiple accordingly. That said, I'll give this a go in my lab and let you know what kind of numbers I find. I'm out until Tuesday after today so I'll plan to follow up then. In the meantime maybe someone else can chime in with real world numbers

[benhank]

The issue I have is that I cant add servers to NLS, until we get the numbers I am talking about lol.talk about catch 22 eh lol.
I was hoping you guys would already have a few in your dev environment and could pull the info from there.
Is there a way that NLS could show me data/storage usage by device or device type?

[cdienger]

Hey Ben,

Here are the results from my lab test with a couple of lab machines:

logstash-2017.08.01 17,482 events 31.8MB
logstash-2017.07.31 24,119 events 21.2MB
logstash-2017.07.30 24,252 events 21.3MB
logstash-2017.07.29 24,050 events 21.3MB
logstash-2017.07.28 24,766 events 22.1MB

Luckily dev was able to provide a larger sample. See attached. It is the output of the command:

Code: Select all

curl -XGET 'http://localhost:9200/_stats?pretty'

Of interest is the primary_size_in_bytes and num_docs. For example:

Code: Select all

       "logstash-2017.07.26": {
            "index": {
                "primary_size_in_bytes": 1649725355,
                "size_in_bytes": 3297816935
            },
            "translog": {
                "operations": 0
            },
            "docs": {
                "num_docs": 1976744,
                "max_doc": 1976744,
                "deleted_docs": 0

The index size is ~1.6gb for ~2million events from 37 Windows machines. The size_in_bytes is the primary size plus a replicate for redundancy. https://www.youtube.com/watch?v=wORXARN ... u.be&t=280 illustrates how multiple instance of NLS will distribute shards between themselves.

[benhank]

THANKS MAN! THANKS!

[cdienger]

Let us know if you have any further questions.