Sigh, the day has come...i mapping and indexing questions

[benhank]

Sorry fellas, I put it off for as long as I could, but ole benhank has questions about complicated things.
BUT before I continue, at least nothing is broken. =) So all of you fellas on the support staff who read that and reached for the headache medicine relax =D
Here we go:
A discussion arose in my office when I mentioned that thru using filters and mapping (two thing I know next to nothing about) we can significantly decrease the amount of space on our NLS. It was proposed to me that and that NLS "should" (this is an IT world should) does this for you.
So here we are about to embark on a wonderful journey of discovery and enlightenment( <---said in the voice of Deadpoool)
As I understand it the elk stack works like so:
Logs are sent to NLS and logstash is ready and waiting for them. Logstash, that cute and cuddly eager beaver is listening for data on the ports set in

Code: Select all

Configure/global/global config/INPUTS

for data on the ports and then based on the filters set up in

Code: Select all

Configure/global/global config/filters

prepares and sends the data to elasticsearch to file away in its indiana jonesesq warehouse to be retrieved by "Top Men" later:

index.jpg

point of contention
I said "Since,I have not set up any special filters or mappings, all of the syslogs, windows event logs etc, are in their raw formats,and spaces in a log are considered characters which count towards file size. This can be overcome thru mapping and filters.
I was then told that No..NLS compresses and decompresses the data, removing and adding the white spaces for displaying the info after a search.
So please help me how does it work?

[scottwilkerson]

The short answer is it's complicated.

They aren't stored in just their raw form, they are stored in an index made up of 5 lucene shards per index which add a little overhead because they allow for the fast querying over billions of documents very fast.

However there is some compression going on as well.

BUT, then for high availability on multiple instance Log Server clusters, and redundancy there is a replica copy of each shard placed on a different server in the cluster, this increases the data but adds redundancy in case of failure.

Finally, for long tern storage, you should be using the mapped backup repository which then is not replicated and when you reach the expire date your indexes can automatically be closed and then deleted, knowing that you can restore them from the repository at any time in the future if you need to.

[benhank]

OK thanks scott, I appreciate the answer. ELK is powerful and NLS makes it easier to work with. one question tho, do white spaces count as characters and there fore increase file size?

[scottwilkerson]

OK thanks scott, I appreciate the answer. ELK is powerful and NLS makes it easier to work with. one question tho, do white spaces count as characters and there fore increase file size?

Honestly I do not know, you would have to ask elastic....

[benhank]

ok thanks! you can lock it

[scottwilkerson]

ok thanks! you can lock it

Locking