Fields with underscore

[WillemDH]

Hello,

I have some general non urgent questions:
1) Could someone please explain me how the fields with an underscore are generated. Can we safely delete these fields? I have for example a '_type' and a 'type' field. Also the_id and _index field. Are they required for Logstash to work?
2) What about the fields with an @ in front of it? SUch as @version, can I delete that field?
2) I seem to also have a highlight field for each log. What purpose does this field has? Can I delete this field?

Grtz

Willem

[jolson]

1) Could someone please explain me how the fields with an underscore are generated. Can we safely delete these fields? I have for example a '_type' and a 'type' field. Also the_id and _index field. Are they required for Logstash to work?

These are known as 'default fields', the definitions can be found on the right-hand side of this page: https://www.elastic.co/guide/en/elastic ... ields.html

The default fields control how document metadata is stored - these fields are controlled by the index mapping.

In general, it's best to leave those fields alone if you don't have a particular need to remove them. The idea here is that '_type' is generated by elasticsearch, where 'type' is generated by a logstash input/filter.

What about the fields with an @ in front of it? SUch as @version, can I delete that field?

Those fields are generated by logstash to keep tabs on logs. The best explanation I could find is as follows:

As for "Why two timestamps?" Probably several reasons. First, history. ES added _timestamp in August 2011 (commit bb02f19f886c47065cab80fa2787774e25219b6c in elasticsearch) where logstash has had @timestamp since since October 2010 (commit 2aba74e2f4fd94ada7bc2078b1e3187bd29b1332 in logstash)
Second, semantically, the @timestamp is a *part* of the data itself, or in ES terms, the document. However, the _timestamp field in ES is more analogous to metadata than it is data.
Third, in ES, _timestamp is optional and disabled by default. In logstash, @timestamp is required.

There used to be several more @fields generated by logstash, but now only two remain - @timestamp and @version. I recommend leaving them alone. Removing either of them will give you problems.
You can view the change that remove most @ fields here: https://logstash.jira.com/browse/LOGSTASH-675

Most minimal schema will be two fields: timestamp and version. All other values are optional.

2) I seem to also have a highlight field for each log. What purpose does this field has? Can I delete this field?

The highlight field is generated on-demand when you issue a query. For example, here is a screenshot of me viewing all of my logs:

2015-09-15 13_53_13-Cortana.png

Here is another after I perform a query:

2015-09-15 13_54_00-Cortana.png

You can see why the highlight field is important - it allows you to search your logs and highlight particular values when they appear.

In essence, I would leave anything with an ampersand or underscore alone (they're used internally by elasticsearch and logstash respectively) - and I would also be sure to leave the highlight field alone.

[WillemDH]

Thanks for the explanations.
I'll leave them all alone :p

Please close!

[jolson]

No problem - will do!