1) Could someone please explain me how the fields with an underscore are generated. Can we safely delete these fields? I have for example a '_type' and a 'type' field. Also the_id and _index field. Are they required for Logstash to work?
These are known as 'default fields', the definitions can be found on the right-hand side of this page:
https://www.elastic.co/guide/en/elastic ... ields.html
The default fields control how document metadata is stored - these fields are controlled by the
index mapping.
In general, it's best to leave those fields alone if you don't have a particular need to remove them. The idea here is that '_type' is generated by elasticsearch, where 'type' is generated by a logstash input/filter.
What about the fields with an @ in front of it? SUch as @version, can I delete that field?
Those fields are generated by logstash to keep tabs on logs. The best explanation I could find is as follows:
As for "Why two timestamps?" Probably several reasons. First, history. ES added _timestamp in August 2011 (commit bb02f19f886c47065cab80fa2787774e25219b6c in elasticsearch) where logstash has had @timestamp since since October 2010 (commit 2aba74e2f4fd94ada7bc2078b1e3187bd29b1332 in logstash)
Second, semantically, the @timestamp is a *part* of the data itself, or in ES terms, the document. However, the _timestamp field in ES is more analogous to metadata than it is data.
Third, in ES, _timestamp is optional and disabled by default. In logstash, @timestamp is required.
There used to be several more @fields generated by logstash, but now only two remain - @timestamp and @version. I recommend leaving them alone. Removing either of them will give you problems.
You can view the change that remove most @ fields here:
https://logstash.jira.com/browse/LOGSTASH-675
Most minimal schema will be two fields: timestamp and version. All other values are optional.
2) I seem to also have a highlight field for each log. What purpose does this field has? Can I delete this field?
The highlight field is generated on-demand when you issue a query. For example, here is a screenshot of me viewing all of my logs:
2015-09-15 13_53_13-Cortana.png
Here is another after I perform a query:
2015-09-15 13_54_00-Cortana.png
You can see why the highlight field is important - it allows you to search your logs and highlight particular values when they appear.
In essence, I would leave anything with an ampersand or underscore alone (they're used internally by elasticsearch and logstash respectively) - and I would also be sure to leave the highlight field alone.
