Index Templates, Mappings and Dynamic models

[ssoliveira]

Hello,

We are working with nxlog to send Windows Performance Counter logs to NLS.

However, nxlog does not support decimal, double, float, etc. data.

To get around the problem; I created a filter in "Global Configuration Filter"; for data converter.

However; I have many demands for data conversion; And I always need to restart logstash for the filter to work. Generating downtime.

I am studying a use of "Mappings, dynamic models"; To create templates;

The goal is to define data models, so that the data is already inserted into the elasticsearch in the correct format. New indexes, created automatically every day.

Do you send me an example of how to configure these templates?

The data you are converting is as follows.

Code: Select all

...
    mutate {
        convert => { 
            "DiskCurrentQueueLength" => "float"
            "DiskPercentDiskTime" => "float"
            "DiskAvgQueueLength" => "float"
            "DiskPercentReadTime" => "float"
            "DiskAvgReadQueueLength" => "float"
            "DiskPercentWriteTime" => "float"
            "DiskAvgWriteQueueLength" => "float"
            "DiskAvgSecPerTransfer" => "float"
            "DiskAvgSecPerRead" => "float"
            "DiskAvgSecPerWrite" => "float"
            "DiskTransfersPerSec" => "float"
            "DiskReadsPerSec" => "float"
            "DiskWritesPerSec" => "float"
            "DiskBytesPerSec" => "float"
            "DiskReadBytesPerSec" => "float"
            "DiskWriteBytesPerSec" => "float"
            "DiskAvgBytesPerTransfer" => "float"
            "DiskAvgBytesPerRead" => "float"
            "DiskAvgBytesPerWrite" => "float"
            "DiskPercentIdleTime" => "float"
            "DiskSplitIOPerSec" => "float"
        }
    }
...

[scottwilkerson]

We do not have a doc that outlines this type of customization, however I did come across how to use logstash to create an existing mapping that your system can then set as the default mapping.

https://www.elastic.co/blog/logstash_le ... ch_mapping

I would recommend dialing this in on a development server before moving to production

[ssoliveira]

Thank you; I can do the creation of the templates, you can close this ticket.

Code: Select all

curl -XDELETE 'localhost:9200/_template/physicaldisk'

curl -XPUT 'localhost:9200/_template/physicaldisk?pretty' -H 'Content-Type: application/json' -d'
{
  "template": "logstash-*",
  "mappings": {
    "physicaldisk": {
      "_source": {
        "enabled": true
      },
      "properties": {
        "DiskLogTime" : {
          "type" : "string"
        },
        "DiskAvgBytesPerRead" : {
          "type" : "double"
        },
        "DiskAvgBytesPerTransfer" : {
          "type" : "double"
        },
        "DiskAvgBytesPerWrite" : {
          "type" : "double"
        },
        "DiskAvgQueueLength" : {
          "type" : "double"
        },
        "DiskAvgReadQueueLength" : {
          "type" : "double"
        },
        "DiskAvgSecPerRead" : {
          "type" : "double"
        },
        "DiskAvgSecPerTransfer" : {
          "type" : "double"
        },
        "DiskAvgSecPerWrite" : {
          "type" : "double"
        },
        "DiskAvgWriteQueueLength" : {
          "type" : "double"
        },
        "DiskBytesPerSec" : {
          "type" : "double"
        },
        "DiskCurrentQueueLength" : {
          "type" : "double"
        },
        "DiskPercentTime" : {
          "type" : "double"
        },
        "DiskPercentIdleTime" : {
          "type" : "double"
        },
        "DiskPercentReadTime" : {
          "type" : "double"
        },
        "DiskPercentWriteTime" : {
          "type" : "double"
        },
        "DiskReadBytesPerSec" : {
          "type" : "double"
        },
        "DiskReadsPerSec" : {
          "type" : "double"
        },
        "DiskSplitIOPerSec" : {
          "type" : "double"
        },
        "DiskTransfersPerSec" : {
          "type" : "double"
        },
        "DiskWriteBytesPerSec" : {
          "type" : "double"
        },
        "DiskWritesPerSec" : {
          "type" : "double"
        }
      }
    }
  }
}
'