order of filters?

[_asp_]

Hi,

I would like to have some performance monitoring of our logstash parsing, to see if we should optimize some of our logstash filters.
Therefore my idea was the following:
1. call a ruby filter which saves the current time to a field startTime.
2. Do the normal processing of all filters as usual
3. call a ruby filter which gets the current time as endTime and calculates the difference.

I noticed that I can drag and drop the filters in the admin panel. Are the filters processing in the shown order? (top as first, buttom as last).

Thanks, Andreas

[mcapra]

From: https://www.elastic.co/guide/en/logstas ... cture.html

If you specify multiple filters, they are applied in the order of their appearance in the configuration file.

So you're correct in terms of how the process is done in Logstash. On the Nagios Log Server end of things, we stage configurations within the elasticsearch database like so:

Code: Select all

"config_filters": [
            {
              "raw": "if [program] == 'apache_access' {\r\n    grok {\r\n        match => [ 'message', '%{COMBINEDAPACHELOG}']\r\n    }\r\n    date {\r\n        match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]\r\n    }\r\n    mutate {\r\n        replace => [ 'type', 'apache_access' ]\r\n         convert => [ 'bytes', 'integer' ]\r\n         convert => [ 'response', 'integer' ]\r\n    }\r\n}\r\n \r\nif [program] == 'apache_error' {\r\n    grok {\r\n        match => [ 'message', '\\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\\] \\[%{WORD:class}\\] \\[%{WORD:originator} %{IP:clientip}\\] %{GREEDYDATA:errmsg}']\r\n    }\r\n    mutate {\r\n        replace => [ 'type', 'apache_error' ]\r\n    }\r\n}",
              "name": "Apache (Default)",
              "active": "1"
            },
            {
              "raw": "if [type] == 'import_raw' {\r\n        if ([message] =~ /^[0-9]{4}-[0-9]{2}-[0-9]{2};/) {\r\n            csv {\r\n\t\t\tseparator => \";\"\r\n\t\t\tcolumns => [\"date\",\"time\",\"cs-uri-stem\",\"cs-uri-query\",\"s-contentpath\",\"sc-status\",\"s-computername\",\"cs(Referer)\",\"sc-win32-status\",\"sc-bytes\",\"cs-bytes\",\"cs-username\",\"cs(User-Agent)\",\"time-local\",\"TimeTakenMS\",\"sc-substatus\",\"s-sitename\",\"s-ip\",\"s-port\",\"s-proxy\",\"cs-version\",\"c-protocol\",\"originalip\",\"cs-method\",\"cs(Host)\",\"EndRequest-UTC\",\"date-local\",\"cs(Cookie)\",\"c-ip\",\"BeginRequest-UTC\",\"X-Forwarded-For\",\"Incap-Client-Ip\",\"coolv-XIP\"]\r\n\t\t\t}\r\n\t\t\tmutate {\r\n\t\t\tremove_field => [\"message\",\"date-local\"]\r\n\t\t\tgsub => [\r\n\t\t\t\"type\", \"eventlog\", \"il2\",\r\n\t\t\t\"_type\", \"eventlog\", \"il2\" \r\n\t\t\t]\r\n\t\t\t}\r\n\t\t\tif \"_csvparsefailure\" in [tags] {\r\n\t\t\tdrop { }\r\n\t\t\t}\r\n\t\t\tif !(\"IL2WEB\" in [s-computername]) {\r\n\t\t\tdrop { }\r\n\t\t }\r\n        }\r\n\t\telse {drop {}}\r\n    }",
              "name": "get vm name",
              "active": "1"
            },
            {
              "raw": "if [type] == 'eventlog' {\r\n    if [Channel] == 'Security' {\r\n        drop{}\r\n    }\r\n}",
              "name": "drop security",
              "active": "0"
            }
          ],

If I were to, from the GUI, move the "drop security" filter up one level then the corresponding config_filters array changes accordingly:

Code: Select all

"config_filters": [
            {
              "raw": "if [program] == 'apache_access' {\r\n    grok {\r\n        match => [ 'message', '%{COMBINEDAPACHELOG}']\r\n    }\r\n    date {\r\n        match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]\r\n    }\r\n    mutate {\r\n        replace => [ 'type', 'apache_access' ]\r\n         convert => [ 'bytes', 'integer' ]\r\n         convert => [ 'response', 'integer' ]\r\n    }\r\n}\r\n \r\nif [program] == 'apache_error' {\r\n    grok {\r\n        match => [ 'message', '\\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\\] \\[%{WORD:class}\\] \\[%{WORD:originator} %{IP:clientip}\\] %{GREEDYDATA:errmsg}']\r\n    }\r\n    mutate {\r\n        replace => [ 'type', 'apache_error' ]\r\n    }\r\n}",
              "name": "Apache (Default)",
              "active": "1"
            },
            {
              "raw": "if [type] == 'eventlog' {\r\n    if [Channel] == 'Security' {\r\n        drop{}\r\n    }\r\n}",
              "name": "drop security",
              "active": "0"
            },
            {
              "raw": "if [type] == 'import_raw' {\r\n        if ([message] =~ /^[0-9]{4}-[0-9]{2}-[0-9]{2};/) {\r\n            csv {\r\n\t\t\tseparator => \";\"\r\n\t\t\tcolumns => [\"date\",\"time\",\"cs-uri-stem\",\"cs-uri-query\",\"s-contentpath\",\"sc-status\",\"s-computername\",\"cs(Referer)\",\"sc-win32-status\",\"sc-bytes\",\"cs-bytes\",\"cs-username\",\"cs(User-Agent)\",\"time-local\",\"TimeTakenMS\",\"sc-substatus\",\"s-sitename\",\"s-ip\",\"s-port\",\"s-proxy\",\"cs-version\",\"c-protocol\",\"originalip\",\"cs-method\",\"cs(Host)\",\"EndRequest-UTC\",\"date-local\",\"cs(Cookie)\",\"c-ip\",\"BeginRequest-UTC\",\"X-Forwarded-For\",\"Incap-Client-Ip\",\"coolv-XIP\"]\r\n\t\t\t}\r\n\t\t\tmutate {\r\n\t\t\tremove_field => [\"message\",\"date-local\"]\r\n\t\t\tgsub => [\r\n\t\t\t\"type\", \"eventlog\", \"il2\",\r\n\t\t\t\"_type\", \"eventlog\", \"il2\" \r\n\t\t\t]\r\n\t\t\t}\r\n\t\t\tif \"_csvparsefailure\" in [tags] {\r\n\t\t\tdrop { }\r\n\t\t\t}\r\n\t\t\tif !(\"IL2WEB\" in [s-computername]) {\r\n\t\t\tdrop { }\r\n\t\t }\r\n        }\r\n\t\telse {drop {}}\r\n    }",
              "name": "get vm name",
              "active": "1"
            }
          ],

Then when configuration is applied we (in the case of filters) cycle through the entire config_filters array in order.

[_asp_]

thanks, can be closed