From:
https://www.elastic.co/guide/en/logstas ... cture.html
If you specify multiple filters, they are applied in the order of their appearance in the configuration file.
So you're correct in terms of how the process is done in Logstash. On the Nagios Log Server end of things, we stage configurations within the elasticsearch database like so:
Code: Select all
"config_filters": [
{
"raw": "if [program] == 'apache_access' {\r\n grok {\r\n match => [ 'message', '%{COMBINEDAPACHELOG}']\r\n }\r\n date {\r\n match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]\r\n }\r\n mutate {\r\n replace => [ 'type', 'apache_access' ]\r\n convert => [ 'bytes', 'integer' ]\r\n convert => [ 'response', 'integer' ]\r\n }\r\n}\r\n \r\nif [program] == 'apache_error' {\r\n grok {\r\n match => [ 'message', '\\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\\] \\[%{WORD:class}\\] \\[%{WORD:originator} %{IP:clientip}\\] %{GREEDYDATA:errmsg}']\r\n }\r\n mutate {\r\n replace => [ 'type', 'apache_error' ]\r\n }\r\n}",
"name": "Apache (Default)",
"active": "1"
},
{
"raw": "if [type] == 'import_raw' {\r\n if ([message] =~ /^[0-9]{4}-[0-9]{2}-[0-9]{2};/) {\r\n csv {\r\n\t\t\tseparator => \";\"\r\n\t\t\tcolumns => [\"date\",\"time\",\"cs-uri-stem\",\"cs-uri-query\",\"s-contentpath\",\"sc-status\",\"s-computername\",\"cs(Referer)\",\"sc-win32-status\",\"sc-bytes\",\"cs-bytes\",\"cs-username\",\"cs(User-Agent)\",\"time-local\",\"TimeTakenMS\",\"sc-substatus\",\"s-sitename\",\"s-ip\",\"s-port\",\"s-proxy\",\"cs-version\",\"c-protocol\",\"originalip\",\"cs-method\",\"cs(Host)\",\"EndRequest-UTC\",\"date-local\",\"cs(Cookie)\",\"c-ip\",\"BeginRequest-UTC\",\"X-Forwarded-For\",\"Incap-Client-Ip\",\"coolv-XIP\"]\r\n\t\t\t}\r\n\t\t\tmutate {\r\n\t\t\tremove_field => [\"message\",\"date-local\"]\r\n\t\t\tgsub => [\r\n\t\t\t\"type\", \"eventlog\", \"il2\",\r\n\t\t\t\"_type\", \"eventlog\", \"il2\" \r\n\t\t\t]\r\n\t\t\t}\r\n\t\t\tif \"_csvparsefailure\" in [tags] {\r\n\t\t\tdrop { }\r\n\t\t\t}\r\n\t\t\tif !(\"IL2WEB\" in [s-computername]) {\r\n\t\t\tdrop { }\r\n\t\t }\r\n }\r\n\t\telse {drop {}}\r\n }",
"name": "get vm name",
"active": "1"
},
{
"raw": "if [type] == 'eventlog' {\r\n if [Channel] == 'Security' {\r\n drop{}\r\n }\r\n}",
"name": "drop security",
"active": "0"
}
],
If I were to, from the GUI, move the "drop security" filter up one level then the corresponding
config_filters array changes accordingly:
Code: Select all
"config_filters": [
{
"raw": "if [program] == 'apache_access' {\r\n grok {\r\n match => [ 'message', '%{COMBINEDAPACHELOG}']\r\n }\r\n date {\r\n match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]\r\n }\r\n mutate {\r\n replace => [ 'type', 'apache_access' ]\r\n convert => [ 'bytes', 'integer' ]\r\n convert => [ 'response', 'integer' ]\r\n }\r\n}\r\n \r\nif [program] == 'apache_error' {\r\n grok {\r\n match => [ 'message', '\\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\\] \\[%{WORD:class}\\] \\[%{WORD:originator} %{IP:clientip}\\] %{GREEDYDATA:errmsg}']\r\n }\r\n mutate {\r\n replace => [ 'type', 'apache_error' ]\r\n }\r\n}",
"name": "Apache (Default)",
"active": "1"
},
{
"raw": "if [type] == 'eventlog' {\r\n if [Channel] == 'Security' {\r\n drop{}\r\n }\r\n}",
"name": "drop security",
"active": "0"
},
{
"raw": "if [type] == 'import_raw' {\r\n if ([message] =~ /^[0-9]{4}-[0-9]{2}-[0-9]{2};/) {\r\n csv {\r\n\t\t\tseparator => \";\"\r\n\t\t\tcolumns => [\"date\",\"time\",\"cs-uri-stem\",\"cs-uri-query\",\"s-contentpath\",\"sc-status\",\"s-computername\",\"cs(Referer)\",\"sc-win32-status\",\"sc-bytes\",\"cs-bytes\",\"cs-username\",\"cs(User-Agent)\",\"time-local\",\"TimeTakenMS\",\"sc-substatus\",\"s-sitename\",\"s-ip\",\"s-port\",\"s-proxy\",\"cs-version\",\"c-protocol\",\"originalip\",\"cs-method\",\"cs(Host)\",\"EndRequest-UTC\",\"date-local\",\"cs(Cookie)\",\"c-ip\",\"BeginRequest-UTC\",\"X-Forwarded-For\",\"Incap-Client-Ip\",\"coolv-XIP\"]\r\n\t\t\t}\r\n\t\t\tmutate {\r\n\t\t\tremove_field => [\"message\",\"date-local\"]\r\n\t\t\tgsub => [\r\n\t\t\t\"type\", \"eventlog\", \"il2\",\r\n\t\t\t\"_type\", \"eventlog\", \"il2\" \r\n\t\t\t]\r\n\t\t\t}\r\n\t\t\tif \"_csvparsefailure\" in [tags] {\r\n\t\t\tdrop { }\r\n\t\t\t}\r\n\t\t\tif !(\"IL2WEB\" in [s-computername]) {\r\n\t\t\tdrop { }\r\n\t\t }\r\n }\r\n\t\telse {drop {}}\r\n }",
"name": "get vm name",
"active": "1"
}
],
Then when configuration is applied we (in the case of filters) cycle through the entire
config_filters array in order.