Discard IP fragment set with more than 24 elements

[rferebee]

Good morning,

We forward syslogs from our Log Server environment to a Symantec Managed Security Services scanning utility and my Information Security Officer would like to know if there is a way to filter out syslogs with the following message:

Discard IP fragment set with more than 24 elements

They seem to be broken syslogs and we do not want to flood the Symantec scanner with them if they don't contain any meaningful data.

Here's what the output looks like in our Global Config (can we filter them out here perhaps?):

if [type] != "syslog" {
udp {
host => '10.135.11.1'
port => 514
}
}

Thank you for your help.

[npolovenko]

@rferebee, You could add a new filter to the LS that looks like this:

Code: Select all

filter {
if [message] =~ "Discard IP fragment set with more than 24 elements
"] { drop{} }
}

Let me know if this works for you.

[rferebee]

Does this go in as a new Filter or a new Output in the configuration section?

[rferebee]

I attempted to add the filter and received the following error after clicking Save & Apply:

The configuration could not be verified. - this was listed for all 3 instances in my cluster

Also, I tried to verify my Global Config and got this:

{:timestamp=>"2019-06-04T13:30:49.871000-0700", :message=>"The given configuration is invalid. Reason: Expected one of #, => at line 212, column 8 (byte 6084) after filter {\n if [program] == 'apache_access' {\n grok {\n match => [ 'message', '%{COMBINEDAPACHELOG}']\n }\n date {\n match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]\n }\n mutate {\n replace => [ 'type', 'apache_access' ]\n convert => [ 'bytes', 'integer' ]\n convert => [ 'response', 'integer' ]\n }\n }\n \n if [program] == 'apache_error' {\n grok {\n match => [ 'message', '\\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\\] \\[%{WORD:class}\\] \\[%{WORD:originator} %{IP:clientip}\\] %{GREEDYDATA:errmsg}']\n }\n mutate {\n replace => [ 'type', 'apache_error' ]\n }\n }\n if [type] == \"syslog\" {\n if \"%ASA-\" in [message] {\n grok {\n match => [\"message\", \"%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:cisco_message}\"]\n }\n \n # Extract fields from the each of the detailed message types\n # The patterns provided below are included in core of LogStash 1.5.1.\n grok {\n \tmatch => [\n \t\t\"cisco_message\", \"%{CISCOFW106001}\",\n \t\t\"cisco_message\", \"%{CISCOFW106006_106007_106010}\",\n \t\t\"cisco_message\", \"%{CISCOFW106014}\",\n \t\t\"cisco_message\", \"%{CISCOFW106015}\",\n \t\t\"cisco_message\", \"%{CISCOFW106021}\",\n \t\t\"cisco_message\", \"%{CISCOFW106023}\",\n \t\t\"cisco_message\", \"%{CISCOFW106100}\",\n \t\t\"cisco_message\", \"%{CISCOFW110002}\",\n \t\t\"cisco_message\", \"%{CISCOFW302010}\",\n \t\t\"cisco_message\", \"%{CISCOFW302013_302014_302015_302016}\",\n \t\t\"cisco_message\", \"%{CISCOFW302020_302021}\",\n \t\t\"cisco_message\", \"%{CISCOFW305011}\",\n \t\t\"cisco_message\", \"%{CISCOFW313001_313004_313008}\",\n \t\t\"cisco_message\", \"%{CISCOFW313005}\",\n \t\t\"cisco_message\", \"%{CISCOFW402117}\",\n \t\t\"cisco_message\", \"%{CISCOFW402119}\",\n \t\t\"cisco_message\", \"%{CISCOFW419001}\",\n \t\t\"cisco_message\", \"%{CISCOFW419002}\",\n \t\t\"cisco_message\", \"%{CISCOFW500004}\",\n \t\t\"cisco_message\", \"%{CISCOFW602303_602304}\",\n \t\t\"cisco_message\", \"%{CISCOFW710001_710002_710003_710005_710006}\",\n \t\t\"cisco_message\", \"%{CISCOFW713172}\",\n \t\t\"cisco_message\", \"%{CISCOFW733100}\"\n \t]\n }\n if [tags] {\n mutate {remove_tag => [_grokparsefailure_sysloginput, _grokparsefailure]}\n }\n }\n }\n \n if [type] == \"iis\" {\n \n grok { \n match => [ \"message\", \"%{DATESTAMP:EventTime} %{WORD:sitename} %{HOSTNAME:computername} %{IP:hostip} %{URIPROTO:method} %{URIPATH:request} (?:%{NOTSPACE:queryparam}|-) %{NUMBER:port} (?:%{WORD:username}|-) %{IP:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:user-agent} (?:%{NOTSPACE:cookie}|-) (?:%{NOTSPACE:referer}|-) (?:%{HOSTNAME:host}|-) %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received} %{NUMBER:bytes-sent} %{NUMBER:time-taken}\" ]\n } \n \n if [tags] {\n mutate {remove_tag => [_jsonparsefailure]}\n }\n date { \n match => [ \"timestamp\", \"yyyy-MM-dd HH:mm:ss\" ]\n } \n \n } \n \n # Filter for Barracuda Web Filter/Web Security Gateway \n # This filter file will do the initial parsing of the log \n # See my github page here for more information: https://github.com/shthead/barracuda-WF-logstash \n \n \n ###### IMPORTANT ###### \n # Before using this filter, please ensure that you correct the host IP below. \n \n \n \n # Set this to the IP of your Barracuda filter. \n # This is set to only process the logs from the Barracuda filter and nothing else. \n \n if [host] == \"10.131.11.64\" { \n \n \n # Set the type field to \"barracuda\" - makes filtering easy. \n mutate { replace => [ \"type\", \"barracuda\" ] } \n \n \n # Match web interface audit logs \n if [message] =~ \"^<\\d+>web\" { \n grok { \n match => { \"message\" => \"^<\\d+>(?<syslog_program>\\w+): \\[\\d+\\.\\d+\\.\\d+\\.\\d+\\] %{GREEDYDATA:syslog_message}\" } \n } \n } \n \n \n # Match access logs \n if [message] =~ \"^<\\d+>(http_scan|barracuda_pqman)\" { \n grok { \n match => { \"message\" => \"^<\\d+>(?<syslog_program>\\w+)\\[(?<syslog_pid>\\d+)\\]: (?<syslog_timestamp>\\d+) %{GREEDYDATA:syslog_message}\" } \n add_field => [ \"received_at\", \"%{@timestamp}\" ] \n } \n \n \n # Fix up date \n date { \n match => [ \"syslog_timestamp\", \"UNIX\" ] \n } \n \n \n # Remove temp date field \n mutate { \n remove_field => [ \"syslog_timestamp\" ] \n } \n } \n \n # Remove unused message fields \n if [syslog_program] =~ \"^\\w+\"{ \n if \"_grokparsefailure\" not in [tags] { \n mutate { \n replace => [ \"message\", \"%{syslog_message}\" ] \n remove_field => [ \"syslog_message\" ] \n } \n } \n } \n \n # Optional - Do a reverse DNS lookup for the Barracuda's IP. You do not need this. \n #dns { \n #reverse => [ \"host\" ] \n #action => \"replace\" \n #} \n if [tags] {\n mutate {remove_tag => [_grokparsefailure_sysloginput, _grokparsefailure]}\n }\n \n } \n filter {\n if ", :level=>:fatal}

[npolovenko]

@rferebee, It'd go into the filter configuration section.
Sorry, you don't need to include the word filter. I also removed one extra ] after the double quote. Please paste this line in the new filter:

Code: Select all

if [message] =~ "Discard IP fragment set with more than 24 elements" { drop{} }

[rferebee]

Yes, that worked. Thank you very much for your assistance.

[scottwilkerson]

Yes, that worked. Thank you very much for your assistance.

great!

Locking thread