Multiple filters

[WillemDH]

Hello,

I've been trying to make an extra filter for our F5 load balancer that only applies if the program equals dcc. But I tried several variations and still don't get it too work...

These are the filters I have atm:

Code: Select all

if [type] == "syslog-f5" {
    grok {     
      match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}" ]
    remove_tag => "_grokparsefailure"
    add_tag => "grokked"
    }   
}

Code: Select all

if [program] == "dcc" {
    grok {     
      match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME2:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}User-Agent: %{GREEDYDATA:useragent}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}", "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME2:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}" ]
}

dcc is one of several possible program that are available in the F5 load balancer. the program field is applied at in the first filter, so I guess NLS should first attempt the first filter and then execute the second filter is the first filter added the program field and the value is dcc.The problem is that only the first filter seems to be applied. I tried putting both filters in once NLS filter, but this also doesn't seem to work.

Any advice how I should handle this?

The logs for dcc look like this by the way:

Code: Select all

<155>Mar 28 13:23:21 slot1/cpf_f5_1_vir_pr err dcc[9206]: 01310033:3: [SECEV] Request blocked, violations: Attack signature detected. HTTP protocol compliance sub violations: N/A. Evasion techniques sub violations: N/A. Web services security sub violations: N/A. Virus name: N/A. Support id: 8375986001652311748, source ip: 40.70.0.8, xff ip: 40.70.0.8, source port: 49949, destination ip: 40.70.1.138, destination port: 80, route_domain: 0, HTTP classifier: /Common/F5_External_1_RAMP_Policy, scheme HTTP, geographic location: , request: www.digipolis.be\r\nUser-Agent: libwww-perl/6.04\r\n>, username: , session_id: 
<155>Mar 28 13:23:20 slot1/cpf_f5_1_vir_pr err dcc[9206]: 01310033:3: [SECEV] Request blocked, violations: Attack signature detected. HTTP protocol compliance sub violations: N/A. Evasion techniques sub violations: N/A. Web services security sub violations: N/A. Virus name: N/A. Support id: 8375986001652311736, source ip: 40.70.0.8, xff ip: 40.70.0.8, source port: 49927, destination ip: 40.70.1.138, destination port: 80, route_domain: 0, HTTP classifier: /Common/F5_External_1_RAMP_Policy, scheme HTTP, geographic location: , request: www.digipolis.be\r\nUser-Agent: libwww-perl/6.04\r\n>, username: , session_id: 
<154>Mar 28 13:31:30 slot1/cpf_f5_1_vir_pr crit dcc[9206]: 01310038:2: [SECEV] Request violations: Illegal parameter,Illegal URL,Illegal file type. HTTP protocol compliance sub violations: N/A. Evasion techniques sub violations: N/A. Web services security sub violations: N/A. Virus name: N/A. Support id: 8375986001652203713, source ip: 109.130.197.61, xff ip: 109.130.197.61, source port: 1601, destination ip: 40.70.1.139, destination port: 80, route_domain: 0, HTTP classifier: /Common/GENSHPPR_class, scheme HTTP, geographic location: , request: http://schem>, username: , session_id: <71f091ce88335c04>
<155>Mar 28 13:30:02 slot1/cpf_f5_1_vir_pr err dcc[9206]: 01310039:3: [SECEV] Request violations: Attack signature detected. HTTP protocol compliance sub violations: N/A. Evasion techniques sub violations: N/A. Web services security sub violations: N/A. Virus name: N/A. Support id: 8375986001652316504, source ip: 35.21.24.59, xff ip: 35.21.24.59, source port: 49945, destination ip: 40.70.1.138, destination port: 80, route_domain: 0, HTTP classifier: /Common/F5_External_1_RAMP_Policy, scheme HTTP, geographic location: , request: , username: , session_id: <5ca8e8ce394c1110>

Other F5 program logs look different so I need several filters.

Grtz and tx

Willem

[tmcdonald]

Have you confirmed that the "program" field is actually being set to "dcc"? What does it show if you run a query and view the log entry details?

[WillemDH]

Yes dcc is recognized as one of the programs.

[ssax]

Can you post your input and filter chains, we're wondering if it's hitting a previous one.

[WillemDH]

Hello,

i think you are asking for this?

Code: Select all

# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Tue, 31 Mar 2015 09:32:44 +0200
#

#
# Global Configuration
#

input {
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }
    syslog {
        type => 'syslog'
        port => 5544
    }
    syslog {
        type => 'syslog-eternus'
        port => 1516
    }
    syslog {
        type => 'syslog-esx'
        port => 514
    }
    syslog {
        type => 'syslog-infoblox'
        port => 5545
    }
    syslog {
        type => 'syslog-linux'
        port => 5546
    }
    syslog {
        type => 'syslog-brocade'
        port => 5547
    }
    syslog {
        type => 'syslog-f5'
        port => 5548
    }
    syslog {
        type => 'naf'
        port => 5549
    }
}

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
     
    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
    if [type] == "syslog-brocade" {
        grok {
          match => [ "message", "<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day}-%{TIME:time}%{GREEDYDATA:data1}WWN %{IPV6:wwn}%{GREEDYDATA:data2}%{LOGLEVEL:loglevel}\, %{HOSTNAME:hostname}\, %{GREEDYDATA:info}" ]
        remove_tag => "_grokparsefailure"
        add_tag => "grokked"
        }     
      }
    
    if [type] == "syslog-f5" {
        grok {     
          match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}" ]
        remove_tag => "_grokparsefailure"
        add_tag => "grokked"
        }   
    }
    
    if [program] == "dcc" {
        grok {     
          match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME2:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}User-Agent: %{GREEDYDATA:useragent}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}", "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME2:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}" ]
    }
    
}

#
# Local Configuration
#

Grtz

[jolson]

The thing about filters is that they are matched on a top-down basis, and typically when a match is found the log will stop descending the filter list.

What this means is that your 'dcc' logs are likely getting 'stolen' by your syslog-brocade or syslog-f5 filters. You may be able to reprimand this by putting the 'dcc' filter above the other brocade filters.

Another option you have is to use the break_on_match definition in your filters:

break_on_match
Value type is boolean
Default value is true
Break on first match. The first successful match by grok will result in the filter being finished. If you want grok to try all patterns (maybe you are parsing different things), then set this to false.

This may not return the results you're looking for either, but I figured you could give it a try.

My recommendation for now is to move the dcc filter above the other brocade filters and see if that makes a difference. If that does not work, please try adding break_on_match => false to your brocade filters. Thank you!

[WillemDH]

Jesse,

The reason I put the F5 dcc filter beneath the syslog-F5 filter is because the program dcc is assigned by the f5 filter... So putting it above the syslog-f5 will certainly not work.

I'll try your other suggestion though.

Grtz

Willem

[jolson]

Understood - that does make sense. Since the logs are matching the syslog-brocade filter first, they are stopping at that point - this explains why they never hit the 'dcc' filter. Let me know the results of setting break_on_match to false for your syslog-brocade filter. I'm hoping that this will help with this problem. Thanks Willem!

[WillemDH]

Jesse,

I tried

Code: Select all

if [type] == "syslog-f5" {
    grok {     
      match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}" ]
    remove_tag => "_grokparsefailure"
    add_tag => "grokked"
    break_on_match => false
    }   
}

and

if [type] == "syslog-f5" {
grok {
match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}" ]
remove_tag => "_grokparsefailure"
add_tag => "grokked"
break_on_match => "false"
}
}

But on both cases was the second filter for F5 dcc not applied.

Code: Select all

if [program] == "dcc" {
    grok {     
      match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME2:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}User-Agent: %{GREEDYDATA:useragent}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}", "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME2:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}" ]
}

I'm not sure what else I can try. Maybe I could try putting the dcc filter inside the f5 syslog filter? Or is this not possible?

[scottwilkerson]

Willem,

Once your item passes through the syslog-f5 grok filter posted in your OM, "message" is going to be exploded into fields specified in the pattern

Code: Select all

if [type] == "syslog-f5" {
    grok {     
      match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}" ]
    remove_tag => "_grokparsefailure"
    add_tag => "grokked"
    }   
}

With this pattern, you will no longer have the "message" field, it is being split up with the final bit being put in "info" field. I would change the dcc filter to something like the following (changing the match field and removing the first few patterns):

Code: Select all

if [program] == "dcc" {
    grok {     
      match => [ "info", "%{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}User-Agent: %{GREEDYDATA:useragent}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}", "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME2:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}" ]
}