Using Filters for problematic log

[cpatterson1]

I am trying to create a grok filter that works for a log we are trying to pull into our system.

Issue 1: The issue is the logfile outputs logs like this "Data1|Data2|Data3|" which prevents us from grabbing a couple pieces of information. So, I followed the instructions here to try and replace that | character with a space.
http://stackoverflow.com/questions/2476 ... terns-file

That makes sense to me. I tried to apply it in our filter and I am getting a configuration error. This is what I have:

Code: Select all

if [type] == 'WAF' {
  mutate {
    gsub => ["message","\|"," "]
    }

The second part of the filter is one where I am trying to get the date output into a single timestamp. Here is an example of the timestamp in the log:
Dec 13 2016 14:25:00

I tried all the timestamp patterns I could find, but not would pull in the full field. So I tried creating a pattern, which worked fine on grokconstructor. I then tried to apply it like this. I haven't been able to thoroughly test this yet as the above piece is not working yet. But I wanted to verify how this would be used.

Code: Select all

  grok {
    match => ["WAF_TIMESTAMP", "%{MONTH} %{DATA} %{YEAR} %{TIME}"]
    match => ['message', %{GREEDYDATA} %{DATA:Device} %{DATA:Model} %{WAF_TIMESTAMP:Timestamp} ....

[mcapra]

Would it be possible for you to give us sample log entries to try and match against?

[cpatterson1]

<7> CEF:1|A10|TH3030S|2.7.2-P7-SP3|WAF|Dec 13 2016 14:25:00|session-id|2|src=10.2.52.252 spt=25049 dst=10.2.208.150 dpt=80 hst="changedev.agoc.com" cs1=?dev?DefaultWebServer cs2=fb76283ae9c71b37 act=learn md=passive svc=http req="GET /images/grid/last.gif HTTP/1.1" 0 msg="New session created: Id=fb76283ae9c71b37"

[mcapra]

The pain in the butt with this is that grok doesn't always handle atomic groups (like lookaheads) very gracefully. Despite that, here's the filter I've come up with. You may notice i'm essentially splitting the message into 2 pieces, one of which can be fed through a key-value filter (kv) to be handled a little more efficiently:

Code: Select all

if [type] == 'WAF' {
    grok {
        match => ['message', "(<)%{BASE10NUM:val1}(>) (?<string:pipe_msg>([A-z].*(\|)))%{GREEDYDATA:kv_msg}"]
    }
    grok {
        match => ['pipe_msg', '%{DATA:val2}\|%{DATA:val3}\|%{DATA:val4}\|%{DATA:val5}\|%{DATA:val6}\|%{DATA:val7}\|%{DATA:val8}\|%{DATA:val9}\|']
    }
    kv {
        source => "kv_msg"
    }
    mutate
    {
        remove_field => [ "kv_msg", "pipe_msg" ]
    }
}

Which breaks up the message like so:

2016_12_15_11_07_30_Dashboard_Nagios_Log_Server.png

You may want to alter the val1-9 fields to make more sense for your desired outcomes.

[cpatterson1]

Great, thanks for you assistance! That is working how I wanted.

[mcapra]

Awesome! Is it alright if we close this thread and mark the issue as resolved?