help creating a filter

[benhank]

Hey guys!

can you guys help me chop up this log file ?

Code: Select all

9:52,013101007499,SYSTEM,globalprotect,0,2020/05/11 09:19:41,,globalprotectgateway-config-succ,Atrius-GW-N,0,0,general,informational,"GlobalProtect gateway client configuration generated. User name: bhankers, Private IP: x.x.x.x ,  Client region: US, Client IP:x.x.x.x  , Client version: version , Device name: AHNITOPS0002L, Client OS version: Microsoft Windows 7 Enterprise Edition Service Pack 1, 64-bit, VPN type:

I am trying to create a field for each item that are separated by commas. Thanks fellas!

[cdienger]

Assuming the line wasn't complete and that there should be a quote at the end, I came up with this very generic filter:

Code: Select all

%{HOUR}:%{MINUTE},%{NUMBER},%{WORD},%{WORD},%{NUMBER},%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME},,%{DATA},%{DATA},%{NUMBER},%{NUMBER},%{WORD},%{WORD},%{QUOTEDSTRING}

You can put the values in more meaningful fieldnames by adding the field names to each pattern. For example:

Code: Select all

%{HOUR:hour}:%{MINUTE:minute},%{NUMBER:this_number_means_something},%{WORD:type},%{WORD:a_meaningful_field_name},%{NUMBER:this_number_means_something_too}...

Use http://grokdebug.herokuapp.com/to see how the data is parsed out. Are you trying to parse out the information in QUOTEDSTRING as well? If so, I think the best approach would be use a filter like the above to get QUOTEDSTRING into its own field and then use kv(https://www.elastic.co/guide/en/logstas ... rs-kv.html) to parse the rest:

Code: Select all

kv{
source => QUOTEDSTRING
field_split =>", "
value_split =>":"
}

[benhank]

can you keep this open I am experimenting with this to get it working and THANKS man!

[cdienger]

No problem. We'll wait for your update.

[benhank]

So, the filter would have worked but then I found out that cisco logs are challenging to parse. There is too much variation in the files to parse correctly.
I noticed that the grokdebugger has something called patterns for cisco firewalls:

Code: Select all

https://grokdebug.herokuapp.com/patterns#

and i am wondering if that is what I should be using?

[cdienger]

Those could be handy. Do you have multiple log types coming in on the same input? I would recommend setting up unique inputs for each type of log and then set up filters to apply to those inputs. For example:

Code: Select all

tcp {
        type => 'cisco_log1'
        port => 5545
    }

Code: Select all

if [type] == 'cisco_log1' {
      grok {
        match => [ 'message', '%{HOUR:hour}:%{MINUTE:minute},%{NUMBER:this_number_means_something},%{WORD:type},%{WORD:a_meaningful_field_name},%{NUMBER:this_number_means_something_too}' ]
      }
}

Feel free to provide examples of the other logs if you need help creating a pattern for those.

[benhank]

yes I and I will provide some goodies in a bit after I clean them up

[benhank]

ok so here they are:

Code: Select all

<14>Jun 11 10:01:25 Panorama.mycompany.org 1,2020/06/11 10:01:25,013101007499,SYSTEM,globalprotect,0,2020/06/11 10:01:20,,globalprotectgateway-config-succ,mycompany2-GW-N,0,0,general,informational,"GlobalProtect gateway client configuration generated. User name: lskywalker, Private IP: 123.123.123.123,  Client region: US, Client IP: 123.123.123.123, Client version: 5.0.7-2, Device name: AHNITERP0002L, Client OS version: Microsoft Windows 7 Enterprise Edition Service Pack 1, 64-bit, VPN type: Device Level VPN.",6822056328467525919,0x8000000000000000,0,0,0,0,,MBO-PA-1234-2
09:55:39,,globalprotectgateway-config-release,mycompany2-GW-N,0,0,general,informational,"GlobalProtect gateway client configuration released. User name: lskywalker, Private IP: 123.123.123.123, Client version: 5.0.7-2, Device name: AHNITHDSK006L, Client OS version: Microsoft Windows 7 Enterprise Edition Service Pack 1, 64-bit, VPN type: Device Level VPN.",6822062779510505745,0x8000000000000000,0,0,0,0,,MBO-PA-1234-1
<14>Jun 10 11:20:07 Panorama.mycompany.org 1,2020/06/10 11:20:07,013101007502,SYSTEM,auth,0,2020/06/10 11:19:59,,auth-success,DUO VIP with failback,0,0,general,informational,"When authenticating user 'lskywalker' from 123.123.123.123', a less secure authentication method PAP is used. Please migrate to PEAP or EAP-TTLS. Authentication Profile 'DUO VIP with failback', vsys 'vsys1', Server Profile 'DUO RADIUS - VIP', Server Address '123.123.123.123'",6822062779510453156,0x8000000000000000,0,0,0,0,,MBO-PA-1234-1
<14>Jun 10 11:19:32 Panorama.mycompany.org 1,2020/06/10 11:19:32,013101007502,SYSTEM,globalprotect,0,2020/06/10 11:19:26,,globalprotectportal-auth-fail,mycompany2-Portal,0,0,general,informational,"GlobalProtect portal user authentication failed. Login from: 123.123.123.123, Source region: US, User name: lskywalker, Client OS version: Microsoft Windows 7 Enterprise Edition Service Pack 1, 64-bit, Reason: Authentication failed: Invalid username or password, Auth type: profile.",6822062779510453137,0x8000000000000000,0,0,0,0,,MBO-PA-1234-1
<14>Jun 11 10:01:22 Panorama.mycompany.org 1,2020/06/11 10:01:22,013101007502,SYSTEM,globalprotect,0,2020/06/11 10:01:20,,globalprotectgateway-regist-fail,mycompany2-GW-N,0,0,general,informational,"GlobalProtect gateway user login failed. Login from: 123.123.123.123, Source region: US, User name: lskywalker, Client OS version: Microsoft Windows 7 Enterprise Edition Service Pack 1, 64-bit, error: Existing user session found.",6822062779510506095,0x8000000000000000,0,0,0,0,,MBO-PA-1234-1

[cdienger]

It's not pretty but this pattern:

Code: Select all

%{NUMBER},%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME},%{NUMBER},%{WORD},%{WORD},%{NUMBER},%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME},,%{DATA},%{DATA},%{NUMBER},%{NUMBER},%{WORD},%{WORD},%{QUOTEDSTRING},%{NUMBER},%{DATA},%{NUMBER},%{NUMBER},%{NUMBER},%{NUMBER},,%{GREEDYDATA}

With this input:

Code: Select all

    syslog {
        type => 'test'
        port => 5545
    }

Should do the trick. Seems to work with the lines provided except:

Code: Select all

09:55:39,,globalprotectgateway-config-release,mycompany2-GW-N,0,0,general,informational,"GlobalProtect gateway client configuration released. User name: lskywalker, Private IP: 123.123.123.123, Client version: 5.0.7-2, Device name: AHNITHDSK006L, Client OS version: Microsoft Windows 7 Enterprise Edition Service Pack 1, 64-bit, VPN type: Device Level VPN.",6822062779510505745,0x8000000000000000,0,0,0,0,,MBO-PA-1234-1

but I think this may have been mistakenly shortened. Can you confirm?

Improvements I would maybe add would be to add meaningful field names and possibly set up a custom pattern to match the date field instead of using "%{NUMBER},%{YEAR}/%{MONTHNUM}/%{MONTHDAY}". These can be done if desired once the rough pattern is confirmed to work.

Custom patterns are discussed in https://www.elastic.co/guide/en/logstas ... -grok.html.

[benhank]

That is the actual syslog, I just confirmed it.

you are looking at the grand-master of cut and paste lol, so just to make sure I do this properly:

Code: Select all

if host == '123.123.123.123
grok {
match "message" =>%{NUMBER},%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME},%{NUMBER},%{WORD},%{WORD},%{NUMBER},%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME},,%{DATA},%{DATA},%{NUMBER},%{NUMBER},%{WORD},%{WORD},%{QUOTEDSTRING},%{NUMBER},%{DATA},%{NUMBER},%{NUMBER},%{NUMBER},%{NUMBER},,%{GREEDYDATA}
}


kv{
source => QUOTEDSTRING
field_split =>", "
value_split =>":"
}

I swear by someone else's mother that I will get the hang of this or someone else's family will pay a dear price !!!