How to parse DHCP LOG message

[dlukinski]

Hello LOG Support

How to parse DHCP Log message

Code: Select all

{"EventReceivedTime":"2019-03-29 09:57:45","SourceModuleName":"windowsfile","SourceModuleType":"im_file","message":"11,03/29/19,09:57:44,Renew,10.64.5.106,PF19TXYZ.res.kcg.global,144F8AF0CD82,,9856455,0,,,,0x4D53465420352E30,MSFT 5.0,,,,0"}

to separate Time, type (Renew), IP address, computer name, MAC into the fields ?

Thank you

[cdienger]

A grok filter like this should do the trick:

Code: Select all

grok {
        match => { "message" => "%{INT:number},%{DATE:date},%{TIME:time},%{WORD:mtype},%{IP:ip},%{HOSTNAME:computername},%{BASE16NUM:mac},%{GREEDYDATA}"}

}

Note that I used mtype(message type) instead of type - type is a common field in logstash that I didn't want to overwrite.

[dlukinski]

A grok filter like this should do the trick:

Code: Select all

grok {
        match => { "message" => "%{INT:number},%{DATE:date},%{TIME:time},%{WORD:mtype},%{IP:ip},%{HOSTNAME:computername},%{BASE16NUM:mac},%{GREEDYDATA}"}

}

Note that I used mtype(message type) instead of type - type is a common field in logstash that I didn't want to overwrite.

This work great, thank you

If I wanted to add more fields, how would I go about it?
- OR I should create a new grok filter for every specific message (just trying to understand the concept)

[cdienger]

Creating new filters is an option. How to do this really depends on how the logs vary and what data you'd like to extract. You could for example extract the next field after the mac address if you wanted or ignore it if it's blank or didn't meet the filter requirement:

Code: Select all

grok {
        match => { "message" => "^%{INT:number},%{DATE:date},%{TIME:time},%{WORD:mtype},%{IP:ip},%{HOSTNAME:computername},%{BASE16NUM:mac},(%{INT:anothernumber})?"}
}

This would create all the same fields from the previous example, but would create an additional field(anothernumber) if the next field contained an integer.

[dlukinski]

Creating new filters is an option. How to do this really depends on how the logs vary and what data you'd like to extract. You could for example extract the next field after the mac address if you wanted or ignore it if it's blank or didn't meet the filter requirement:

Code: Select all

grok {
        match => { "message" => "^%{INT:number},%{DATE:date},%{TIME:time},%{WORD:mtype},%{IP:ip},%{HOSTNAME:computername},%{BASE16NUM:mac},(%{INT:anothernumber})?"}
}

This would create all the same fields from the previous example, but would create an additional field(anothernumber) if the next field contained an integer.

I am struggling with this new one now and few questions about it

Code: Select all

232 <134> 2019-04-17T23:01:39+08:00 PulseSecure: 2019-04-17 23:01:39 - kc-ssl-apac-node1 - [222.65.214.176] yanxia(KC password authentication)[KC Users Secure PW] - VPN Tunneling: Session ended for user with IPv4 address 10.106.12.92
OR
240 <134> 2019-04-17T23:15:36+08:00  PulseSecure: 2019-04-17 23:15:36 - kc-ssl-apac-node1 - [47.31.84.174] kaushas(KC token authentication)[KC Users Secure Pulse] - VPN Tunneling: User with IP 10.106.10.105 connected with ESP transport mode.

match => { "PulseSecure" => "%{DATE:date} %{TIME:time} %{WORD:mtype} %{IP:ip} ..

- How to deal with "space" instead of "," in between fields ?
- How to omit unwanted fields ?
- How to figure required fields format ?

This is what is needed: 2019-04-17 23:01:39 - kc-ssl-apac-node1 IP - yanxia - [KC Users Secure PW] - Message
- yanxia is a user name
- We onyl need (KC token authentication) OR [KC Users Secure Pulse]
- IP is an external IP
- Message is a full message in the end of the operation

[cdienger]

Here's a filter I came up with:

Code: Select all

grok {
        match => { "message" => "%{DATE:date} %{TIME:time} - %{DATA:mtype} - \[%{IP:ip}\] %{WORD:username}%{GREEDYDATA:authstring} - %{GREEDYDATA:message}"}
overwrite => [ "message" ]
}

I use the grok debugger to come up with patterns. Some examples of creating custom filters are covered in https://support.nagios.com/kb/article/n ... ew-98.html as well as https://assets.nagios.com/downloads/nag ... ilters.pdf.

[dlukinski]

Here's a filter I came up with:

Code: Select all

grok {
        match => { "message" => "%{DATE:date} %{TIME:time} - %{DATA:mtype} - \[%{IP:ip}\] %{WORD:username}%{GREEDYDATA:authstring} - %{GREEDYDATA:message}"}
overwrite => [ "message" ]
}

I use the grok debugger to come up with patterns. Some examples of creating custom filters are covered in https://support.nagios.com/kb/article/n ... ew-98.html as well as https://assets.nagios.com/downloads/nag ... ilters.pdf.

This also worked well. Thank you. We could close the case.

Any documentation on parsing messages (with object types to use)?

[cdienger]

If you're ask which predefined patterns to use, I'd suggest taking a look at the patterns listed at https://grokdebug.herokuapp.com/patterns. You'll find that you'll probably use a lot of the same ones if you create multiple grok filters.