Page 1 of 1
How to parse DHCP LOG message
Posted: Tue Apr 02, 2019 1:45 pm
by dlukinski
Hello LOG Support
How to parse DHCP Log message
Code: Select all
{"EventReceivedTime":"2019-03-29 09:57:45","SourceModuleName":"windowsfile","SourceModuleType":"im_file","message":"11,03/29/19,09:57:44,Renew,10.64.5.106,PF19TXYZ.res.kcg.global,144F8AF0CD82,,9856455,0,,,,0x4D53465420352E30,MSFT 5.0,,,,0"}
to separate Time, type (Renew), IP address, computer name, MAC into the fields ?
Thank you
Re: How to parse DHCP LOG message
Posted: Tue Apr 02, 2019 2:52 pm
by cdienger
A grok filter like this should do the trick:
Code: Select all
grok {
match => { "message" => "%{INT:number},%{DATE:date},%{TIME:time},%{WORD:mtype},%{IP:ip},%{HOSTNAME:computername},%{BASE16NUM:mac},%{GREEDYDATA}"}
}
Note that I used mtype(message type) instead of type - type is a common field in logstash that I didn't want to overwrite.
Re: How to parse DHCP LOG message
Posted: Thu Apr 11, 2019 8:27 am
by dlukinski
cdienger wrote:A grok filter like this should do the trick:
Code: Select all
grok {
match => { "message" => "%{INT:number},%{DATE:date},%{TIME:time},%{WORD:mtype},%{IP:ip},%{HOSTNAME:computername},%{BASE16NUM:mac},%{GREEDYDATA}"}
}
Note that I used mtype(message type) instead of type - type is a common field in logstash that I didn't want to overwrite.
This work great, thank you
If I wanted to add more fields, how would I go about it?
- OR I should create a new grok filter for every specific message (just trying to understand the concept)
Re: How to parse DHCP LOG message
Posted: Thu Apr 11, 2019 2:05 pm
by cdienger
Creating new filters is an option. How to do this really depends on how the logs vary and what data you'd like to extract. You could for example extract the next field after the mac address if you wanted or ignore it if it's blank or didn't meet the filter requirement:
Code: Select all
grok {
match => { "message" => "^%{INT:number},%{DATE:date},%{TIME:time},%{WORD:mtype},%{IP:ip},%{HOSTNAME:computername},%{BASE16NUM:mac},(%{INT:anothernumber})?"}
}
This would create all the same fields from the previous example, but would create an additional field(anothernumber) if the next field contained an integer.
Re: How to parse DHCP LOG message
Posted: Wed Apr 17, 2019 10:20 am
by dlukinski
cdienger wrote:Creating new filters is an option. How to do this really depends on how the logs vary and what data you'd like to extract. You could for example extract the next field after the mac address if you wanted or ignore it if it's blank or didn't meet the filter requirement:
Code: Select all
grok {
match => { "message" => "^%{INT:number},%{DATE:date},%{TIME:time},%{WORD:mtype},%{IP:ip},%{HOSTNAME:computername},%{BASE16NUM:mac},(%{INT:anothernumber})?"}
}
This would create all the same fields from the previous example, but would create an additional field(anothernumber) if the next field contained an integer.
I am struggling with this new one now and few questions about it
Code: Select all
232 <134> 2019-04-17T23:01:39+08:00 PulseSecure: 2019-04-17 23:01:39 - kc-ssl-apac-node1 - [222.65.214.176] yanxia(KC password authentication)[KC Users Secure PW] - VPN Tunneling: Session ended for user with IPv4 address 10.106.12.92
OR
240 <134> 2019-04-17T23:15:36+08:00 PulseSecure: 2019-04-17 23:15:36 - kc-ssl-apac-node1 - [47.31.84.174] kaushas(KC token authentication)[KC Users Secure Pulse] - VPN Tunneling: User with IP 10.106.10.105 connected with ESP transport mode.
match => { "PulseSecure" => "%{DATE:date} %{TIME:time} %{WORD:mtype} %{IP:ip} ..
- How to deal with "space" instead of "," in between fields ?
- How to omit unwanted fields ?
- How to figure required fields format ?
This is what is needed: 2019-04-17 23:01:39 - kc-ssl-apac-node1 IP - yanxia - [KC Users Secure PW] - Message
- yanxia is a user name
- We onyl need (KC token authentication) OR [KC Users Secure Pulse]
- IP is an external IP
- Message is a full message in the end of the operation
Re: How to parse DHCP LOG message
Posted: Wed Apr 17, 2019 1:55 pm
by cdienger
Here's a filter I came up with:
Code: Select all
grok {
match => { "message" => "%{DATE:date} %{TIME:time} - %{DATA:mtype} - \[%{IP:ip}\] %{WORD:username}%{GREEDYDATA:authstring} - %{GREEDYDATA:message}"}
overwrite => [ "message" ]
}
I use the
grok debugger to come up with patterns. Some examples of creating custom filters are covered in
https://support.nagios.com/kb/article/n ... ew-98.html as well as
https://assets.nagios.com/downloads/nag ... ilters.pdf.
Re: How to parse DHCP LOG message
Posted: Thu Apr 25, 2019 12:08 pm
by dlukinski
This also worked well. Thank you. We could close the case.
Any documentation on parsing messages (with object types to use)?
Re: How to parse DHCP LOG message
Posted: Thu Apr 25, 2019 2:27 pm
by cdienger
If you're ask which predefined patterns to use, I'd suggest taking a look at the patterns listed at
https://grokdebug.herokuapp.com/patterns. You'll find that you'll probably use a lot of the same ones if you create multiple grok filters.