Page 1 of 1
Optional Grok Fields
Posted: Wed May 04, 2016 10:08 am
by aer9480
Hello, I am having an issue when I try to parse a logstash log that has an optional field. The optional field is called installer_name, but when I try
Code: Select all
(installer_name=%{QS:Bit9_installer_exe})?
it doesn't match. I am using
http://grokconstructor.appspot.com/do/match#result to test my pattern, and it doesn't get past the installer_name. I have other fields after this that need to be parsed, but it's not letting me get past the installer_name field.
My log looks similar to this:
Code: Select all
<13>May 04 12:40:31 Bit9.xxxx.com Bit9 event: text="File 'c:\windows\temp\low\skypeclicktocall\download\skypetoolbars.msi' [8fb885c633a92a5391b9493afff529b12bdd90cbcbe15045612711280ad4b027] was blocked because it was unapproved." type="Policy Enforcement" subtype="Execution block (unapproved file)" hostname="523-AAPALMER-LT" username="NT AUTHORITY\SYSTEM" date="5/4/2016 12:40:21 PM" ip_address="192.168.1.1" process="c:\windows\system32\msiexec.exe" file_path="c:\windows\temp\low\skypeclicktocall\download\skypetoolbars.msi" file_name="skypetoolbars.msi" file_hash="8fb885c633a92a5391b9493afff529b12bdd90cbcbe15045612711280ad4b027" installer_name="skypec2cautoupdatesvc.exe" policy="MPA-High" rule_name="Block unapproved scripts" process_key="00000178-0000-1954-01d1-a6021ef1126c" server_version="7.2.1.1562" file_trust="-1" file_threat="-1" process_trust="10" process_threat="0"
Not all of them have installer_name.
Any ideas? Thanks!
Re: Optional Grok Fields
Posted: Wed May 04, 2016 10:12 am
by hsmith
Can you post the full grok filter here? I have a feeling the issue might be coming from a spacing issue.
Re: Optional Grok Fields
Posted: Wed May 04, 2016 10:31 am
by aer9480
Code: Select all
<%{POSINT:SysLogPri}>%{CISCOTIMESTAMP} %{GREEDYDATA:Bit9_Server} Bit9 event: +text=%{QS:Bit9_text} type=%{QS:Bit9_Type} subtype=%{QS:Bit9_subtype} hostname=%{QS:Bit9_Hostname} username=%{QS:Bit9_User} date=%{QS:Bit9_Date} ip_address=%{QS:Bit9_client_ip} process=%{QS:Bit9_Process} file_path=%{QS:Bit9_Path} file_name=%{QS:Bit9_File} file_hash=%{QS:Bit9_hash} (installer_name=%{QS:Bit9_installer_exe})? policy=%{QS:Bit9_Policy} rule_name=%{QS:Bit9_rule_name} process_key=%{QS:Bit9_proc_key} server_version=%{QS:Bit9_server_ver} file_trust=%{QS:Bit9_file_trust} file_threat=%{QS:Bit9_file_threat} process_trust=%{QS:Bit9_process_trust} process_threat=%{QS:Bit9_process_threat}
I changed the GREEDYDATA to Quoted String like you advised last time. I also changed a few things around because of our log formats. I remembered you said to use ()? for optional fields, but I cant seem to get it working. Thanks!
Re: Optional Grok Fields
Posted: Wed May 04, 2016 10:41 am
by hsmith
Try removing the space between {QS:Bit9_hash} and (installer_name=%{QS:Bit9_installer_exe})?
Reason for this it's looking for two spaces now if installer_name is not there. One after Bit9_hash and another before policy. Do you see where I'm coming from?
Re: Optional Grok Fields
Posted: Wed May 04, 2016 11:18 am
by aer9480
Worked perfectly. I didn't even think of that. Thanks buddy!
Re: Optional Grok Fields
Posted: Wed May 04, 2016 12:02 pm
by hsmith
Glad to hear it!
Should I go ahead and close the thread?
Re: Optional Grok Fields
Posted: Wed May 04, 2016 12:04 pm
by aer9480
Sure. Should be all set.