XML input

[billy_strath]

I have logs that come in and look like the below. How can I tell Log Server that it is XML so that it "automatically" creates the fields and values without having to grok it to death? Thanks


<29>1 2019-01-25T01:04:24.0Z ITS-ORCH EPOEvents - EventFwd [agentInfo@3401 tenantId="1" bpsId="1" tenantGUID="{00000000-0000-0000-0000-000000000000}" tenantNodePath="1\2"] ?<?xml version="1.0" encoding="UTF-8"?><EPOEvent><MachineInfo><MachineName>ADM123</MachineName><AgentGUID>{59a428aa-4f6e-11e8-3fa3-b4b686296a37}</AgentGUID><IPAddress>x.x.x.x</IPAddress><OSName>Windows 10 Workstation</OSName><UserName>%CTX_DOMAIN_USER%</UserName><TimeZoneBias>0</TimeZoneBias><RawMACAddress>34415d167b00</RawMACAddress></MachineInfo><SoftwareInfo ProductName="McAfee Endpoint Security" ProductVersion="10.6.1.1124" ProductFamily="TVD"><CommonFields><Analyzer>ENDP_GS_1060</Analyzer><AnalyzerName>McAfee Endpoint Security</AnalyzerName><AnalyzerVersion>10.6.1.1124</AnalyzerVersion><AnalyzerHostName>ADM123</AnalyzerHostName><AnalyzerDATVersion></AnalyzerDATVersion><AnalyzerEngineVersion></AnalyzerEngineVersion></CommonFields><Event><EventID>1118</EventID><Severity>0</Severity><GMTTime>2019-01-25T12:48:05</GMTTime><CommonFields><AnalyzerDetectionMethod></AnalyzerDetectionMethod><ThreatName>_</ThreatName><ThreatType></ThreatType><ThreatCategory>ops.update.end</ThreatCategory><ThreatHandled>1</ThreatHandled><ThreatActionTaken>none</ThreatActionTaken><ThreatSeverity>6</ThreatSeverity></CommonFields></Event></SoftwareInfo></EPOEvent>

[billy_strath]

I found this https://discuss.elastic.co/t/xml-encode ... h/104380/5 which strips the front part out of the message an leaves me with a field starting with <?xml version="1.0" encoding="UTF-8"?> but I still don't have individual fields from with in the XML part

[mcapra]

Here's an older thread from the customer section that discusses some XML things:
https://support.nagios.com/forum/viewto ... ML#p229746

The short answer is use a mutate filter step to strip this garbage out:

Code: Select all

<29>1 2019-01-25T01:04:24.0Z ITS-ORCH EPOEvents - EventFwd [agentInfo@3401 tenantId="1" bpsId="1" tenantGUID="{00000000-0000-0000-0000-000000000000}" tenantNodePath="1\2"] ?

hich strips the front part out of the message an leaves me with a field starting with <?xml version="1.0" encoding="UTF-8"?> but I still don't have individual fields from with in the XML part

I think you'll also need to strip out the <?xml?> block prior to running this message through an xml filter. This does that:

Code: Select all

mutate {
  gsub => [
    'message', '^<.*\?>', ''
  ]
}

Then use an xml filter on the resulting message (which was altered by a mutate step previously):

Code: Select all

<EPOEvent><MachineInfo><MachineName>ADM123</MachineName><AgentGUID>{59a428aa-4f6e-11e8-3fa3-b4b686296a37}</AgentGUID><IPAddress>x.x.x.x</IPAddress><OSName>Windows 10 Workstation</OSName><UserName>%CTX_DOMAIN_USER%</UserName><TimeZoneBias>0</TimeZoneBias><RawMACAddress>34415d167b00</RawMACAddress></MachineInfo><SoftwareInfo ProductName="McAfee Endpoint Security" ProductVersion="10.6.1.1124" ProductFamily="TVD"><CommonFields><Analyzer>ENDP_GS_1060</Analyzer><AnalyzerName>McAfee Endpoint Security</AnalyzerName><AnalyzerVersion>10.6.1.1124</AnalyzerVersion><AnalyzerHostName>ADM123</AnalyzerHostName><AnalyzerDATVersion></AnalyzerDATVersion><AnalyzerEngineVersion></AnalyzerEngineVersion></CommonFields><Event><EventID>1118</EventID><Severity>0</Severity><GMTTime>2019-01-25T12:48:05</GMTTime><CommonFields><AnalyzerDetectionMethod></AnalyzerDetectionMethod><ThreatName>_</ThreatName><ThreatType></ThreatType><ThreatCategory>ops.update.end</ThreatCategory><ThreatHandled>1</ThreatHandled><ThreatActionTaken>none</ThreatActionTaken><ThreatSeverity>6</ThreatSeverity></CommonFields></Event></SoftwareInfo></EPOEvent>

[cdienger]

@billy_strath - please update us after following @mcapra's suggestion.

[billy_strath]

Ta
Got there in the end with a few mutates and xpath
Thanks

mutate {
gsub => ['message', '^<.*\?>', '']
replace => [ 'type', 'ePO1']

}
xml {
source => "message"
store_xml => false
xpath => { "/EPOEvent/MachineInfo/OSName/text()" => "OSName" }
xpath => { "/EPOEvent/SoftwareInfo/Event/CommonFields/ThreatCategory/text()" => "ThreatCategory" }
}

mutate {
replace => { "OSName" => "%{OSName}"}
replace => { "ThreatCategory" => "%{ThreatCategory}"}
}

[cdienger]

Glad to hear!