Parsing XML in Message
Parsing XML in Message
Hello:
I have looked around at the forum and Google in general but can't wrap my head around creating an input filter for a Mcafee EPO syslog message. It is currently arriving with the following as the message field (some fields altered for public):
<29>1 2017-08-14T02:58:27.0Z EPOSVR1 EPOEvents - EventFwd [agentInfo@3401 tenantId="1"] <?xml version="1.0" encoding="UTF-8"?><EE_Event><MachineInfo><MachineName>Computer_Name</MachineName><AgentGUID>{942a970f-c4ed-424a-ae94-b573676a0d6d}</AgentGUID><IPAddress>10.10.10.12</IPAddress><OSName>Windows 7</OSName><UserName>SYSTEM</UserName><TimeZoneBias>240</TimeZoneBias><RawMACAddress>94659cb1e8fb</RawMACAddress></MachineInfo><EventData ProductName="Drive Encryption" ProductVersion="7.2.1.24" ProductFamily="MCAFEE_EED"><EventNode><EventID>30030</EventID><Severity>0</Severity><GMTTime>2017-08-14T14:57:45</GMTTime><Data><?xml version="1.0" encoding="UTF-8"?> <ESAuditLogItemList xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:ns1="ns1" xsi:type="ns1:ESAuditLogItemList"> <maxEntries>1000</maxEntries> <audits xsi:type="ns1:ESAuditLogItem"> <id>0</id> <type>0</type> <event>30113</event> <timestamp xsi:type="ns1:MfeEpeTimestamp"> <milliseconds>13147192837657</milliseconds> </timestamp> <parameters xsi:type="ns1:ESAuditLogItemParameter"> <name>userUuid</name> <value>mrobinson</value> </parameters> </audits> </ESAuditLogItemList></Data><DataType>ESAuditLogItemList</DataType></EventNode></EventData></EE_Event>
Can I get some advice on converting that message into something helpful? Thanks!
I have looked around at the forum and Google in general but can't wrap my head around creating an input filter for a Mcafee EPO syslog message. It is currently arriving with the following as the message field (some fields altered for public):
<29>1 2017-08-14T02:58:27.0Z EPOSVR1 EPOEvents - EventFwd [agentInfo@3401 tenantId="1"] <?xml version="1.0" encoding="UTF-8"?><EE_Event><MachineInfo><MachineName>Computer_Name</MachineName><AgentGUID>{942a970f-c4ed-424a-ae94-b573676a0d6d}</AgentGUID><IPAddress>10.10.10.12</IPAddress><OSName>Windows 7</OSName><UserName>SYSTEM</UserName><TimeZoneBias>240</TimeZoneBias><RawMACAddress>94659cb1e8fb</RawMACAddress></MachineInfo><EventData ProductName="Drive Encryption" ProductVersion="7.2.1.24" ProductFamily="MCAFEE_EED"><EventNode><EventID>30030</EventID><Severity>0</Severity><GMTTime>2017-08-14T14:57:45</GMTTime><Data><?xml version="1.0" encoding="UTF-8"?> <ESAuditLogItemList xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:ns1="ns1" xsi:type="ns1:ESAuditLogItemList"> <maxEntries>1000</maxEntries> <audits xsi:type="ns1:ESAuditLogItem"> <id>0</id> <type>0</type> <event>30113</event> <timestamp xsi:type="ns1:MfeEpeTimestamp"> <milliseconds>13147192837657</milliseconds> </timestamp> <parameters xsi:type="ns1:ESAuditLogItemParameter"> <name>userUuid</name> <value>mrobinson</value> </parameters> </audits> </ESAuditLogItemList></Data><DataType>ESAuditLogItemList</DataType></EventNode></EventData></EE_Event>
Can I get some advice on converting that message into something helpful? Thanks!
Re: Parsing XML in Message
Use a mutate filter to strip this part out:
And you can store it in it's own field or just drop it entirely, then pass the resulting message through an xml filter. Example:
This filter rule would first replace everything in message that matches the <.*]\s regular expression, then parse the rest of the message as XML. This seems to work based on my tests on regexr (I am working with a small sample set; The regex may need to be refined):
http://regexr.com/3gi64
So as part of the mutate step, everything preceding the XML object in the message gets dropped. Then we can just apply a regular old xml filter against the resulting message field and get nice, neat individual fields (hopefully).
Code: Select all
<29>1 2017-08-14T02:58:27.0Z EPOSVR1 EPOEvents - EventFwd [agentInfo@3401 tenantId="1"]
Code: Select all
if [type] == "McAfee" {
mutate {
gsub => [
"message", "^<.*]\s", ""
]
}
xml {
source => "message"
}
}
http://regexr.com/3gi64
So as part of the mutate step, everything preceding the XML object in the message gets dropped. Then we can just apply a regular old xml filter against the resulting message field and get nice, neat individual fields (hopefully).
Last edited by mcapra on Tue Aug 15, 2017 9:40 am, edited 2 times in total.
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: Parsing XML in Message
Thanks for the assist, mcapra!
CameronWP, let us know once you've had a chance to test this suggestion.
CameronWP, let us know once you've had a chance to test this suggestion.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: Parsing XML in Message
Thanks for that response! I am having a bit of a problem implementing it though as it seems when the filter is applied, I don't get any messages in the log at all. If I remove the XML portion of the filter the messages appear with the front of the message removed as per the GSUB command but stiull in that raw XML format. If I disable the filter they show up as before of course. Sorry if this is a basic question, I am new to the filtering aspect of this application. I currently have things set as:
Input:
tcp {
type => 'mcafee'
port => 6514
ssl_cacert => "/etc/pki/tls/certs/rootCA.pem"
ssl_cert => "/etc/pki/tls/certs/device-nls.crt"
ssl_key => "/etc/pki/tls/private/device-nls.key"
ssl_enable => true
ssl_verify => false
}
Filter:
if [type] == 'mcafee' {
mutate {
gsub => [
'message', '<.*]\s', ''
]
}
}
xml {
source => 'message'
}
Input:
tcp {
type => 'mcafee'
port => 6514
ssl_cacert => "/etc/pki/tls/certs/rootCA.pem"
ssl_cert => "/etc/pki/tls/certs/device-nls.crt"
ssl_key => "/etc/pki/tls/private/device-nls.key"
ssl_enable => true
ssl_verify => false
}
Filter:
if [type] == 'mcafee' {
mutate {
gsub => [
'message', '<.*]\s', ''
]
}
}
xml {
source => 'message'
}
Re: Parsing XML in Message
Your xml step is outside the mcafee logic. This means the xml step is applied to all messages; no good. Make sure the xml step is included in your mcafee logic:
I also modified the regex a bit to match the head of the message a bit better. See if those changes get the logs flowing, and if they don't share a copy of your Logstash log. It can typically be found here:
Code: Select all
if [type] == 'mcafee' {
mutate {
gsub => [
'message', '^<.*]\s', ''
]
}
xml {
source => 'message'
}
}
Code: Select all
/var/log/logstash/logstash.log
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: Parsing XML in Message
Thanks for the response! Things are flowing again. I will need to do some experimentation around the parsing of the XML but I think I am on the right track. Thanks!
Re: Parsing XML in Message
You might also need to strip out the <?xml version="1.0" encoding="UTF-8"?> section from the message. logstash-filter-xml is a bit brittle from what I remember. The Logstash log may have some hints if it's encountering parsing errors, at any rate.
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: Parsing XML in Message
Thanks for the update. Keep us in the loop with any questions or issues that may pop up.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: Parsing XML in Message
Hi:
I am finally getting back to this and working towards a solution. I have the XML parsing from the message as expected into new fields that contain information in a JSON string. I am trying to pass that resultant field into a JSON filter but it isn't working as expected. Here is my filter:
Here is the result in Logserver with that filter in place:
I am finally getting back to this and working towards a solution. I have the XML parsing from the message as expected into new fields that contain information in a JSON string. I am trying to pass that resultant field into a JSON filter but it isn't working as expected. Here is my filter:
Code: Select all
if [type] == 'mcafee' {
mutate {
gsub => [
'message', '^<.*\?>', ''
]
}
xml {
source => 'message'
target => doc
}
json {
source => 'doc.MachineInfo'
target => EventDetails
}
}
You do not have the required permissions to view the files attached to this post.
Re: Parsing XML in Message
We seem to have enough data here to try and reproduce it. I'll be able to look more into it tomorrow and will get back to you.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.