Parsing XML in Message

[CameronWP]

Hello:

I have looked around at the forum and Google in general but can't wrap my head around creating an input filter for a Mcafee EPO syslog message. It is currently arriving with the following as the message field (some fields altered for public):

<29>1 2017-08-14T02:58:27.0Z EPOSVR1 EPOEvents - EventFwd [agentInfo@3401 tenantId="1"] <?xml version="1.0" encoding="UTF-8"?><EE_Event><MachineInfo><MachineName>Computer_Name</MachineName><AgentGUID>{942a970f-c4ed-424a-ae94-b573676a0d6d}</AgentGUID><IPAddress>10.10.10.12</IPAddress><OSName>Windows 7</OSName><UserName>SYSTEM</UserName><TimeZoneBias>240</TimeZoneBias><RawMACAddress>94659cb1e8fb</RawMACAddress></MachineInfo><EventData ProductName="Drive Encryption" ProductVersion="7.2.1.24" ProductFamily="MCAFEE_EED"><EventNode><EventID>30030</EventID><Severity>0</Severity><GMTTime>2017-08-14T14:57:45</GMTTime><Data><?xml version="1.0" encoding="UTF-8"?> <ESAuditLogItemList xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:ns1="ns1" xsi:type="ns1:ESAuditLogItemList"> <maxEntries>1000</maxEntries> <audits xsi:type="ns1:ESAuditLogItem"> <id>0</id> <type>0</type> <event>30113</event> <timestamp xsi:type="ns1:MfeEpeTimestamp"> <milliseconds>13147192837657</milliseconds> </timestamp> <parameters xsi:type="ns1:ESAuditLogItemParameter"> <name>userUuid</name> <value>mrobinson</value> </parameters> </audits> </ESAuditLogItemList></Data><DataType>ESAuditLogItemList</DataType></EventNode></EventData></EE_Event>

Can I get some advice on converting that message into something helpful? Thanks!

[mcapra]

Use a mutate filter to strip this part out:

Code: Select all

<29>1 2017-08-14T02:58:27.0Z EPOSVR1 EPOEvents - EventFwd [agentInfo@3401 tenantId="1"] 

And you can store it in it's own field or just drop it entirely, then pass the resulting message through an xml filter. Example:

Code: Select all

if [type] == "McAfee" {
    mutate {
        gsub => [
            "message", "^<.*]\s", ""
        ]
    }
    xml {
        source => "message"
    }
}

This filter rule would first replace everything in message that matches the <.*]\s regular expression, then parse the rest of the message as XML. This seems to work based on my tests on regexr (I am working with a small sample set; The regex may need to be refined):
http://regexr.com/3gi64

So as part of the mutate step, everything preceding the XML object in the message gets dropped. Then we can just apply a regular old xml filter against the resulting message field and get nice, neat individual fields (hopefully).

[cdienger]

Thanks for the assist, mcapra!

CameronWP, let us know once you've had a chance to test this suggestion.

[CameronWP]

Thanks for that response! I am having a bit of a problem implementing it though as it seems when the filter is applied, I don't get any messages in the log at all. If I remove the XML portion of the filter the messages appear with the front of the message removed as per the GSUB command but stiull in that raw XML format. If I disable the filter they show up as before of course. Sorry if this is a basic question, I am new to the filtering aspect of this application. I currently have things set as:

Input:

tcp {
type => 'mcafee'
port => 6514
ssl_cacert => "/etc/pki/tls/certs/rootCA.pem"
ssl_cert => "/etc/pki/tls/certs/device-nls.crt"
ssl_key => "/etc/pki/tls/private/device-nls.key"
ssl_enable => true
ssl_verify => false
}

Filter:

if [type] == 'mcafee' {
mutate {
gsub => [
'message', '<.*]\s', ''
]
}
}

xml {
source => 'message'
}

[mcapra]

Your xml step is outside the mcafee logic. This means the xml step is applied to all messages; no good. Make sure the xml step is included in your mcafee logic:

Code: Select all

if [type] == 'mcafee' {
    mutate {
        gsub => [
            'message', '^<.*]\s', ''
        ]
    }
    xml {
        source => 'message'
    }
}

I also modified the regex a bit to match the head of the message a bit better. See if those changes get the logs flowing, and if they don't share a copy of your Logstash log. It can typically be found here:

Code: Select all

/var/log/logstash/logstash.log

[CameronWP]

Thanks for the response! Things are flowing again. I will need to do some experimentation around the parsing of the XML but I think I am on the right track. Thanks!

[mcapra]

You might also need to strip out the <?xml version="1.0" encoding="UTF-8"?> section from the message. logstash-filter-xml is a bit brittle from what I remember. The Logstash log may have some hints if it's encountering parsing errors, at any rate.

[cdienger]

Thanks for the update. Keep us in the loop with any questions or issues that may pop up.

[CameronWP]

Hi:

I am finally getting back to this and working towards a solution. I have the XML parsing from the message as expected into new fields that contain information in a JSON string. I am trying to pass that resultant field into a JSON filter but it isn't working as expected. Here is my filter:

Code: Select all

if [type] == 'mcafee' {
    mutate {
        gsub => [
            'message', '^<.*\?>', ''
        ]
    }
    xml {
        source => 'message'
        target => doc
        }
    json {
         source => 'doc.MachineInfo'
         target => EventDetails
         }
}

Here is the result in Logserver with that filter in place:

Mcafee.JPG

[cdienger]

We seem to have enough data here to try and reproduce it. I'll be able to look more into it tomorrow and will get back to you.