Parsing XML in Message

[cdienger]

I'm not having much luck with the json filter either, but the xml filter is behaving much better:

Code: Select all

if [type] == 'mcafee' {
    mutate {
        gsub => [
            'message', '^<.*\?>', ''
        ]
    }
    xml {
        source => 'message'
        target => doc
		xpath => [ "EE_Event/MachineInfo", "EventDetails" ]
}
}

[CameronWP]

Hi:

So it appears that it is by design that logstash doesn't parse any fields with a period in it. There is a new filter called de_dot that removes the period from field names that might make things work.

De_Dot announcement: https://www.elastic.co/guide/en/logstas ... e_dot.html
Period Issue: https://www.elastic.co/guide/en/elastic ... names.html
Another Article: https://discuss.elastic.co/t/field-name ... tain/33251

I have an insanely long filter parsing with XML and it kind of works but it would be nice to make it simpler. I am playing with a bit of Ruby code and will post what I get working. Thanks!

[cdienger]

Thanks for the update! Look forward to anything you come up with!

[CameronWP]

Hi all:

Here is the finished version I came up with that is parsing things reasonably well:

if [type] == 'mcafee' {
mutate {
gsub => [
'message', '^<.*\?>', ''
]
}

if [message] =~ /^<EE_Event>/ {
xml {
source => 'message'
store_xml => false
xpath => [ "EE_Event/MachineInfo", "HostDetails" ]
xpath => [ "EE_Event/EventData", "EventDetails" ]
}

xml {
source => HostDetails
target => MachineDetails
}
xml {
source => EventDetails
target => Event
}
}

if [message] =~ /^<DLPAGENT9400>/ {
xml {
source => 'message'
store_xml => false
xpath => [ "DLPAGENT9400/MachineInfo", "HostDetails" ]
xpath => [ "DLPAGENT9400/EventList", "EventDetails" ]
}
xml {
source => HostDetails
target => MachineDetails
}
xml {
source => EventDetails
target => Event
}
}

if [message] =~ /^<BehaviourBlockEvent>/ {
xml {
source => 'message'
store_xml => false
xpath => [ "BehaviourBlockEvent/MachineInfo", "HostDetails" ]
xpath => [ "BehaviourBlockEvent/ScannerSoftware", "EventDetails" ]
}
xml {
source => HostDetails
target => MachineDetails
}
xml {
source => EventDetails
target => Event
}
}

if [message] =~ /^<UpdateEvents>/ {
xml {
source => 'message'
store_xml => false
xpath => [ "UpdateEvents/MachineInfo", "HostDetails" ]
xpath => [ "UpdateEvents/McAfeeCommonUpdater", "EventDetails" ]
}
xml {
source => HostDetails
target => MachineDetails
}
xml {
source => EventDetails
target => Event
}
}

if [message] =~ /^<DLPAGENT9300>/ {
xml {
source => 'message'
store_xml => false
xpath => [ "DLPAGENT9300/MachineInfo", "HostDetails" ]
xpath => [ "DLPAGENT9300/EventList", "EventDetails" ]
}
xml {
source => HostDetails
target => MachineDetails
}
xml {
source => EventDetails
target => Event
}

}

if [message] =~ /^<EPO[Ee]vent>/ {
xml {
source => 'message'
store_xml => false
xpath => [ "EPOevent/MachineInfo", "HostDetails" ]
xpath => [ "EPOevent/SoftwareInfo", "EventDetails" ]
xpath => [ "EPOevent/SoftwareInfo/Event/CommonFields", "CommonDescription" ]
xpath => [ "EPOevent/SoftwareInfo/CommonFields", "CommonDescription" ]
}
xml {
source => HostDetails
target => MachineDetails
}
xml {
source => EventDetails
target => Event
}
xml {
source => CommonDescription
target => Common
}
if [message] =~ /<CustomFields/ {
xml {
source => 'message'
store_xml => false
xpath => [ "EPOevent/SoftwareInfo/Event/CustomFields", "CustomDescription" ]
}
}
xml {
source => CustomDescription
target => Custom
}
}
mutate {
remove_field => [ "message" ]
remove_field => [ "HostDetails" ]
remove_field => [ "EventDetails" ]
}
}

[kyang]

Thanks for sharing!

Did you have any more questions or are we okay to lock this up?

[CameronWP]

I am good, the parser seems to be doing what it is supposed to. Thanks!

[kyang]

Sounds good! I'll be closing this thread!

If you have any more questions, feel free to create another thread.

Thanks for using the Nagios Support Forum!