Page 1 of 1
Lowercase fields
Posted: Wed May 13, 2015 5:57 am
by WillemDH
Hello,
How could I lowercase all Fields? At the moment some inputs have Hostname, other hostname, same for ipadress and IpAddress and many more. I'd like to streamline the fields a bit, so making them all lowercase would be the best idea I think. I know I can use mutate to lowercase a field value, but this doesn't work for the Field itself.
Grtz
Willem
Re: Lowercase fields
Posted: Wed May 13, 2015 9:55 am
by jolson
It looks like this may be possible using a ruby filter. Please add this to the top of your filter list:
Code: Select all
filter {
ruby {
code => "event.overwrite(LogStash::Event.new(event.to_hash.inject({}){|r, (k, v)| r[k.downcase] = v; r }))"
}
}
This should lowercase your field names. Let me know if it works for you.
Re: Lowercase fields
Posted: Wed May 13, 2015 1:33 pm
by WillemDH
Jesse,
Code: Select all
ruby {
code => "event.overwrite(LogStash::Event.new(event.to_hash.inject({}){|r, (k, v)| r[k.downcase] = v; r }))"
}
Does not seem to work.
Stumbled on these:
https://groups.google.com/forum/#!topic ... OkOs3jWdxk
https://logstash.jira.com/browse/LOGSTASH-732
We were on logstash 1.4 right? So I'm guessing this is not possible at the moment. It might be possible to do it on the nxlog side?
Grtz
Willem
Re: Lowercase fields
Posted: Wed May 13, 2015 2:03 pm
by jolson
Yep - looks like version 1.4 broke that code. The github issue for this request is here:
https://github.com/elastic/logstash/issues/2526 - looks like the last activity was 29 days ago.
We can do this manually on the logstash-end of things for now as a workaround. Please add the following to the top of your filter-chain.
Code: Select all
mutate {
type => "eventlog"
rename => [ "Message", "message" ]
rename => [ "Severity", "severity" ]
rename => [ "Hostname", "hostname" ]
rename => [ "Channel", "channel" ]
rename => [ "EventID", "eventID" ]
rename => [ "EventReceivedTime", "event-received-time" ]
rename => [ "EventTime", "event-time" ]
rename => [ "EventType", "event-type" ]
rename => [ "RecordNumber", "record-number" ]
rename => [ "ProcessID", "process-id" ]
rename => [ "OpcodeValue", "opcode-value" ]
rename => [ "SourceModuleName", "source-module-name" ]
rename => [ "SeverityValue", "severity-value" ]
rename => [ "SourceModuleType", "source-module-type" ]
rename => [ "ProviderGuid", "provider-guid" ]
rename => [ "SourceName", "source-name" ]
rename => [ "Task", "task" ]
rename => [ "ThreadID", "thread-id" ]
rename => [ "Version", "version" ]
rename => [ "ProcessID", "process-id" ]
rename => [ "ProcessID", "process-id" ]
}
You can of course add any field translation you'd like to here. Hope that helps!
Re: Lowercase fields
Posted: Wed May 13, 2015 2:39 pm
by WillemDH
Just some small questions. Why does the type = "eventlog" has to be excluded if it is not in an if statement? And don't I need a break_on_match = "false"?
Re: Lowercase fields
Posted: Wed May 13, 2015 2:57 pm
by jolson
Why does the type = "eventlog" has to be excluded if it is not in an if statement?
The reason I included type = eventlog is because the filter I posted above is for use with nxlog - which by default outputs with the type field equal to eventlog. Feel free to remove 'type' entirely to process this filter against
all incoming logs.
And don't I need a break_on_match = "false"?
The break_on_match parameter is specific to the 'grok' filter - the 'mutate' filter has no such parameter. See this document for more details about the mutate filter:
http://www.logstash.net/docs/1.4.2/filters/mutate
Re: Lowercase fields
Posted: Thu May 14, 2015 11:43 am
by WillemDH
Jesse, when I tried your filter, eventlogs stopped showing up in my NLS somehow. I tuned it a bit. THis works for me:
Code: Select all
if [type] == "eventlog" {
mutate {
remove => [ "SourceModuleType", "EventTimeWritten", "EventTime", "EventReceivedTime", "EventType" ]
rename => [ "Severity", "severity_label" ]
lowercase => [ "severity_label" ]
rename => [ "SeverityValue", "severity" ]
rename => [ "Hostname", "hostname" ]
lowercase => [ "hostname" ]
}
}
Some small remark. It seems renaming the field, also capitalizes the field value...
I guess this works for me. You can close the thread.
Grtz
Willem
Re: Lowercase fields
Posted: Thu May 14, 2015 12:01 pm
by jolson
Sounds good - I'll lock the thread up.