Lowercase fields

[WillemDH]

Hello,

How could I lowercase all Fields? At the moment some inputs have Hostname, other hostname, same for ipadress and IpAddress and many more. I'd like to streamline the fields a bit, so making them all lowercase would be the best idea I think. I know I can use mutate to lowercase a field value, but this doesn't work for the Field itself.

Grtz

Willem

[jolson]

It looks like this may be possible using a ruby filter. Please add this to the top of your filter list:

Code: Select all

filter {
  ruby {
    code => "event.overwrite(LogStash::Event.new(event.to_hash.inject({}){|r, (k, v)| r[k.downcase] = v; r }))"
  }
}

This should lowercase your field names. Let me know if it works for you.

[WillemDH]

Jesse,

Code: Select all

    ruby {
        code => "event.overwrite(LogStash::Event.new(event.to_hash.inject({}){|r, (k, v)| r[k.downcase] = v; r }))"
    } 

Does not seem to work.

Stumbled on these:

https://groups.google.com/forum/#!topic ... OkOs3jWdxk
https://logstash.jira.com/browse/LOGSTASH-732

We were on logstash 1.4 right? So I'm guessing this is not possible at the moment. It might be possible to do it on the nxlog side?

Grtz

Willem

[jolson]

Yep - looks like version 1.4 broke that code. The github issue for this request is here: https://github.com/elastic/logstash/issues/2526 - looks like the last activity was 29 days ago.

We can do this manually on the logstash-end of things for now as a workaround. Please add the following to the top of your filter-chain.

Code: Select all

mutate {
    type   => "eventlog"
    rename => [ "Message", "message" ]
    rename => [ "Severity", "severity" ]
    rename => [ "Hostname", "hostname" ]
    rename => [ "Channel", "channel" ]
    rename => [ "EventID", "eventID" ]
    rename => [ "EventReceivedTime", "event-received-time" ]
    rename => [ "EventTime", "event-time" ]
    rename => [ "EventType", "event-type" ]
    rename => [ "RecordNumber", "record-number" ]
    rename => [ "ProcessID", "process-id" ]
    rename => [ "OpcodeValue", "opcode-value" ]
    rename => [ "SourceModuleName", "source-module-name" ]
    rename => [ "SeverityValue", "severity-value" ]
    rename => [ "SourceModuleType", "source-module-type" ]
    rename => [ "ProviderGuid", "provider-guid" ]
    rename => [ "SourceName", "source-name" ]
    rename => [ "Task", "task" ]
    rename => [ "ThreadID", "thread-id" ]
    rename => [ "Version", "version" ]
    rename => [ "ProcessID", "process-id" ]
    rename => [ "ProcessID", "process-id" ]
}

You can of course add any field translation you'd like to here. Hope that helps!

[WillemDH]

Just some small questions. Why does the type = "eventlog" has to be excluded if it is not in an if statement? And don't I need a break_on_match = "false"?

[jolson]

Why does the type = "eventlog" has to be excluded if it is not in an if statement?

The reason I included type = eventlog is because the filter I posted above is for use with nxlog - which by default outputs with the type field equal to eventlog. Feel free to remove 'type' entirely to process this filter against all incoming logs.

And don't I need a break_on_match = "false"?

The break_on_match parameter is specific to the 'grok' filter - the 'mutate' filter has no such parameter. See this document for more details about the mutate filter: http://www.logstash.net/docs/1.4.2/filters/mutate

[WillemDH]

Jesse, when I tried your filter, eventlogs stopped showing up in my NLS somehow. I tuned it a bit. THis works for me:

Code: Select all

if [type] == "eventlog" {
    mutate {
        remove => [ "SourceModuleType", "EventTimeWritten", "EventTime", "EventReceivedTime", "EventType" ]
        rename => [ "Severity", "severity_label" ]
        lowercase => [ "severity_label" ]
        rename => [ "SeverityValue", "severity" ]
        rename => [ "Hostname", "hostname" ]
        lowercase => [ "hostname" ]
    }
}

Some small remark. It seems renaming the field, also capitalizes the field value...

I guess this works for me. You can close the thread.

Grtz

Willem

[jolson]

Sounds good - I'll lock the thread up.