Mutate lowercase and gsub no longer working since upgrade

[WillemDH]

Hello,

It seems the some of my filters are no longer getting applied since upgrading to R2.1.

So I'm using this filter to lowercase all eventlog type logs:

Code: Select all

if [type] == "eventlog" {
    mutate {
        remove => [ "SourceModuleType", "EventTimeWritten", "EventTime", "EventReceivedTime", "EventType" ]
        rename => [ "Severity", "severity_label" ]
        lowercase => [ "severity_label" ]
        rename => [ "SeverityValue", "severity" ]
        rename => [ "Hostname", "hostname" ]
        lowercase => [ "hostname" ]
        rename => [ "AccountName", "accountname" ]
        rename => [ "AccountType", "accounttype" ]
        rename => [ "ActivityID", "activityid" ]
        rename => [ "AuthenticationPackageName", "authenticationpackagename" ]
        rename => [ "Category", "category" ]
        rename => [ "Channel", "channel" ]
        rename => [ "ContextInfo", "contextinfo" ]
        rename => [ "Domain", "domain" ]
        rename => [ "EventID", "eventid" ]
        rename => [ "ErrorCode", "errorcode" ]
        rename => [ "FolderId", "folderid" ]
        rename => [ "hrError", "hrerror" ]
        rename => [ "IpAddress", "ipaddress" ]
        rename => [ "IpPort", "ipport" ]
        rename => [ "InstanceId", "instanceid" ]
        rename => [ "KeyLength", "keylength" ]
        rename => [ "Keywords", "keywords" ]
        rename => [ "LmPackageName", "lmpackagename" ]
        rename => [ "LogonGuid", "logonguid" ]
        rename => [ "LogonProcessName", "logonprocessname" ]
        rename => [ "LogonType", "logontype" ]
        rename => [ "Opcode", "opcode" ]
        rename => [ "OpcodeValue", "opcodevalue" ]
        lowercase => [ "opcode" ]
        rename => [ "Path", "path" ]
        rename => [ "PrivilegeList", "privilegelist" ]
        rename => [ "ProcessID", "processid" ]
        rename => [ "ProcessName", "processname" ]
        rename => [ "ProviderGuid", "providerguid" ]
        rename => [ "RecordNumber", "recordnumber" ]
        rename => [ "ResultCode", "resultcode" ]
        rename => [ "SourceModuleName", "sourcemodulename" ]
        rename => [ "SourceName", "sourcename" ]
        rename => [ "SubjectDomainName", "subjectdomainname" ]
        rename => [ "SubjectLogonId", "subjectlogonid" ]
        rename => [ "SubjectUserName", "subjectusername" ]
        rename => [ "SubjectUserSid", "subjectusersid" ]
        rename => [ "TargetDomainName", "targetdomainname" ]
        rename => [ "TargetInfo", "targetinfo" ]
        rename => [ "TargetLogonId", "targetlogonid" ]
        rename => [ "TargetServerName", "targetservername" ]
        rename => [ "TargetUserName", "targetusername" ]
        rename => [ "TargetUserSid", "targetusersid" ]
        rename => [ "Task", "task" ]
        rename => [ "TaskInstanceId", "taskinstanceid" ]
        rename => [ "TaskName", "taskname" ]
        rename => [ "ThreadID", "threadid" ]
        rename => [ "TransmittedServices", "transmittedservices" ]
        rename => [ "UserContent", "usercontent" ]
        rename => [ "UserID", "userid" ]
        rename => [ "Version", "version" ]
        rename => [ "WorkstationName", "workstationname" ]
    }
    mutate {
        gsub => [ 
            "severity_label", "info", "informational"
        ]
    }
}

Before the upgrade the values of severity_label were properly lowercased. The 1.5.x documentation still gives the same examples https://www.elastic.co/guide/en/logstas ... utate.html

The gsub mutate part is also no longer working it seems. Any tips how to handle lowercase and gsub in R2.1?

Grtz

Willem

Code: Select all

filter {
  mutate {
    lowercase => [ "fieldname" ]
  }
}

[WillemDH]

Found a workaround here : https://github.com/logstash-plugins/log ... /issues/33

Code: Select all

ruby {
        code => "event['severity_label'] = event['severity_label'].to_s.force_encoding('ISO-8859-1').downcase"
    }

Just need to find a workaround for the gsub mutate filter.

[WillemDH]

it appears the gsub does still work. It didn't work for me as the lowercase wasn't working, the field was wrong in the gsub.

Code: Select all

https://github.com/logstash-plugins/logstash-filter-mutate/issues/33

[jolson]

You're doing all of the work for us!

I'll keep an eye on the bug report you've linked - it looks like that is the root of your problem. The Logstash team is pretty responsive to issues like this, I'm hoping they can get a bug fix out soon.

[WillemDH]

Well half of the time I'm writing a post on the Nagios Support forum, I find the solution or a workaround half way while writing the post lol... But let's indeed keep this open until the issue is fixed.

[jolson]

Sounds good. I'll be checking on the bug every morning that I arrive - I have a couple of gsub filters on my test cluster and didn't notice that they weren't working until this point.

[WillemDH]

Jesse,

I'm asked to:

This looks like a jruby bug. Its fixed in jruby 9.0.0.0 - jruby/jruby#2847

Please try setting the JRUBY_OPTS env var like so JRUBY_OPTS="-Ku" and report back.

Could you tell me how to do this?

Grtz

Willem

[jolson]

Log into your NLS instance that you'd like to make this change on. Log in as the user that logstash is running under (by default 'nagios' - 'root' if you've made changes).

Run the following commands to temporarily set the environmental variable and restart logstash:

Code: Select all

export JRUBY_OPTS="-Ku"
service logstash restart

If this does resolve your problem, you can set the environmental variable permanently like so:

Code: Select all

echo "export JRUBY_OPTS="-Ku"" >> ~/.bashrc

[WillemDH]

Hey Jesse,

Sorry for not responding for a long time. Holidays and stuff... I have the Ruby workaround working, but I know there is a fix in the meantime for the lowercase issue. It just tested if it was solved in R2.2, but that does not seem the case.
https://github.com/logstash-plugins/log ... te/pull/42

Is there some way to implement this fix with just an update of the mutate plugin? Or does NLS need to be updated for that to work?

Grtz

Willem

[jolson]

Please note that this procedure is considered experimental - and while I have performed it successfully on my test box with no visible problems, that does not mean the procedure is flawless. It hasn't been vetted by the Nagios team - take backups and proceed at your own risk.

To upgrade to a version of Logstash containing the patch for this issue, please follow these instructions:
Stop logstash:

Code: Select all

service logstash stop

Change to your tmp dir:

Code: Select all

cd /tmp

Download Logstash 1.5.3, untar it, and change directory to it:

Code: Select all

wget https://download.elastic.co/logstash/logstash/logstash-1.5.4.tar.gz
tar zxf logstash-1.5.4.tar.gz
cd logstash-1.5.4

Copy all logstash files to our default logstash location:

Code: Select all

/bin/cp -R -p * /usr/local/nagioslogserver/logstash/ #calling copy with absolute path to avoid cp -i alias problems

Change permissions appropriately:

Code: Select all

chown -R nagios:nagios /usr/local/nagioslogserver/logstash
chmod -R g+w /usr/local/nagioslogserver/logstash/etc

Start Logstash:

Code: Select all

service logstash start

Please ensure there are no horrible errors in the log:

Code: Select all

cat /var/log/logstash/logstash.log

Note that your configurations will be retained. I tested Apply Configuration, Verify, changing the status via the WebGUI, and ensured that logstash was collecting the same amount of logs as it was before I upgraded. I think that this is a generally safe procedure, but I may have missed something.

A big issue that I noted with the lowercase mutate filter is that if the field is already lowercase and it attempts to run through the lowercase mutate filter, the entire field will be dropped. I think this is a new bug, and it's important to be aware of.

Let me know how this works for you - I didn't have any problems besides the weird field-dropping.

EDIT: The following bug is likely related: https://github.com/logstash-plugins/log ... te/pull/45

EDIT2: Updating the mutate filter after running the above procedure fixes the lowercase removal problem!

Code: Select all

/usr/local/nagioslogserver/logstash/bin/plugin update logstash-filter-mutate

Code: Select all

service logstash restart