Received an event that has a different character encoding

[ssoliveira]

Hello

I'm using nxlog to send the IIS logs to my Nagios Log Server servers.

Logs are being sent, processed and displayed correctly in Kibana.

However; Today, I noticed that some logs are being lost because logstash is not able to work with some logs.

Apparently it is considering some strings, like different charset.

My setting is default.

My log files in IIS are UTF-8.

How can I manage this problem?

tcp {
type => 'eventlog'
port => 3515
codec => json {
charset => 'CP1252'
}
}

Code: Select all

{:timestamp=>"2017-08-10T22:25:15.981000-0300", :message=>"Received an event that has a different character encoding than you configured.", :text=>"{\\\"EventReceivedTime\\\":\\\"2017-08-10 22:25:16\\\",\\\"SourceModuleName\\\":\\\"iisw3c\\\",\\\"date\\\":\\\"2017-08-11\\\",\\\"time\\\":\\\"01:25:04\\\",\\\"s_sitename\\\":\\\"W3SVC2\\\",\\\"s_computername\\\":\\\"SAASPEXMBX02\\\",\\\"s_ip\\\":\\\"10.20.10.122\\\",\\\"cs_method\\\":\\\"POST\\\",\\\"cs_uri_stem\\\":\\\"/EWS/Exchange.asmx\\\",\\\"cs_uri_query\\\":null,\\\"s_port\\\":\\\"444\\\",\\\"cs_username\\\":\\\"SAASPEX\\\\\\\\mkt_brasilgrafica.co\\\",\\\"c_ip\\\":\\\"10.20.10.101\\\",\\\"cs_version\\\":\\\"HTTP/1.1\\\",\\\"cs_user_agent\\\":\\\"MacOutlook/15.36.1.170721+(Intelx64+Mac+OS+X+Vers\\xE3o+10.13+(Fase+17A330h))\\\",\\\"cs_cookie\\\":\\\"ClientId=OGRMHNKCIYXKEQLEDW;+exchangecookie=23378b5845294c28a1c6476d080d0839\\\",\\\"cs_referer\\\":null,\\\"cs_host\\\":\\\"saaspexmbx02.saaspex.local:444\\\",\\\"sc_status\\\":\\\"200\\\",\\\"sc_substatus\\\":\\\"0\\\",\\\"sc_win32_status\\\":\\\"0\\\",\\\"sc_bytes\\\":\\\"5646\\\",\\\"cs_bytes\\\":\\\"8686\\\",\\\"time_taken\\\":\\\"93\\\",\\\"FileName\\\":\\\"C:\\\\\\\\Inetpub\\\\\\\\Logs\\\\\\\\LogFiles\\\\\\\\W3SVC2\\\\\\\\u_ex170811.log\\\",\\\"EventTime\\\":\\\"2017-08-11T01:25:04.000Z\\\",\\\"message\\\":\\\"POST /EWS/Exchange.asmx\\\",\\\"Env\\\":\\\"UOLDIVEO\\\",\\\"EnvType\\\":\\\"Exchange2013\\\"}\\r", :expected_charset=>"UTF-8", :level=>:warn}

Code: Select all

{:timestamp=>"2017-08-10T22:25:15.981000-0300", :message=>"Received an event that has a different character encoding than you configured.", :text=>"{\\\"EventReceivedTime\\\":\\\"2017-08-10 22:25:16\\\",\\\"SourceModuleName\\\":\\\"iisw3c\\\",\\\"date\\\":\\\"2017-08-11\\\",\\\"time\\\":\\\"01:25:04\\\",\\\"s_sitename\\\":\\\"W3SVC2\\\",\\\"s_computername\\\":\\\"SAASPEXMBX02\\\",\\\"s_ip\\\":\\\"10.20.10.122\\\",\\\"cs_method\\\":\\\"POST\\\",\\\"cs_uri_stem\\\":\\\"/EWS/Exchange.asmx\\\",\\\"cs_uri_query\\\":null,\\\"s_port\\\":\\\"444\\\",\\\"cs_username\\\":\\\"SAASPEX\\\\\\\\mkt_brasilgrafica.co\\\",\\\"c_ip\\\":\\\"10.20.10.102\\\",\\\"cs_version\\\":\\\"HTTP/1.1\\\",\\\"cs_user_agent\\\":\\\"MacOutlook/15.36.1.170721+(Intelx64+Mac+OS+X+Vers\\xE3o+10.13+(Fase+17A330h))\\\",\\\"cs_cookie\\\":\\\"ClientId=OGRMHNKCIYXKEQLEDW;+exchangecookie=23378b5845294c28a1c6476d080d0839\\\",\\\"cs_referer\\\":null,\\\"cs_host\\\":\\\"saaspexmbx02.saaspex.local:444\\\",\\\"sc_status\\\":\\\"200\\\",\\\"sc_substatus\\\":\\\"0\\\",\\\"sc_win32_status\\\":\\\"0\\\",\\\"sc_bytes\\\":\\\"1934\\\",\\\"cs_bytes\\\":\\\"8426\\\",\\\"time_taken\\\":\\\"15\\\",\\\"FileName\\\":\\\"C:\\\\\\\\Inetpub\\\\\\\\Logs\\\\\\\\LogFiles\\\\\\\\W3SVC2\\\\\\\\u_ex170811.log\\\",\\\"EventTime\\\":\\\"2017-08-11T01:25:04.000Z\\\",\\\"message\\\":\\\"POST /EWS/Exchange.asmx\\\",\\\"Env\\\":\\\"UOLDIVEO\\\",\\\"EnvType\\\":\\\"Exchange2013\\\"}\\r", :expected_charset=>"UTF-8", :level=>:warn}

Code: Select all

{:timestamp=>"2017-08-10T22:25:15.983000-0300", :message=>"Received an event that has a different character encoding than you configured.", :text=>"{\\\"EventReceivedTime\\\":\\\"2017-08-10 22:25:16\\\",\\\"SourceModuleName\\\":\\\"iisw3c\\\",\\\"date\\\":\\\"2017-08-11\\\",\\\"time\\\":\\\"01:25:04\\\",\\\"s_sitename\\\":\\\"W3SVC2\\\",\\\"s_computername\\\":\\\"SAASPEXMBX02\\\",\\\"s_ip\\\":\\\"10.20.10.122\\\",\\\"cs_method\\\":\\\"POST\\\",\\\"cs_uri_stem\\\":\\\"/EWS/Exchange.asmx\\\",\\\"cs_uri_query\\\":null,\\\"s_port\\\":\\\"444\\\",\\\"cs_username\\\":\\\"SAASPEX\\\\\\\\mkt_brasilgrafica.co\\\",\\\"c_ip\\\":\\\"10.20.10.102\\\",\\\"cs_version\\\":\\\"HTTP/1.1\\\",\\\"cs_user_agent\\\":\\\"MacOutlook/15.36.1.170721+(Intelx64+Mac+OS+X+Vers\\xE3o+10.13+(Fase+17A330h))\\\",\\\"cs_cookie\\\":\\\"ClientId=OGRMHNKCIYXKEQLEDW;+exchangecookie=23378b5845294c28a1c6476d080d0839\\\",\\\"cs_referer\\\":null,\\\"cs_host\\\":\\\"saaspexmbx02.saaspex.local:444\\\",\\\"sc_status\\\":\\\"200\\\",\\\"sc_substatus\\\":\\\"0\\\",\\\"sc_win32_status\\\":\\\"0\\\",\\\"sc_bytes\\\":\\\"2574\\\",\\\"cs_bytes\\\":\\\"6866\\\",\\\"time_taken\\\":\\\"31\\\",\\\"FileName\\\":\\\"C:\\\\\\\\Inetpub\\\\\\\\Logs\\\\\\\\LogFiles\\\\\\\\W3SVC2\\\\\\\\u_ex170811.log\\\",\\\"EventTime\\\":\\\"2017-08-11T01:25:04.000Z\\\",\\\"message\\\":\\\"POST /EWS/Exchange.asmx\\\",\\\"Env\\\":\\\"UOLDIVEO\\\",\\\"EnvType\\\":\\\"Exchange2013\\\"}\\r", :expected_charset=>"UTF-8", :level=>:warn}

[cdienger]

Hi ssoliveria,

Can you post the nxlog.conf as well as the raw log lines that caused this error? It looks like you can find them in C:\Inetpub\Logs\LogFiles\W3SVC2\u_ex170811.log.

[ssoliveira]

Hello,

Follow the configuration files.

There are few lines that have this problem.
From what I've analyzed, errors occur when the user agent is from an "Apple MAC" computer. Apparently there is some character that he can not manage.

The configuration file is apparently correct, including the line to ignore the UTF-8 header

Attached is an example file, which has the lines "MacOutlook/15.36.1.170721+(Intelx64+Mac+OS+X+Versão+10.13+(Fase+17A330h))"

Code: Select all

define ROOT C:\Program Files (x86)\nxlog
define CERT %ROOT%\cert
     
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
     
######################################## 
# Modules                              # 
######################################## 

<Processor pattern>
    Module pm_pattern
    PatternFile %ROOT%\conf\patterndb.xml
</Processor>
    
<Extension json>
    Module xm_json
</Extension>
     
<Extension syslog>
    Module xm_syslog
</Extension>

<Extension ExtIISW3C> 
    Module xm_csv 
    Fields $date,$time,$s_sitename,$s_computername,$s_ip,$cs_method,$cs_uri_stem,$cs_uri_query,$s_port,$cs_username,$c_ip,$cs_version,$cs_user_agent,$cs_cookie,$cs_referer,$cs_host,$sc_status,$sc_substatus,$sc_win32_status,$sc_bytes,$cs_bytes,$time_taken
    Delimiter ' '
    QuoteChar '"'
    EscapeControl FALSE
    UndefValue -
</Extension>

######################################## 
# Inputs                               # 
######################################## 

<Input internal>
    Module im_internal
</Input>
     
<Input filenx>
    Module im_file
    File '%ROOT%\data\nxlog.log'
    SavePos TRUE
    Exec $Message = $raw_event;
</Input>
     
<Input eventlog>
    Module im_msvistalog
</Input>

<Input iisw3c>
    Module im_file
    File "C:\Inetpub\Logs\LogFiles\*ex*.log"
    SavePos TRUE
    Recursive TRUE
    InputType LineBased
    Exec if file_name() !~ /W3SVC/ drop();
    Exec if $raw_event =~ /^#/ drop(); 
    Exec if $raw_event =~ /^\xEF\xBB\xBF#/ drop();

    Exec ExtIISW3C->parse_csv();
    Exec $FileName = file_name();
    Exec $EventTime = strftime(parsedate($date + " " + $time), "%Y-%m-%dT%H:%M:%S.000Z");
    Exec $Message = $cs_method + " " + $cs_uri_stem;
    Exec delete($SourceModuleType);
</Input>

######################################## 
# Outputs                              # 
######################################## 
    
<Output out>
    Module om_tcp
    # GLETE(10.154.4.103)|TAMBORE(10.154.9.209)
    Host 10.154.9.209
    Port 3515

    Exec $Env     = "UOLDIVEO";
    Exec $EnvType = "Exchange2013";
    	
    Exec rename_field("Message","message");
    Exec $raw_event = to_json();
</Output>
    
<Route 1>
    Path eventlog => pattern => out
</Route>

<Route 2>
    Path internal, filenx, iisw3c => out
</Route>

Code: Select all

<?xml version='1.0' encoding='UTF-8'?>
<patterndb>
 <created>2017-01-09 19:00:00</created>
 <version>1</version>
    <group>
        <name>eventlog</name>
        <id>1</id>
        <pattern>
            <id>1</id>
            <name>Drop Success Logon Of HealthMonitor</name>
            <matchfield>
                <name>EventType</name>
                <type>exact</type>
                <value>AUDIT_SUCCESS</value>
            </matchfield>
            <matchfield>
                <name>TargetUserName</name>
                <type>REGEXP</type>
                <value>HealthMonitor|^HealthMailbox*</value>
            </matchfield>
            <exec>
                drop();
            </exec>
        </pattern>
        <pattern>
            <id>2</id>
            <name>Drop Massive Info Events</name>
            <matchfield>
                <name>EventType</name>
                <type>exact</type>
                <value>INFO</value>
            </matchfield>
            <matchfield>
                <name>EventID</name>
                <type>REGEXP</type>
                <value>2|3|4|5|6|25|26|27|28|29</value>
            </matchfield>
            <exec>
                drop();
            </exec>
         </pattern>
	 <pattern> 
            <id>3</id>
            <name>Drop Massive Audit Events</name>
            <matchfield>
                <name>EventType</name>
                <type>exact</type>
                <value>AUDIT_SUCCESS</value>
            </matchfield>
            <matchfield>
                <name>EventID</name>
                <type>REGEXP</type>
                <value>4624|4634|4648|4672|5156|5158|4656|4658</value>
            </matchfield>
            <exec>
                drop();
            </exec>
        </pattern>
    </group>
</patterndb>

[ssoliveira]

Please disregard the attached file (deleted)

I edited the file in the "notepad" to remove rows (due to file size). And when I saved it on the "notepad", the editor changed the file's enconding.

The file is large; And I can not attach to the forum.

I'll make it available in another location, and attach the URL.

[ssoliveira]

Please try to download it through Google Drive.

https://drive.google.com/open?id=0BySaW ... nROVTVzYzg

Compressed file in RAR

[cdienger]

It's failing the parse the ã character as it appears to be ANSI and not UTF-8.

You should create two inputs on the NLS side - one for Windows event logs and one for the IIS logs. I would use the default "Import Files - Raw(Default)" which looks like:

Code: Select all

tcp {
    type => 'import_raw'
    tags => 'import_raw'
    port => 2056
}

and configure nxlog with a new port:

Code: Select all

    <Output out2>
        Module om_tcp
        # GLETE(10.154.4.103)|TAMBORE(10.154.9.209)
        Host 10.154.9.209
        Port 2056

        Exec $Env     = "UOLDIVEO";
        Exec $EnvType = "Exchange2013";
           
        Exec rename_field("Message","message");
        Exec $raw_event = to_json();
    </Output>

and modify route 2:

Code: Select all

    <Route 2>
        Path internal, filenx, iisw3c => out2
    </Route>

[ssoliveira]

Hi,

I configured as instructed, but the logs were not converted from JSON

I believe it is necessary to parameterize the input "codec => json", correct?

But which charset should I configure?

RAW: Attached image

tcp {
type => 'eventlog'
port => 3515
codec => json {
charset => 'CP1252'
}
}

tcp {
type => 'raw'
tags => 'raw'
port => 3516
}

[tacolover101]

what happens if you remove the charset? it should run fine with just codec set to json.

[mcapra]

But which charset should I configure?

The one in-use by the Windows machine. I would recommend consulting with your Windows administrator for suggestions.

Typically this command run can tell you what codepage the Windows machine is using:

Code: Select all

chcp

Some more modern Windows machines are using 437 rather than 1252, for example. Yours may be using another. But Logstash needs to know which one is being used to interpret the data correctly.

[cdienger]

Hi ssoliveira,

My previous toying around with the problematic logline line brought me to the wrong encoding conclusion. Were mcapra's or tacolover101's suggestion able to help?