Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Received an event that has a different character encoding

https://support.nagios.com/forum/viewtopic.php?t=45118

Page 1 of 2

Received an event that has a different character encoding

Posted: Thu Aug 10, 2017 8:35 pm
by ssoliveira
Hello

I'm using nxlog to send the IIS logs to my Nagios Log Server servers.

Logs are being sent, processed and displayed correctly in Kibana.

However; Today, I noticed that some logs are being lost because logstash is not able to work with some logs.

Apparently it is considering some strings, like different charset.

My setting is default.

My log files in IIS are UTF-8.

How can I manage this problem?

tcp {
type => 'eventlog'
port => 3515
codec => json {
charset => 'CP1252'
}
}

Code: Select all

{:timestamp=>"2017-08-10T22:25:15.981000-0300", :message=>"Received an event that has a different character encoding than you configured.", :text=>"{\\\"EventReceivedTime\\\":\\\"2017-08-10 22:25:16\\\",\\\"SourceModuleName\\\":\\\"iisw3c\\\",\\\"date\\\":\\\"2017-08-11\\\",\\\"time\\\":\\\"01:25:04\\\",\\\"s_sitename\\\":\\\"W3SVC2\\\",\\\"s_computername\\\":\\\"SAASPEXMBX02\\\",\\\"s_ip\\\":\\\"10.20.10.122\\\",\\\"cs_method\\\":\\\"POST\\\",\\\"cs_uri_stem\\\":\\\"/EWS/Exchange.asmx\\\",\\\"cs_uri_query\\\":null,\\\"s_port\\\":\\\"444\\\",\\\"cs_username\\\":\\\"SAASPEX\\\\\\\\mkt_brasilgrafica.co\\\",\\\"c_ip\\\":\\\"10.20.10.101\\\",\\\"cs_version\\\":\\\"HTTP/1.1\\\",\\\"cs_user_agent\\\":\\\"MacOutlook/15.36.1.170721+(Intelx64+Mac+OS+X+Vers\\xE3o+10.13+(Fase+17A330h))\\\",\\\"cs_cookie\\\":\\\"ClientId=OGRMHNKCIYXKEQLEDW;+exchangecookie=23378b5845294c28a1c6476d080d0839\\\",\\\"cs_referer\\\":null,\\\"cs_host\\\":\\\"saaspexmbx02.saaspex.local:444\\\",\\\"sc_status\\\":\\\"200\\\",\\\"sc_substatus\\\":\\\"0\\\",\\\"sc_win32_status\\\":\\\"0\\\",\\\"sc_bytes\\\":\\\"5646\\\",\\\"cs_bytes\\\":\\\"8686\\\",\\\"time_taken\\\":\\\"93\\\",\\\"FileName\\\":\\\"C:\\\\\\\\Inetpub\\\\\\\\Logs\\\\\\\\LogFiles\\\\\\\\W3SVC2\\\\\\\\u_ex170811.log\\\",\\\"EventTime\\\":\\\"2017-08-11T01:25:04.000Z\\\",\\\"message\\\":\\\"POST /EWS/Exchange.asmx\\\",\\\"Env\\\":\\\"UOLDIVEO\\\",\\\"EnvType\\\":\\\"Exchange2013\\\"}\\r", :expected_charset=>"UTF-8", :level=>:warn}

Code: Select all

{:timestamp=>"2017-08-10T22:25:15.981000-0300", :message=>"Received an event that has a different character encoding than you configured.", :text=>"{\\\"EventReceivedTime\\\":\\\"2017-08-10 22:25:16\\\",\\\"SourceModuleName\\\":\\\"iisw3c\\\",\\\"date\\\":\\\"2017-08-11\\\",\\\"time\\\":\\\"01:25:04\\\",\\\"s_sitename\\\":\\\"W3SVC2\\\",\\\"s_computername\\\":\\\"SAASPEXMBX02\\\",\\\"s_ip\\\":\\\"10.20.10.122\\\",\\\"cs_method\\\":\\\"POST\\\",\\\"cs_uri_stem\\\":\\\"/EWS/Exchange.asmx\\\",\\\"cs_uri_query\\\":null,\\\"s_port\\\":\\\"444\\\",\\\"cs_username\\\":\\\"SAASPEX\\\\\\\\mkt_brasilgrafica.co\\\",\\\"c_ip\\\":\\\"10.20.10.102\\\",\\\"cs_version\\\":\\\"HTTP/1.1\\\",\\\"cs_user_agent\\\":\\\"MacOutlook/15.36.1.170721+(Intelx64+Mac+OS+X+Vers\\xE3o+10.13+(Fase+17A330h))\\\",\\\"cs_cookie\\\":\\\"ClientId=OGRMHNKCIYXKEQLEDW;+exchangecookie=23378b5845294c28a1c6476d080d0839\\\",\\\"cs_referer\\\":null,\\\"cs_host\\\":\\\"saaspexmbx02.saaspex.local:444\\\",\\\"sc_status\\\":\\\"200\\\",\\\"sc_substatus\\\":\\\"0\\\",\\\"sc_win32_status\\\":\\\"0\\\",\\\"sc_bytes\\\":\\\"1934\\\",\\\"cs_bytes\\\":\\\"8426\\\",\\\"time_taken\\\":\\\"15\\\",\\\"FileName\\\":\\\"C:\\\\\\\\Inetpub\\\\\\\\Logs\\\\\\\\LogFiles\\\\\\\\W3SVC2\\\\\\\\u_ex170811.log\\\",\\\"EventTime\\\":\\\"2017-08-11T01:25:04.000Z\\\",\\\"message\\\":\\\"POST /EWS/Exchange.asmx\\\",\\\"Env\\\":\\\"UOLDIVEO\\\",\\\"EnvType\\\":\\\"Exchange2013\\\"}\\r", :expected_charset=>"UTF-8", :level=>:warn}

Code: Select all

{:timestamp=>"2017-08-10T22:25:15.983000-0300", :message=>"Received an event that has a different character encoding than you configured.", :text=>"{\\\"EventReceivedTime\\\":\\\"2017-08-10 22:25:16\\\",\\\"SourceModuleName\\\":\\\"iisw3c\\\",\\\"date\\\":\\\"2017-08-11\\\",\\\"time\\\":\\\"01:25:04\\\",\\\"s_sitename\\\":\\\"W3SVC2\\\",\\\"s_computername\\\":\\\"SAASPEXMBX02\\\",\\\"s_ip\\\":\\\"10.20.10.122\\\",\\\"cs_method\\\":\\\"POST\\\",\\\"cs_uri_stem\\\":\\\"/EWS/Exchange.asmx\\\",\\\"cs_uri_query\\\":null,\\\"s_port\\\":\\\"444\\\",\\\"cs_username\\\":\\\"SAASPEX\\\\\\\\mkt_brasilgrafica.co\\\",\\\"c_ip\\\":\\\"10.20.10.102\\\",\\\"cs_version\\\":\\\"HTTP/1.1\\\",\\\"cs_user_agent\\\":\\\"MacOutlook/15.36.1.170721+(Intelx64+Mac+OS+X+Vers\\xE3o+10.13+(Fase+17A330h))\\\",\\\"cs_cookie\\\":\\\"ClientId=OGRMHNKCIYXKEQLEDW;+exchangecookie=23378b5845294c28a1c6476d080d0839\\\",\\\"cs_referer\\\":null,\\\"cs_host\\\":\\\"saaspexmbx02.saaspex.local:444\\\",\\\"sc_status\\\":\\\"200\\\",\\\"sc_substatus\\\":\\\"0\\\",\\\"sc_win32_status\\\":\\\"0\\\",\\\"sc_bytes\\\":\\\"2574\\\",\\\"cs_bytes\\\":\\\"6866\\\",\\\"time_taken\\\":\\\"31\\\",\\\"FileName\\\":\\\"C:\\\\\\\\Inetpub\\\\\\\\Logs\\\\\\\\LogFiles\\\\\\\\W3SVC2\\\\\\\\u_ex170811.log\\\",\\\"EventTime\\\":\\\"2017-08-11T01:25:04.000Z\\\",\\\"message\\\":\\\"POST /EWS/Exchange.asmx\\\",\\\"Env\\\":\\\"UOLDIVEO\\\",\\\"EnvType\\\":\\\"Exchange2013\\\"}\\r", :expected_charset=>"UTF-8", :level=>:warn}

Re: Received an event that has a different character encodin

Posted: Fri Aug 11, 2017 9:27 am
by cdienger
Hi ssoliveria,

Can you post the nxlog.conf as well as the raw log lines that caused this error? It looks like you can find them in C:\Inetpub\Logs\LogFiles\W3SVC2\u_ex170811.log.

Re: Received an event that has a different character encodin

Posted: Fri Aug 11, 2017 12:23 pm
by ssoliveira
Hello,

Follow the configuration files.

There are few lines that have this problem.
From what I've analyzed, errors occur when the user agent is from an "Apple MAC" computer. Apparently there is some character that he can not manage.

The configuration file is apparently correct, including the line to ignore the UTF-8 header

Attached is an example file, which has the lines "MacOutlook/15.36.1.170721+(Intelx64+Mac+OS+X+Versão+10.13+(Fase+17A330h))"

Code: Select all

define ROOT C:\Program Files (x86)\nxlog
define CERT %ROOT%\cert
     
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
     
######################################## 
# Modules                              # 
######################################## 

<Processor pattern>
    Module pm_pattern
    PatternFile %ROOT%\conf\patterndb.xml
</Processor>
    
<Extension json>
    Module xm_json
</Extension>
     
<Extension syslog>
    Module xm_syslog
</Extension>

<Extension ExtIISW3C> 
    Module xm_csv 
    Fields $date,$time,$s_sitename,$s_computername,$s_ip,$cs_method,$cs_uri_stem,$cs_uri_query,$s_port,$cs_username,$c_ip,$cs_version,$cs_user_agent,$cs_cookie,$cs_referer,$cs_host,$sc_status,$sc_substatus,$sc_win32_status,$sc_bytes,$cs_bytes,$time_taken
    Delimiter ' '
    QuoteChar '"'
    EscapeControl FALSE
    UndefValue -
</Extension>

######################################## 
# Inputs                               # 
######################################## 

<Input internal>
    Module im_internal
</Input>
     
<Input filenx>
    Module im_file
    File '%ROOT%\data\nxlog.log'
    SavePos TRUE
    Exec $Message = $raw_event;
</Input>
     
<Input eventlog>
    Module im_msvistalog
</Input>

<Input iisw3c>
    Module im_file
    File "C:\Inetpub\Logs\LogFiles\*ex*.log"
    SavePos TRUE
    Recursive TRUE
    InputType LineBased
    Exec if file_name() !~ /W3SVC/ drop();
    Exec if $raw_event =~ /^#/ drop(); 
    Exec if $raw_event =~ /^\xEF\xBB\xBF#/ drop();

    Exec ExtIISW3C->parse_csv();
    Exec $FileName = file_name();
    Exec $EventTime = strftime(parsedate($date + " " + $time), "%Y-%m-%dT%H:%M:%S.000Z");
    Exec $Message = $cs_method + " " + $cs_uri_stem;
    Exec delete($SourceModuleType);
</Input>

######################################## 
# Outputs                              # 
######################################## 
    
<Output out>
    Module om_tcp
    # GLETE(10.154.4.103)|TAMBORE(10.154.9.209)
    Host 10.154.9.209
    Port 3515

    Exec $Env     = "UOLDIVEO";
    Exec $EnvType = "Exchange2013";
    	
    Exec rename_field("Message","message");
    Exec $raw_event = to_json();
</Output>
    
<Route 1>
    Path eventlog => pattern => out
</Route>

<Route 2>
    Path internal, filenx, iisw3c => out
</Route>

Code: Select all

<?xml version='1.0' encoding='UTF-8'?>
<patterndb>
 <created>2017-01-09 19:00:00</created>
 <version>1</version>
    <group>
        <name>eventlog</name>
        <id>1</id>
        <pattern>
            <id>1</id>
            <name>Drop Success Logon Of HealthMonitor</name>
            <matchfield>
                <name>EventType</name>
                <type>exact</type>
                <value>AUDIT_SUCCESS</value>
            </matchfield>
            <matchfield>
                <name>TargetUserName</name>
                <type>REGEXP</type>
                <value>HealthMonitor|^HealthMailbox*</value>
            </matchfield>
            <exec>
                drop();
            </exec>
        </pattern>
        <pattern>
            <id>2</id>
            <name>Drop Massive Info Events</name>
            <matchfield>
                <name>EventType</name>
                <type>exact</type>
                <value>INFO</value>
            </matchfield>
            <matchfield>
                <name>EventID</name>
                <type>REGEXP</type>
                <value>2|3|4|5|6|25|26|27|28|29</value>
            </matchfield>
            <exec>
                drop();
            </exec>
         </pattern>
	 <pattern> 
            <id>3</id>
            <name>Drop Massive Audit Events</name>
            <matchfield>
                <name>EventType</name>
                <type>exact</type>
                <value>AUDIT_SUCCESS</value>
            </matchfield>
            <matchfield>
                <name>EventID</name>
                <type>REGEXP</type>
                <value>4624|4634|4648|4672|5156|5158|4656|4658</value>
            </matchfield>
            <exec>
                drop();
            </exec>
        </pattern>
    </group>
</patterndb>

Re: Received an event that has a different character encodin

Posted: Fri Aug 11, 2017 12:35 pm
by ssoliveira
Please disregard the attached file (deleted)

I edited the file in the "notepad" to remove rows (due to file size). And when I saved it on the "notepad", the editor changed the file's enconding.

The file is large; And I can not attach to the forum.

I'll make it available in another location, and attach the URL.

Re: Received an event that has a different character encodin

Posted: Fri Aug 11, 2017 12:50 pm
by ssoliveira
Please try to download it through Google Drive.

https://drive.google.com/open?id=0BySaW ... nROVTVzYzg

Compressed file in RAR

Re: Received an event that has a different character encodin

Posted: Fri Aug 11, 2017 2:40 pm
by cdienger
It's failing the parse the ã character as it appears to be ANSI and not UTF-8.

You should create two inputs on the NLS side - one for Windows event logs and one for the IIS logs. I would use the default "Import Files - Raw(Default)" which looks like:

Code: Select all

tcp {
    type => 'import_raw'
    tags => 'import_raw'
    port => 2056
}
and configure nxlog with a new port:

Code: Select all

    <Output out2>
        Module om_tcp
        # GLETE(10.154.4.103)|TAMBORE(10.154.9.209)
        Host 10.154.9.209
        Port 2056

        Exec $Env     = "UOLDIVEO";
        Exec $EnvType = "Exchange2013";
           
        Exec rename_field("Message","message");
        Exec $raw_event = to_json();
    </Output>
and modify route 2:

Code: Select all

    <Route 2>
        Path internal, filenx, iisw3c => out2
    </Route>

Re: Received an event that has a different character encodin

Posted: Fri Aug 11, 2017 4:47 pm
by ssoliveira
Hi,

I configured as instructed, but the logs were not converted from JSON

I believe it is necessary to parameterize the input "codec => json", correct?

But which charset should I configure?

RAW: Attached image

tcp {
type => 'eventlog'
port => 3515
codec => json {
charset => 'CP1252'
}
}

tcp {
type => 'raw'
tags => 'raw'
port => 3516
}

Re: Received an event that has a different character encodin

Posted: Mon Aug 14, 2017 7:46 am
by tacolover101
what happens if you remove the charset? it should run fine with just codec set to json.

Re: Received an event that has a different character encodin

Posted: Mon Aug 14, 2017 9:59 am
by mcapra
ssoliveira wrote: But which charset should I configure?
The one in-use by the Windows machine. I would recommend consulting with your Windows administrator for suggestions.

Typically this command run can tell you what codepage the Windows machine is using:

Code: Select all

chcp
Some more modern Windows machines are using 437 rather than 1252, for example. Yours may be using another. But Logstash needs to know which one is being used to interpret the data correctly.

Re: Received an event that has a different character encodin

Posted: Mon Aug 14, 2017 11:04 am
by cdienger
Hi ssoliveira,

My previous toying around with the problematic logline line brought me to the wrong encoding conclusion. Were mcapra's or tacolover101's suggestion able to help?
All times are UTC-05:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited