logstash.log filling up again

[rferebee]

Good morning team,

Could someone help me out. Trying to identify why my Log Server environment is having trouble processing a large amount of logs. I logged into one of my nodes and the logstash.log file was 42GBs... I'm seeing thousands of entries for this:

Code: Select all

{:timestamp=>"2021-06-23T07:30:24.225000-0700", :message=>"Received an event that has a different character encoding than you configured.", :text=>"{\\\"EventReceivedTime\\\":\\\"2021-06-23 07:30:24\\\",\\\"SourceModuleName\\\":\\\"in\\\",\\\"SourceModuleType\\\":\\\"im_file\\\",\\\"ThreadId\\\":\\\"0F84\\\",\\\"Context\\\":\\\"PACKET\\\",\\\"InternalPacketIdentifier\\\":\\\"000000F2DE3785D0\\\",\\\"Protocol\\\":\\\"UDP\\\",\\\"SendReceiveIndicator\\\":\\\"Snd\\\",\\\"RemoteIP\\\":\\\"10.131.66.2\\\",\\\"Xid\\\":\\\"b438\\\",\\\"QueryType\\\":\\\"R\\\",\\\"OpCode\\\":\\\"Q\\\",\\\"QFlags\\\":\\\"[8281   DR SERVFAIL]\\\",\\\"QuestionType\\\":\\\"PTR\\\",\\\"QuestionName\\\":\\\"(1)r(7)_dns-sd(4)_udp(3)\\xE0\\xF5g(0)\\\",\\\"LogInfo\\\":\\\"UDP response info at 000000F2DE3785D0\\\",\\\"Socket\\\":\\\"488\\\",\\\"RemoteAddr\\\":\\\"10.131.66.2\\\",\\\"PortNum\\\":\\\"62741\\\",\\\"TimeQuery\\\":\\\"649054\\\",\\\"Queued\\\":\\\"649054\\\",\\\"Expire\\\":\\\"649064\\\",\\\"BufLen\\\":\\\"4000\\\",\\\"MsgLen\\\":\\\"47\\\",\\\"EventTime\\\":null,\\\"message\\\":\\\"XID       0xb438\\\\r\\\\n    Flags     0x8182\\\\r\\\\n      QR        1 (RESPONSE)\\\\r\\\\n      OPCODE    0 (QUERY)\\\\r\\\\n      AA        0\\\\r\\\\n      TC        0\\\\r\\\\n      RD        1\\\\r\\\\n      RA        1\\\\r\\\\n      Z         0\\\\r\\\\n      CD        0\\\\r\\\\n      AD        0\\\\r\\\\n      RCODE     2 (SERVFAIL)\\\\r\\\\n    QCOUNT    1\\\\r\\\\n    ACOUNT    0\\\\r\\\\n    NSCOUNT   0\\\\r\\\\n    ARCOUNT   1\\\\r\\\\n    QUESTION SECTION:\\\\r\\\\n    Offset = 0x000c, RR count = 0\\\\r\\\\n    Name      \\\\\\\"(1)r(7)_dns-sd(4)_udp(3)\\xE0\\xF5g(0)\\\\\\\"\\\\r\\\\n      QTYPE   PTR (12)\\\\r\\\\n      QCLASS  1\\\\r\\\\n    ANSWER SECTION:\\\\r\\\\n      empty\\\\r\\\\n    AUTHORITY SECTION:\\\\r\\\\n      empty\\\\r\\\\n    ADDITIONAL SECTION:\\\\r\\\\n    Offset = 0x0024, RR count = 0\\\\r\\\\n    Name      \\\\\\\"(0)\\\\\\\"\\\\r\\\\n      TYPE   OPT  (41)\\\\r\\\\n      CLASS  4000\\\\r\\\\n      TTL    32768\\\\r\\\\n      DLEN   0\\\\r\\\\n      DATA   \\\\r\\\\n\\\\t\\\\tBuffer Size  = 4000\\\\r\\\\n\\\\t\\\\tRcode Ext    = 0\\\\r\\\\n\\\\t\\\\tRcode Full   = 2\\\\r\\\\n\\\\t\\\\tVersion      = 0\\\\r\\\\n\\\\t\\\\tFlags        = 80 DO\\\\r\\\\n\\\"}\\r", :expected_charset=>"UTF-8", :level=>:warn}

Forgive me, I don't remember exactly what it is you need to look at to determine where the issue is. Let me know and I'll get it to you ASAP. Thank you.

[ssax]

It's saying that logstash was expecting a UTF-8 character encoded message to the input but the message it recevied was not in UTF-8 format so it's complaining about it.

What input did that log come into? Please post that input configuration.

[rferebee]

I believe these are coming from our DNS logging.

I'll PM you the config file we're using.

[ssax]

Responded in PM, I think you need to set the charset on the input in Log Server:

Code: Select all

    tcp {
        type => 'dnslogs'
        port => XXXX
        codec => json {
            charset => 'CP1252'
        }
    }

See if that fixes it.