Logs Not Appearing with Failed to Parse Date Error

Problem Description

Logs are not coming in and you see an error like this in the Logstash log:

:response=>{"create"=>{"_index"=>"logstash-2019.12.13", "_type"=>"syslog", "_id"=>"AW8Ab04im8e-JsUH61
c5", "status"=>400, "error"=>"MapperParsingException[failed to parse [timestamp8601]]; nested: MapperParsingException[
failed to parse date field [2019-12-13 18:04:52.81], tried both date format [dateOptionalTime], and timestamp number w
ith locale []]; nested: IllegalArgumentException[Invalid format: \"2019-12-13 18:04:52.81\" is malformed at \" 18:04:5
2.81\"]; "}}

You also have multiple identical input types (ex. syslog) in your input configuration, something like the following:

syslog {
  port => xxx
  type => 'syslog'
}

syslog {
    port => xxxx
    type => 'syslog'
    tags => 'Linux-Max'
}

Solution 

Logs coming in on the same input need to use the same formatting so that parsing works properly. The format that the input expects is somewhat flexible initially but once the first message comes in, the format is set(until the next day's index is created). For example, the syslog input expects all input to follow rfc3164 which can send a message like:

<0>1990 Oct 22 10:52:01 TZ-6 scapegoat.dmz.example.org 10.1.2.3 sched[0]: That's All Folks!

If a message with a different date format then comes in([2019-12-13 18:04:52.81]) you'll see a message logged like the one you see.

The fix is to make sure that all devices use the same date format or configure another input for these devices. For example:

syslog {
    port => xxxx
    type => 'alternative-syslog'
    tags => 'alternative Linux-Max'
}

See the Advanced Configuration section of the Sending ESXi Logs to Nagios Log Server document for more information on how to set the type field for specific hosts(in cases where different logs formats need to come in on the same input.