Problem with timestamp

[piotrromaniuk]

Hello,

A few day ago i install NLS, add 3 host for test and everyting works fine. NLS receive logs from hosts syslogs. Today when i log i see that NLS receive logs from my esxi and not receive logs from linux host (it worked fine before). In log from logstash i see this:

Code: Select all

"@timestamp"=>"2020-09-02T10:44:46.000Z", "type"=>"syslog", "host"=>"0:0:0:0:0:0:0:1", "priority"=>86, "timestamp"=>"Sep  2 12:44:46", "logsource"=>"swanagiossyslog01v", "program"=>"sudo", "severity"=>6, "facility"=>10, "facility_label"=>"security/authorization", "severity_label"=>"Informational"}, "type"]}>>], :response=>{"create"=>{"_index"=>"logstash-2020.09.02", "_type"=>"syslog", "_id"=>"AXROa0aPmnhjSH9SzMRv", "status"=>400, "error"=>"MapperParsingException[failed to parse [timestamp]]; nested: MapperParsingException[failed to parse date field [Sep  2 12:44:46], tried both date format [dateOptionalTime], and timestamp number with locale []]; nested: IllegalArgumentException[Invalid format: \"Sep  2 12:44:46\"]; "}}, :level=>:warn}

Code: Select all

{:timestamp=>"2020-09-02T12:50:02.152000+0200", :message=>"Failed action. ", :status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2020.09.02", :_type=>"syslog", :_routing=>nil}, #<LogStash::Event:0x58bb1951 @metadata_accessors=#<LogStash::Util::Accessors:0x7ccf5432 @store={}, @lut={}>, @cancelled=false, @data={"message"=>"(root) CMD (/usr/lib64/sa/sa1 1 1)\n", "@version"=>"1", "@timestamp"=>"2020-09-02T10:50:01.000Z", "type"=>"syslog", "host"=>"somehost", "priority"=>78, "timestamp"=>"Sep  2 12:50:01", "logsource"=>"somehosy", "program"=>"CROND", "pid"=>"20067", "severity"=>6, "facility"=>9, "facility_label"=>"clock", "severity_label"=>"Informational"}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x2df3f722 @store={"message"=>"(root) CMD (/usr/lib64/sa/sa1 1 1)\n", "@version"=>"1", "@timestamp"=>"2020-09-02T10:50:01.000Z", "type"=>"syslog", "host"=>"someip", "priority"=>78, "timestamp"=>"Sep  2 12:50:01", "logsource"=>"somehost", "program"=>"CROND", "pid"=>"20067", "severity"=>6, "facility"=>9, "facility_label"=>"clock", "severity_label"=>"Informational"}, @lut={"severity_label"=>[{"message"=>"(root) CMD (/usr/lib64/sa/sa1 1 1)\n", "@version"=>"1", "@timestamp"=>"2020-09-02T10:50:01.000Z", "type"=>"syslog", "host"=>"someip", "priority"=>78, "timestamp"=>"Sep  2 12:50:01", "logsource"=>"somehost", "program"=>"CROND", "pid"=>"20067", "severity"=>6, "facility"=>9, "facility_label"=>"clock", "severity_label"=>"Informational"}, "severity_label"], "[program]"=>[{"message"=>"(root) CMD (/usr/lib64/sa/sa1 1 1)\n", "@version"=>"1",  "@timestamp"=>"2020-09-02T10:50:01.000Z", "type"=>"syslog", "host"=>"somehost", "priority"=>78, "timestamp"=>"Sep  2 12:50:01", "logsource"=>"somehost", "program"=>"CROND", "pid"=>"20067", "severity"=>6, "facility"=>9, "facility_label"=>"clock", "severity_label"=>"Informational"}, "type"]}>>], :response=>{"create"=>{"_index"=>"logstash-2020.09.02", "_type"=>"syslog", "_id"=>"AXROcBP1mnhjSH9SzPh6", "status"=>400, "error"=>"MapperParsingException[failed to parse [timestamp]]; nested: MapperParsingException[failed to parse date field [Sep  2 12:50:01], tried both date format [dateOptionalTime], and timestamp number with locale []]; nested: IllegalArgumentException[Invalid format: \"Sep  2 12:50:01\"]; "}}, :level=>:warn}

i comment in syslog config on client host line #$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat logstash does not see new logs. only old bad timestamps.

[cdienger]

You'll see this error if the time formats differ. You can get around this by following the steps in https://assets.nagios.com/downloads/nag ... Server.pdf.

[piotrromaniuk]

From esxi NLS see logs normally. The problem is with the linux hosts. where i have error: tried both date format [dateOptionalTime], and timestamp number with locale []]; nested: IllegalArgumentException[Invalid format: \"Sep 2 12:44:46\"]

[cdienger]

Are the logs from the Linux machines and the esxi machines going to the same input on the NLS system?