Page 1 of 2
Logstash timestamp error
Posted: Thu Jan 28, 2016 5:19 pm
by GhostRider2110
I'm trying to figure out how to eleminate this error from my logstash.log file:
{:timestamp=>"2016-01-28T03:51:09.340000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Jan 28 03:51:09", :exception=>java.lang.IllegalArgumentException: Invalid format: "Jan 28 03:51:09",
:level=>:warn}
How can I tell what log is sending it and how to correct the parsing?
Nagios Log Server • 1.4.0
2 Cluster system
71 systems feeding logs in.
Thanks in advanced...
--Mitch
Re: Logstash timestamp error
Posted: Thu Jan 28, 2016 5:43 pm
by jolson
Hi Mitch!
My bet is that the malformed date is hitting one of your 'syslog' inputs. The syslog input forces a particular date format. We can avoid this by either:
1. Sending the logs to a different input (tcp or udp) that has its own proper timestamp parser
-or-
2. Changing your syslog parsers to tcp/udp parsers so that the timestamp format restriction is lifted.
I recommend number 1, but that requires figuring out which host is sending the malformed logs to begin with. To figure this out, we can query for that particular log:
Code: Select all
timestamp:03 AND timestamp:51 AND timestamp:09
Let me know how you'd like to proceed. Thanks!
Re: Logstash timestamp error
Posted: Thu Jan 28, 2016 6:00 pm
by GhostRider2110
Thanks,
I would like to figure out where it is coming from and then adjust things to parse it properly. It is probably coming from a few servers running a custom python/wsgi app.
BTW, it was great meeting you guys at the World Conf. Really enjoyed it and plan on attending 2016...
See-ya
Mitch
Re: Logstash timestamp error
Posted: Fri Jan 29, 2016 3:10 pm
by jolson
BTW, it was great meeting you guys at the World Conf. Really enjoyed it and plan on attending 2016...
Glad you enjoyed it! It was a fantastic time.
I would like to figure out where it is coming from and then adjust things to parse it properly
Did the query I suggested above help you out? Let me know if you need some more direction - thanks!
Re: Logstash timestamp error
Posted: Fri Jan 29, 2016 3:38 pm
by GhostRider2110
Grabbed a newer entry from the logs
Code: Select all
timestamp:15 AND timestamp:33 AND timestamp:22
Here is what I returned.
Screenshot from 2016-01-29 15-36-02.png
Well part of what it returned.
--Mitch
Re: Logstash timestamp error
Posted: Mon Feb 01, 2016 1:11 pm
by tmcdonald
Can you expand out that sudo entry? And just to confirm, the new timestamp filter you used matches what date/time was in logstash.log for an incorrect format, correct?
Re: Logstash timestamp error
Posted: Mon Feb 01, 2016 2:24 pm
by GhostRider2110
Correct. I just tailed the logstash.log and got one of the later entries to search for.
Re: Logstash timestamp error
Posted: Mon Feb 01, 2016 4:02 pm
by jolson
Excellent. Could you expand out that sudo entry please? I'd like to see a screenshot of all of the information, including the timestamp field - after we have that information I can let you know how to take care of this issue making use of a syslog filter. Thanks!
Jesse
Re: Logstash timestamp error
Posted: Mon Feb 01, 2016 5:08 pm
by GhostRider2110
This what you are looking for?
Screenshot from 2016-02-01 17-07-07.png
Re: Logstash timestamp error
Posted: Mon Feb 01, 2016 5:49 pm
by jolson
Yes! Note that your timestamp has two spaces in between the month and the day - that could be breaking your syslog parsing. We'll be following this article:
http://kartar.net/2014/09/when-logstash ... -go-wrong/
First, access 'Administration -> Global Configuration'. Now, we'll erase the default syslog parser and replace it with two individual inputs:
Code: Select all
tcp {
port => 5544
type => syslog
}
Code: Select all
udp {
port => 5544
type => syslog
}
Now we'll make a filter:
Code: Select all
if [type] == "syslog" {
grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
}
In the end, your page should look something like this:
2016-02-01 16_46_49-Instance Configuration • Nagios Log Server - Chromium.png
This is where we'll start - if you still receive errors we can adjust the filter accordingly. Thanks!