Logstash timestamp error

[GhostRider2110]

I'm trying to figure out how to eleminate this error from my logstash.log file:

{:timestamp=>"2016-01-28T03:51:09.340000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Jan 28 03:51:09", :exception=>java.lang.IllegalArgumentException: Invalid format: "Jan 28 03:51:09",
:level=>:warn}

How can I tell what log is sending it and how to correct the parsing?

Nagios Log Server • 1.4.0
2 Cluster system
71 systems feeding logs in.

Thanks in advanced...

--Mitch

[jolson]

Hi Mitch!

My bet is that the malformed date is hitting one of your 'syslog' inputs. The syslog input forces a particular date format. We can avoid this by either:

1. Sending the logs to a different input (tcp or udp) that has its own proper timestamp parser

-or-

2. Changing your syslog parsers to tcp/udp parsers so that the timestamp format restriction is lifted.

I recommend number 1, but that requires figuring out which host is sending the malformed logs to begin with. To figure this out, we can query for that particular log:

Code: Select all

timestamp:03 AND timestamp:51 AND timestamp:09

Let me know how you'd like to proceed. Thanks!

[GhostRider2110]

Thanks,

I would like to figure out where it is coming from and then adjust things to parse it properly. It is probably coming from a few servers running a custom python/wsgi app.

BTW, it was great meeting you guys at the World Conf. Really enjoyed it and plan on attending 2016...

See-ya
Mitch

[jolson]

BTW, it was great meeting you guys at the World Conf. Really enjoyed it and plan on attending 2016...

Glad you enjoyed it! It was a fantastic time.

I would like to figure out where it is coming from and then adjust things to parse it properly

Did the query I suggested above help you out? Let me know if you need some more direction - thanks!

[GhostRider2110]

Grabbed a newer entry from the logs

Code: Select all

timestamp:15 AND timestamp:33 AND timestamp:22

Here is what I returned.

Screenshot from 2016-01-29 15-36-02.png

Well part of what it returned.

--Mitch

[tmcdonald]

Can you expand out that sudo entry? And just to confirm, the new timestamp filter you used matches what date/time was in logstash.log for an incorrect format, correct?

[GhostRider2110]

Correct. I just tailed the logstash.log and got one of the later entries to search for.

[jolson]

Excellent. Could you expand out that sudo entry please? I'd like to see a screenshot of all of the information, including the timestamp field - after we have that information I can let you know how to take care of this issue making use of a syslog filter. Thanks!

Jesse

[GhostRider2110]

This what you are looking for?

Screenshot from 2016-02-01 17-07-07.png

[jolson]

Yes! Note that your timestamp has two spaces in between the month and the day - that could be breaking your syslog parsing. We'll be following this article: http://kartar.net/2014/09/when-logstash ... -go-wrong/

First, access 'Administration -> Global Configuration'. Now, we'll erase the default syslog parser and replace it with two individual inputs:

Code: Select all

  tcp {
    port => 5544
    type => syslog
  }

Code: Select all

  udp {
    port => 5544
    type => syslog
  }

Now we'll make a filter:

Code: Select all

  if [type] == "syslog" {
    grok {
      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }

In the end, your page should look something like this:

2016-02-01 16_46_49-Instance Configuration • Nagios Log Server - Chromium.png

This is where we'll start - if you still receive errors we can adjust the filter accordingly. Thanks!