_grokparsefailure on all events

[vAJ]

I must have missed Luke asking about _grokparsefailure back in November. But I'm now noticing that all events have have this tag.

What's the best way for tracking this down? I believe the three filters I have active right now are valid, not sure why I'm still getting this. Is there logging I need to increase level on to catch these better?

Current filters:

Code: Select all

if [program] == 'apache_access' {
    grok {
        match => [ 'message', '%{COMBINEDAPACHELOG}']
    }
    date {
        match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
    }
    mutate {
        replace => [ 'type', 'apache_access' ]
         convert => [ 'bytes', 'integer' ]
         convert => [ 'response', 'integer' ]
    }
}
 
if [program] == 'apache_error' {
    grok {
        match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
    }
    mutate {
        replace => [ 'type', 'apache_error' ]
    }
}

Code: Select all

if [type] == 'IIS_Requests' {
  grok {
    match => ['message', '%{DATESTAMP:timestamp} %{IPORHOST:hostip} %{WORD:method} %{URIPATH:request} (?:%{NOTSPACE:param}|-) %{NUMBER:port} (?:%{USER:username}|-) %{IPORHOST:clientip} (?:%{NOTSPACE:agent}|-) - %{NUMBER:response} %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:time-taken}']
  }
  date {
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss"]
  }
  geoip {
    source => "c-ip"
  }    
}

Code: Select all

if [progam] == 'mongod.perf.tp1' {
	
	mutate {
        replace => [ 'type', 'mongod_log' ]

-AJ

[tmcdonald]

You can take a look at /var/log/logstash/logstash.log for starters. Wanna post the last 20 lines or so?

[vAJ]

Code: Select all

{:timestamp=>"2015-02-06T19:58:49.500000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Feb  6 21:05:47", :exception=>java.lang.IllegalArgumentException: Invalid format: "Feb  6 21:05:47", :level=>:warn}
{:timestamp=>"2015-02-06T19:58:49.501000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Feb  6 21:05:47", :exception=>java.lang.IllegalArgumentException: Invalid format: "Feb  6 21:05:47", :level=>:warn}
{:timestamp=>"2015-02-06T19:58:49.503000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Feb  6 21:05:47", :exception=>java.lang.IllegalArgumentException: Invalid format: "Feb  6 21:05:47", :level=>:warn}
{:timestamp=>"2015-02-06T19:58:49.506000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Feb  6 21:05:47", :exception=>java.lang.IllegalArgumentException: Invalid format: "Feb  6 21:05:47", :level=>:warn}
{:timestamp=>"2015-02-06T19:58:49.507000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Feb  6 21:05:47", :exception=>java.lang.IllegalArgumentException: Invalid format: "Feb  6 21:05:47", :level=>:warn}
{:timestamp=>"2015-02-06T19:58:49.508000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Feb  6 21:05:47", :exception=>java.lang.IllegalArgumentException: Invalid format: "Feb  6 21:05:47", :level=>:warn}
{:timestamp=>"2015-02-06T19:58:49.508000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Feb  6 21:05:47", :exception=>java.lang.IllegalArgumentException: Invalid format: "Feb  6 21:05:47", :level=>:warn}
{:timestamp=>"2015-02-06T19:58:49.509000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Feb  6 21:05:47", :exception=>java.lang.IllegalArgumentException: Invalid format: "Feb  6 21:05:47", :level=>:warn}
{:timestamp=>"2015-02-06T19:58:49.510000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Feb  6 21:05:47", :exception=>java.lang.IllegalArgumentException: Invalid format: "Feb  6 21:05:47", :level=>:warn}
{:timestamp=>"2015-02-06T19:58:49.511000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Feb  6 21:05:47", :exception=>java.lang.IllegalArgumentException: Invalid format: "Feb  6 21:05:47", :level=>:warn}

It's nothing but these.

[vAJ]

I do notice that in /usr/local/nagioslogserver/logstash/etc/conf.d the input, filter and output files there all have a timestamp from yesterday, even though I've remove inactive filters from the UI.

[tmcdonald]

Did you make sure to apply the configuration? Just covering all the bases.

[vAJ]

Yep. No change over the weekend either. Current events still have the failure tag.

[tmcdonald]

Can you give us an ls -l of /usr/local/nagioslogserver/logstash/etc/conf.d please?

[vAJ]

Code: Select all

-rwxrwxr-x 1 nagios nagios  786 Feb  5 00:01 000_inputs.conf
-rwxrwxr-x 1 nagios nagios 1590 Feb  5 00:01 500_filters.conf
-rwxrwxr-x 1 nagios nagios  493 Feb  5 00:01 999_outputs.conf

[vAJ]

Even when I save and applied changes, these timestamps did not update.

[vAJ]

Is it worth adding

Code: Select all

tag_on_failure => []

to all filters?