Apache LogStash Filter Not Working?

[StormTheGates]

I have the following filter in 500_filters.conf

Code: Select all

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }

    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
}

And the following Apache access log request:

Code: Select all

216.81.94.73 - - [31/Aug/2016:15:21:22 -0400] "GET /etrr/ajax_filter.php?searchfor=DHS%20Antideficiency%20Act&tab=2 HTTP/1.1" 200 420

I get the following data point in my Log Server:

Code: Select all

{
  "_index": "logstash-2016.08.31",
  "_type": "apache_access",
  "_id": "AVbiDk5MPPpf_5wirD3R",
  "_score": null,
  "_source": {
    "message": "216.81.94.73 - - [31/Aug/2016:15:17:25 -0400] \"GET /etrr/ajax_filter.php?searchfor=Behavior%20Based%20Safety%20for%20Supervisors&tab=2 HTTP/1.1\" 200 258\n",
    "@version": "1",
    "@timestamp": "2016-08-31T19:17:34.000Z",
    "type": "apache_access",
    "host": "10.10.10.181",
    "priority": 133,
    "timestamp": "Aug 31 15:17:34",
    "logsource": "uscisreports",
    "program": "apache_access",
    "severity": 5,
    "facility": 16,
    "facility_label": "local0",
    "severity_label": "Notice",
    "tags": [
      "_grokparsefailure"
    ]
  },
  "sort": [
    1472671054000
  ]
}

And it seems like it can not break apart the response code or other meaningful information to filter on. I note the tag "_grokparsefailure" but I am not sure what to do with it.

How can I get useful information like the response code to be properly broken out? Is this an Apache configuration or a Nagios configuration? The Apache is configured for:

Code: Select all

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
CustomLog "logs/access_log" combined

[mcapra]

Is this an Apache configuration or a Nagios configuration?

Technically, it's both. You could adjust your apache logs to match the COMBINEDAPACHELOG grok pattern, or you could change the grok pattern itself that NLS is using. Easier and safer to change the grok pattern, in my opinion.

COMBINEDAPACHELOG is going to look for a referer and an agent as a part of the event. If it can't find those, you're likely to see some grokparsefailures. You could swap out COMBINEDAPACHELOG for COMMONAPACHELOG for starters. Make sure you're making the changes via the NLS GUI though rather than the flat file.

Code: Select all

if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMMONAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }

Bearing in mind the COMMONAPACHELOG pattern won't include agent or referrer information.

[StormTheGates]

Hmmm not sure why it dosnt have that info when %{Referer}i\" \"%{User-Agent}i is in the LogFormat.

However, changing to Common did the trick. Thank you! This item can be marked closed.