Local Filters

[CFT6Server]

I have local inputs on one of the 3 nodes that we have which pulls in apache files. That is working great, however I cannot apply filters to then. I've tried putting filters in the local or global configuration and it doesn't seem to process for the local input. So the input will set type and tags and I've tried putting the filter based on either of those. Doesn't seem to work. Checked the local files and configuration and it seems to be there, just doesn't seem to pick up the filter....

Sanitized configs:
Input

Code: Select all

file {
  path => [
  "/path1/logs/*",
  "/path2/logs/*"
  ]
  type => "apache"
  tags => "Apache"
}

Filter

Code: Select all

if [type] == "apache" {

    grok {
        match => [ 'message', '%{IP:Additional_IP}, %{COMBINEDAPACHELOG}',
                   'message', '%{COMBINEDAPACHELOG}' ]
    }
    date {
        match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
    }
}

[jolson]

Are there any failures in /var/log/logstash/logstash.log that seem relevant? I'm wondering if your timestamp isn't being parsed appropriately or similar.

I tested this in the lab and have no problems getting instance-specific inputs/filters working.

[CFT6Server]

Looks like I am getting this, but not sure if it is related....

Code: Select all

{:timestamp=>"2015-12-18T08:11:49.867000-0800", :message=>"Got error to send bulk of actions: None of the configured nodes are available: []", :level=>:error}
{:timestamp=>"2015-12-18T08:11:49.868000-0800", :message=>"Failed to flush outgoing items", :outgoing_count=>555, :exception=>org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [

The logs are showing up and tagged. Just won't process the filter.....

[jolson]

Could I see your entire logstash config? I'd like to reproduce this on my end.

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/*

[CFT6Server]

I've PM'd you the config file.

[jolson]

Thank you for that - after reviewing your configuration file I couldn't find any obvious problems.

I performed a test on my test server using the following per-instance config:

Code: Select all

file {
      path => [
      "/tmp/apache/*"
      ]
      type => "apache"
      tags => "Apache-DMZ"
}

    if [type] == "apache" {
    
        grok {
            match => [ 'message', '%{IP:src_IP}, %{COMBINEDAPACHELOG}',
                       'message', '%{COMBINEDAPACHELOG}' ]
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
    }

This worked perfectly. I then added another path:

Code: Select all

file {
      path => [
      "/tmp/apache/*",
      "/tmp/apache2/*"
      ]
      type => "apache"
      tags => "Apache-DMZ"
}

    if [type] == "apache" {
    
        grok {
            match => [ 'message', '%{IP:src_IP}, %{COMBINEDAPACHELOG}',
                       'message', '%{COMBINEDAPACHELOG}' ]
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
    }

This also worked well, this implies to me that the configuration itself is working, but there's something with your system in particular that's preventing the filter from being parsed.

Are you certain that the contents of the log files weren't dumped _before_ the filter was in place?

I would like to see a small portion of one of the log files you are processing.

{:timestamp=>"2015-12-18T08:11:49.867000-0800", :message=>"Got error to send bulk of actions: None of the configured nodes are available: []", :level=>:error}

I do not think that this is related.

[CFT6Server]

I've sent screenshots on PM. I am continuing to monitor this to see if it is a certain type of logs or what.

[jolson]

I cannot find a problem with your configuration. Could you send an email to [email protected] and reference this thread please? I'll pick up the ticket and we can perform a remote session to get this resolved.