Filter not working

[r34220]

Using the default syslog input I have entry like..

Code: Select all

{
  "_index": "logstash-2015.03.19",
  "_type": "syslog",
  "_id": "Y9vs2TAbTbmua0cnpXdpiQ",
  "_score": null,
  "_source": {
    "message": "10.64.2.48 - - [18/Mar/2015:17:18:55 -0700] \"GET / HTTP/1.0\" 200 1147",
    "@version": "1",
    "@timestamp": "2015-03-19T00:19:01.000Z",
    "type": "syslog",
    "host": "10.64.2.78",
    "priority": 133,
    "timestamp": "Mar 18 17:19:01",
    "logsource": "az84ap93v",
    "program": "Planning_dev_apache_access_log",
    "severity": 5,
    "facility": 16,
    "facility_label": "local0",
    "severity_label": "Notice"
  },
  "sort": [
    1426724341000,
    1426724341000
  ]
}

I have a filter setup like ...

Code: Select all

if [program] == 'Planning_dev_apache_access_log' {
    grok {
        match => [ 'message', '%{COMMONAPACHELOG}']
    }
    date {
        match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
    }
    mutate {
        replace => [ 'type', 'apache_access' ]
         convert => [ 'bytes', 'integer' ]
         convert => [ 'response', 'integer' ]
    }
}

But it doesn't seem to trigger. What am I doing wrong?

[jolson]

The syntax of your filter looks fine. Are you sure that you have information matching 'type' = 'Planning_dev_apache_access_log'?

I would like you to run the following command on each of your nodes:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf

Please let us know if any of your nodes do not include the filter you've defined.

[r34220]

The syntax of your filter looks fine. Are you sure that you have information matching 'type' = 'Planning_dev_apache_access_log'?

I would like you to run the following command on each of your nodes:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf

Please let us know if any of your nodes do not include the filter you've defined.

Here is the output

Code: Select all

[root@az84implico13c log]# cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Mon, 09 Mar 2015 11:22:18 -0700
#

#
# Global filters
#

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }

    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
}

#
# Local filters
#

In the GUI on Global Configuration page my added filter shows under Apache (Default) but not in this file. BTW, I am running in a single instance for my trial. Not sure if that would be the issue?

[jolson]

It looks like your custom filter is not making it to the server in question. Please run the following command:

Code: Select all

ll /usr/local/nagioslogserver/logstash/etc/conf.d/

This checks on the permissions of your logstash files. Permissions and ownership should be 644 and nagios:users respectively.
If the permissions are not as I specified, please change them:

Code: Select all

chmod 644 /usr/local/nagioslogserver/logstash/etc/conf.d/*

Code: Select all

chown nagios:users /usr/local/nagioslogserver/logstash/etc/conf.d/*

If the permissions look good, please run the 'verify' command from your webGUI (see attached image). Ensure that verify completes properly. Be sure that you 'Apply Configuration' for your filter to take effect.

[r34220]

The files were owned by apache

Code: Select all

-rw-rw-r-- 1 apache apache  498 Mar  9 11:22 999_outputs.conf
-rw-rw-r-- 1 apache apache  957 Mar  9 11:22 500_filters.conf
-rw-rw-r-- 1 apache apache  636 Mar  9 11:22 000_inputs.conf

I have changed ownership but the files do not get updated still.

[jolson]

After you change the ownership to nagios:users, please perform the following-
Restart services that touch the files in question:

Code: Select all

service logstash restart

Code: Select all

service httpd restart

Verify that permissions stuck:

Code: Select all

ll /usr/local/nagioslogserver/logstash/etc/conf.d/

Verify directory permissions (should be 775 nagios:nagios):

Code: Select all

ll -d /usr/local/nagioslogserver/logstash/etc/conf.d/

Go to the WebGUI and press 'Verify' - does verify work properly?
Go to the WebGUI and press 'Apply Configuration. Does your custom filter propagate?

Let us know. Thanks!

[r34220]

Verify completes with "Configuration is OK!"

When I click "Apply". I am asked if I am Sure then I select "Yes, Apply Now" and immediately I am return to the Apply Configuration page. But the 500_filters.conf file does not change.

[jolson]

Try deleting all files in that directory.

Code: Select all

rm -f /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf

Code: Select all

rm -f /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf

Code: Select all

rm -f /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf

After this is done, press 'apply configuration' again. Are the files generated properly?

[r34220]

None of the files were recreated.

[jolson]

On my test machine, they regenerate. One thing that comes to mind is selinux:

Code: Select all

getenforce

If it is enabled, try:

Code: Select all

setenforce 0

And re-apply your configuration.

You will also want to make sure that your groups file looks similar to mine:

Code: Select all

cat /etc/group

nagios500:nagios,apache

Failing this, we should check permissions all the way down.

Code: Select all

namei -l /usr/local/nagioslogserver/logstash/etc/conf.d/

Let me know the results. Thanks!