Page 1 of 3
Filter not working
Posted: Wed Mar 18, 2015 7:31 pm
by r34220
Using the default syslog input I have entry like..
Code: Select all
{
"_index": "logstash-2015.03.19",
"_type": "syslog",
"_id": "Y9vs2TAbTbmua0cnpXdpiQ",
"_score": null,
"_source": {
"message": "10.64.2.48 - - [18/Mar/2015:17:18:55 -0700] \"GET / HTTP/1.0\" 200 1147",
"@version": "1",
"@timestamp": "2015-03-19T00:19:01.000Z",
"type": "syslog",
"host": "10.64.2.78",
"priority": 133,
"timestamp": "Mar 18 17:19:01",
"logsource": "az84ap93v",
"program": "Planning_dev_apache_access_log",
"severity": 5,
"facility": 16,
"facility_label": "local0",
"severity_label": "Notice"
},
"sort": [
1426724341000,
1426724341000
]
}
I have a filter setup like ...
Code: Select all
if [program] == 'Planning_dev_apache_access_log' {
grok {
match => [ 'message', '%{COMMONAPACHELOG}']
}
date {
match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
}
mutate {
replace => [ 'type', 'apache_access' ]
convert => [ 'bytes', 'integer' ]
convert => [ 'response', 'integer' ]
}
}
But it doesn't seem to trigger. What am I doing wrong?
Re: Filter not working
Posted: Thu Mar 19, 2015 9:31 am
by jolson
The syntax of your filter looks fine. Are you sure that you have information matching 'type' = 'Planning_dev_apache_access_log'?
I would like you to run the following command on
each of your nodes:
Code: Select all
cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf
Please let us know if any of your nodes do
not include the filter you've defined.
Re: Filter not working
Posted: Thu Mar 19, 2015 10:08 am
by r34220
jolson wrote:The syntax of your filter looks fine. Are you sure that you have information matching 'type' = 'Planning_dev_apache_access_log'?
I would like you to run the following command on
each of your nodes:
Code: Select all
cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf
Please let us know if any of your nodes do
not include the filter you've defined.
Here is the output
Code: Select all
[root@az84implico13c log]# cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Mon, 09 Mar 2015 11:22:18 -0700
#
#
# Global filters
#
filter {
if [program] == 'apache_access' {
grok {
match => [ 'message', '%{COMBINEDAPACHELOG}']
}
date {
match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
}
mutate {
replace => [ 'type', 'apache_access' ]
convert => [ 'bytes', 'integer' ]
convert => [ 'response', 'integer' ]
}
}
if [program] == 'apache_error' {
grok {
match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
}
mutate {
replace => [ 'type', 'apache_error' ]
}
}
}
#
# Local filters
#
In the GUI on Global Configuration page my added filter shows under Apache (Default) but not in this file. BTW, I am running in a single instance for my trial. Not sure if that would be the issue?
Re: Filter not working
Posted: Thu Mar 19, 2015 10:30 am
by jolson
It looks like your custom filter is not making it to the server in question. Please run the following command:
Code: Select all
ll /usr/local/nagioslogserver/logstash/etc/conf.d/
This checks on the permissions of your logstash files. Permissions and ownership should be 644 and nagios:users respectively.
If the permissions are
not as I specified, please change them:
Code: Select all
chmod 644 /usr/local/nagioslogserver/logstash/etc/conf.d/*
Code: Select all
chown nagios:users /usr/local/nagioslogserver/logstash/etc/conf.d/*
If the permissions look good, please run the 'verify' command from your webGUI (see attached image). Ensure that verify completes properly. Be sure that you 'Apply Configuration' for your filter to take effect.
Re: Filter not working
Posted: Thu Mar 19, 2015 11:25 am
by r34220
The files were owned by apache
Code: Select all
-rw-rw-r-- 1 apache apache 498 Mar 9 11:22 999_outputs.conf
-rw-rw-r-- 1 apache apache 957 Mar 9 11:22 500_filters.conf
-rw-rw-r-- 1 apache apache 636 Mar 9 11:22 000_inputs.conf
I have changed ownership but the files do not get updated still.
Re: Filter not working
Posted: Thu Mar 19, 2015 11:27 am
by jolson
After you change the ownership to nagios:users, please perform the following-
Restart services that touch the files in question:
Verify that permissions stuck:
Code: Select all
ll /usr/local/nagioslogserver/logstash/etc/conf.d/
Verify directory permissions (should be 775 nagios:nagios):
Code: Select all
ll -d /usr/local/nagioslogserver/logstash/etc/conf.d/
Go to the WebGUI and press 'Verify' - does verify work properly?
Go to the WebGUI and press 'Apply Configuration. Does your custom filter propagate?
Let us know. Thanks!
Re: Filter not working
Posted: Thu Mar 19, 2015 11:49 am
by r34220
Verify completes with "Configuration is OK!"
When I click "Apply". I am asked if I am Sure then I select "Yes, Apply Now" and immediately I am return to the Apply Configuration page. But the 500_filters.conf file does not change.
Re: Filter not working
Posted: Thu Mar 19, 2015 11:55 am
by jolson
Try deleting all files in that directory.
Code: Select all
rm -f /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf
Code: Select all
rm -f /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf
Code: Select all
rm -f /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf
After this is done, press 'apply configuration' again. Are the files generated properly?
Re: Filter not working
Posted: Thu Mar 19, 2015 12:06 pm
by r34220
None of the files were recreated.
Re: Filter not working
Posted: Thu Mar 19, 2015 12:13 pm
by jolson
On my test machine, they regenerate. One thing that comes to mind is selinux:
If it is enabled, try:
And re-apply your configuration.
You will also want to make sure that your groups file looks similar to mine:
nagios
500:nagios,apache
Failing this, we should check permissions all the way down.
Code: Select all
namei -l /usr/local/nagioslogserver/logstash/etc/conf.d/
Let me know the results. Thanks!
500:nagios,apache