Page 1 of 1
KV plugin for Logstash
Posted: Thu Sep 08, 2016 10:38 am
by krobertson71
I am trying to test out the KV filter but I keep failing verification.
Code: Select all
if [tags] == 'auditd' {
filter {
kv { }
}
}
It is complaining about missing => or #.
Is the KV plugin not already included in logstash. I thought I read and it was. If not, how do I add it to NLS?
Re: KV plugin for Logstash
Posted: Thu Sep 08, 2016 10:44 am
by mcapra
The filter doesn't appear to be included with the logstash that ships with NLS.
You should be able to install it manually though:
Code: Select all
/usr/local/nagioslogserver/logstash/bin/plugin install logstash-filter-kv
Re: KV plugin for Logstash
Posted: Thu Sep 08, 2016 11:38 am
by krobertson71
That was easy, but still have the error:
Code: Select all
Error: Expected one of #, => at line 124, column 12 (byte 2586) after filter {
Here is the filter I created. I am just staring off using the basic functionality of the KV plug-in.
Code: Select all
if [tags] == 'auditd' {
filter {
kv { }
}
}
That is the way logstash explains who to use KV.. First "Filter {" then KV inside of that.
Re: KV plugin for Logstash
Posted: Thu Sep 08, 2016 11:47 am
by mcapra
From the NLS GUI, using a
filter definition is likely to trip things up a bit due to how it gets translated into the logstash configuration files. Try removing
filter:
The logstash documentation is correct though, it's just a quirk of how things are handled via the NLS GUI. Check out
/usr/local/nagioslogserver/logstash/etc/conf.d/ to see what i'm talking about in terms of how the GUI input gets translated into configuration files.
Re: KV plugin for Logstash
Posted: Thu Sep 08, 2016 1:03 pm
by krobertson71
That passed the verification process but the filter does not seem to be taking affect at all:
Here is the event:
Code: Select all
<174>Sep 8 13:58:07 Serverx auditd: type=USER_START msg=audit(1473357481.478:232322): user pid=16059 uid=0 auid=0 msg='PAM: session open acct="root" : exe="/usr/sbin/crond" (hostname=?, addr=?, terminal=cron res=success)'
Here is the filter:
The input filter I am using is 'import_raw' and adding a tag 'auditd'. The filter is based on incoming event with that tag. At least KV should be creating fields where key=value but that is not happening... HELP.
Re: KV plugin for Logstash
Posted: Thu Sep 08, 2016 1:08 pm
by mcapra
You might give the following logic a try:
Assuming only auditd entries have the auditd tag.
Re: KV plugin for Logstash
Posted: Thu Sep 08, 2016 1:15 pm
by mcapra
That logic worked for me, but one thing worth mentioning is that the
type field is being overwritten by the
kv filter:
2016_09_08_13_14_25_Dashboard_Nagios_Log_Server.png
Re: KV plugin for Logstash
Posted: Fri Sep 09, 2016 8:24 am
by krobertson71
Thanks for the info but I ran into another issue. I did not apply the changes you suggested yet. When I logged in this morning all the inputs and filters, in the GUI, were gone. All blank. However in ../etc/conf.d the input and filter conf files are still present with all the proper configurations. It's seems like NLS is not loading them. I performed a restart on both logstash and Elasticsearch but that did not correct the issue.
Permissions on the files are rwxrwxr-w and owned by nagios nagios. This has always been the case. Not sure what would be causing this.. What is funny, even though the GUI says the inputs are gone, data is still coming in on the port that were defined, just now everything is coming in due to everything not loading.
HELP
Re: KV plugin for Logstash
Posted: Fri Sep 09, 2016 9:08 am
by rkennedy
If there were no inputs, generally logstash will fail after a while. As a frame of reference, could you show us what your global configuration page looks like currently?