KV plugin for Logstash

[krobertson71]

I am trying to test out the KV filter but I keep failing verification.

Code: Select all

if [tags] == 'auditd' {
  filter {
    kv { }
  }
}

It is complaining about missing => or #.

Is the KV plugin not already included in logstash. I thought I read and it was. If not, how do I add it to NLS?

[mcapra]

The filter doesn't appear to be included with the logstash that ships with NLS.

You should be able to install it manually though:

Code: Select all

/usr/local/nagioslogserver/logstash/bin/plugin install logstash-filter-kv

[krobertson71]

That was easy, but still have the error:

Code: Select all

Error: Expected one of #, => at line 124, column 12 (byte 2586) after filter {

Here is the filter I created. I am just staring off using the basic functionality of the KV plug-in.

Code: Select all

if [tags] == 'auditd' {
  filter {
    kv { }
  }
}

That is the way logstash explains who to use KV.. First "Filter {" then KV inside of that.

[mcapra]

From the NLS GUI, using a filter definition is likely to trip things up a bit due to how it gets translated into the logstash configuration files. Try removing filter:

Code: Select all

if [tags] == 'auditd' {
    kv { }
}

The logstash documentation is correct though, it's just a quirk of how things are handled via the NLS GUI. Check out /usr/local/nagioslogserver/logstash/etc/conf.d/ to see what i'm talking about in terms of how the GUI input gets translated into configuration files.

[krobertson71]

That passed the verification process but the filter does not seem to be taking affect at all:

Here is the event:

Code: Select all

<174>Sep  8 13:58:07 Serverx auditd: type=USER_START msg=audit(1473357481.478:232322): user pid=16059 uid=0 auid=0 msg='PAM: session open acct="root" : exe="/usr/sbin/crond" (hostname=?, addr=?, terminal=cron res=success)'

Here is the filter:

Code: Select all

if [tags] == "auditd" {
    kv { }
}

The input filter I am using is 'import_raw' and adding a tag 'auditd'. The filter is based on incoming event with that tag. At least KV should be creating fields where key=value but that is not happening... HELP.

[mcapra]

You might give the following logic a try:

Code: Select all

if "auditd" in [tags] {
    kv { }
}

Assuming only auditd entries have the auditd tag.

[mcapra]

That logic worked for me, but one thing worth mentioning is that the type field is being overwritten by the kv filter:

2016_09_08_13_14_25_Dashboard_Nagios_Log_Server.png

[krobertson71]

Thanks for the info but I ran into another issue. I did not apply the changes you suggested yet. When I logged in this morning all the inputs and filters, in the GUI, were gone. All blank. However in ../etc/conf.d the input and filter conf files are still present with all the proper configurations. It's seems like NLS is not loading them. I performed a restart on both logstash and Elasticsearch but that did not correct the issue.

Permissions on the files are rwxrwxr-w and owned by nagios nagios. This has always been the case. Not sure what would be causing this.. What is funny, even though the GUI says the inputs are gone, data is still coming in on the port that were defined, just now everything is coming in due to everything not loading.

HELP

[rkennedy]

If there were no inputs, generally logstash will fail after a while. As a frame of reference, could you show us what your global configuration page looks like currently?