Logstash: filter not filtering(?)

[mbeebe]

Hello,

We're seeing an issue with Logstash filtering in NLS.

To start with, here is the filter we're using:

if [program] == 'program_multiline' {
mutate {
replace => { "message" => "Test Message" }
replace => { "type" => "program_multiline" }
}
}

The issue we're seeing is that "type" gets set to "program_multiline", but "message" never gets set to "Test Message".

Here is the input we're working from:

tcp {
type => program_multiline
codec => multiline {
pattern => '^<133>%{GREEDYDATA}["program_multiline"][:][ ]\[[0-9]{4}[/][0-9]{2}[/][0-9]{2}[-][0-9]{2}[:][0-9]{2}[:][0-9]{2}.[0-9]{3}\]%{GREEDYDATA:message_body}'
negate => true
what => previous
}
port => 6688
}

We are correctly receiving input on port 6688 and the pattern matching is correct as well, so we don't believe the issue is on the input.

Any idea on what we're doing wrong?

Thanks,

-- Mike Beebe

[scottwilkerson]

Do you have you end up with a field in these records in Log Server called message? Because your pattern in the input suggests that it is being put into a field called message_body

[mbeebe]

Hi Scott,

Yes, we do. In fact, the whole impetus of this issue is trying to get "message" to correctly display a field of information without displaying the entire string.

Explanation:

We're ingesting logs from an application that comes in as a multiline block of text:

<133>Nov 6 10:40:53 servername program_multiline: ****Operations Support Alert****
<133>Nov 6 10:40:53 servername program_multiline: Information we want "message" to display
<133>Nov 6 10:40:53 servername program_multiline: Information we want "message" to display
<133>Nov 6 10:40:53 servername program_multiline: Information we want "message" to display
<133>Nov 6 10:40:53 servername program_multiline: ****Operations Support Alert ends****

We have a grok pattern that matches everthing left of the ":", then "greedydatas" everything to the right and stores it in a variable. We then try to use a "mutate/replace" to replace the content of "message" with the content of that variable -- but it never works. So in my troubleshooting, I decided to try to just use a literal value for "message" (the example I posted above). Even that doesn't work, however, hence my question.

The end goal would be to have "message" contain all the "information we want "message" to display" lines in so that when we get alerted, the "message" field only has those lines in it. Sorry, I know that's a bit of a convoluted explanation, but I hope what I'm trying to do gets across.

-- Mike Beebe

[scottwilkerson]

Just so I am clear, is the end goal to have the value of message be the contents of message_body ?

If so I think you can do this

Code: Select all

if [program] == 'program_multiline' {
mutate {
replace => { "message" => "%{message_body}" }
replace => { "type" => "program_multiline" }
}
}

[mbeebe]

That's what we though, too, but the result we get is the entire string, not the "message body" part.

I'll send you the actual input, filter and output in a private message and maybe you'll see something we're missing.

Thanks,

-- Mike Beebe

[scottwilkerson]

responded via PM

[scottwilkerson]

Based on what you have sent I believe you are going to need to add another grok filter to somehow break this into the pieces you want.

You have a pattern match in the input, but I believe you are going to need to have a grok filter that breaks that into just the pieces you want as everything in the message is still all together

[mbeebe]

Based on what you have sent I believe you are going to need to add another grok filter to somehow break this into the pieces you want.

You have a pattern match in the input, but I believe you are going to need to have a grok filter that breaks that into just the pieces you want as everything in the message is still all together

Hi Scott,

Sounds like I need to take this to the LogStash forum, as opposed to here.

Thanks for your help,

-- Mike Beebe

[mbeebe]

Sorry, before this thread is locked, I have another question:

Is it possible to run LogStash from the command line on a NLS server?

-- Mike Beebe

[scottwilkerson]

You can but be aware that the config you pass cannot contain the same ports that you are using while running as a service

basically

Code: Select all

/usr/local/nagioslogserver/logstash/bin/logstash -f /path/to/new/configs

Code: Select all

/usr/local/nagioslogserver/logstash/bin/logstash --help

Options:
    -f, --config CONFIG_PATH      Load the logstash config from a specific file
                                  or directory.  If a directory is given, all
                                  files in that directory will be concatenated
                                  in lexicographical order and then parsed as a
                                  single config file. You can also specify
                                  wildcards (globs) and any matched files will
                                  be loaded in the order described above.
    -e CONFIG_STRING              Use the given string as the configuration
                                  data. Same syntax as the config file. If no
                                  input is specified, then the following is
                                  used as the default input:
                                  "input { stdin { type => stdin } }"
                                  and if no output is specified, then the
                                  following is used as the default output:
                                  "output { stdout { codec => rubydebug } }"
                                  If you wish to use both defaults, please use
                                  the empty string for the '-e' flag.
                                   (default: "")
    -w, --pipeline-workers COUNT  Sets the number of pipeline workers to run.
                                   (default: 4)
    -b, --pipeline-batch-size SIZE Size of batches the pipeline is to work in.
                                   (default: 125)
    -u, --pipeline-batch-delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling
                                  for the next event.
                                   (default: 5)
    --filterworkers COUNT         DEPRECATED. Now an alias for --pipeline-workers and -w
    -l, --log FILE                Write logstash internal logs to the given
                                  file. Without this flag, logstash will emit
                                  logs to standard output.
    -v                            Increase verbosity of logstash internal logs.
                                  Specifying once will show 'informational'
                                  logs. Specifying twice will show 'debug'
                                  logs. This flag is deprecated. You should use
                                  --verbose or --debug instead.
    --quiet                       Quieter logstash logging. This causes only
                                  errors to be emitted.
    --verbose                     More verbose logging. This causes 'info'
                                  level logs to be emitted.
    --debug                       Most verbose logging. This causes 'debug'
                                  level logs to be emitted.
    --debug-config                Print the compiled config ruby code out as a debug log (you must also have --debug enabled).
                                  WARNING: This will include any 'password' options passed to plugin configs as plaintext, and may result
                                  in plaintext passwords appearing in your logs!
                                   (default: false)
    -V, --version                 Emit the version of logstash and its friends,
                                  then exit.
    -p, --pluginpath PATH         A path of where to find plugins. This flag
                                  can be given multiple times to include
                                  multiple paths. Plugins are expected to be
                                  in a specific directory hierarchy:
                                  'PATH/logstash/TYPE/NAME.rb' where TYPE is
                                  'inputs' 'filters', 'outputs' or 'codecs'
                                  and NAME is the name of the plugin.
    -t, --configtest              Check configuration for valid syntax and then exit.
    --[no-]allow-unsafe-shutdown  Force logstash to exit during shutdown even
                                  if there are still inflight events in memory.
                                  By default, logstash will refuse to quit until all
                                  received events have been pushed to the outputs.
                                   (default: false)
    -r, --[no-]auto-reload        Monitor configuration changes and reload
                                  whenever it is changed.
                                  NOTE: use SIGHUP to manually reload the config
                                   (default: false)
    --reload-interval RELOAD_INTERVAL How frequently to poll the configuration location
                                  for changes, in seconds.
                                   (default: 3)
    --allow-env                   EXPERIMENTAL. Enables templating of environment variable
                                  values. Instances of "${VAR}" in strings will be replaced
                                  with the respective environment variable value named "VAR".
                                   (default: false)
    --[no-]log-in-json            Specify that Logstash should write its own logs in JSON form - one
                                  event per line. If false, Logstash will log using Ruby's
                                  Object#inspect (not easy to machine-parse)
                                   (default: false)
    -h, --help                    print help