geoip conflict

[benhank]

ok ill give it a shot

[cdienger]

Keep us posted. One other thing too - I was able to get similar results when the geoip lines were added:

Code: Select all

        geoip {
        database => "/usr/share/GeoIP/GeoLiteCity.dat"
        source => "IPAddress"
        }

In my case it was failing because the dat file didn't exist. The database line isn't necessary so it may be worth testing without this part of the config and/or make sure that /usr/share/GeoIP/GeoLiteCity.dat exists and has write access set.

[benhank]

how do I change the field IP to Ipaddress.
that database exists ion the servers, we are using so maybe it will work if I change the field.


The real problem here is that I am clueless when it comes to creating filters. If you guys would be kind enough to answer a few questions for me, it will go a long way towards me being able tyo create and troubleshoot filters like this myself. STOP LAUGHING! i know ya'll are laughing....

so here goes:
first, looking closely at this filter I see that it is listed with with one difference:

Code: Select all

match => { "message" => "result=\"%{WORD:result}\" ip=\"%{IP:IP}\" action=\"%{WORD:action}\" params=\"Username: %{USER:params}\" user=\"%{USER:user}\" tenant=\"%{WORD:tenant}\""}
match => { "message" => "result=\"%{WORD:result}\" ip=\"%{IP:IP}\" action=\"%{WORD:action}\" params=\"Username=%{USER:params}\" user=\"%{USER:user}\" tenant=\"%{WORD:tenant}\""}

line 1

Code: Select all

"Username: %{USER:params}

Line 2

Code: Select all

"Username=%{USER:params}

Second, I know that Scott must be correct when he said :

You were putting your match in { instead of [

'
my question is whats the difference between the two symbols? and why would it bomb out when I use the suggested correction Scott gave me, but work when I used { instead of the ]?

And third using the following as an example:

Code: Select all

result=\"%{WORD:result}

what does each term in that line do?
Thanks guys!

[swolf]

how do I change the field IP to Ipaddress.

If you're trying to change one incoming field to another, I would just create another filter. So if the log is coming in with 'IP' as the field, we can do:

Code: Select all

mutate {
    replace => { "Ipaddress" => "%{IP}" }
  }

inside a filter.
Make sure this is happening before you use geoip.

so here goes:
first, looking closely at this filter I see that it is listed with with one difference:

Code: Select all

match => { "message" => "result=\"%{WORD:result}\" ip=\"%{IP:IP}\" action=\"%{WORD:action}\" params=\"Username: %{USER:params}\" user=\"%{USER:user}\" tenant=\"%{WORD:tenant}\""}
match => { "message" => "result=\"%{WORD:result}\" ip=\"%{IP:IP}\" action=\"%{WORD:action}\" params=\"Username=%{USER:params}\" user=\"%{USER:user}\" tenant=\"%{WORD:tenant}\""}

line 1

Code: Select all

"Username: %{USER:params}

Line 2

Code: Select all

"Username=%{USER:params}

In this case, the difference is just matching the literal text of the log line. If you have a log format where you use a colon, like

Code: Select all

result: "okay" IP: "127.0.0.1" action: "something"

Then the username section of the first line would be correct. Since your log is instead using "tag=value" syntax, you just want to make sure you're always writing the equals.

Second, I know that Scott must be correct when he said :

You were putting your match in { instead of [

'
my question is whats the difference between the two symbols? and why would it bomb out when I use the suggested correction Scott gave me, but work when I used { instead of the ]?

The difference here is that {} is referring to a logstash hash (which other languages call a 'map', 'dictionary', 'object', or 'associative array'), where [] is referring to an array (which other languages call a 'list' or 'vector'). Hashes have a list of keys pointing to values, and arrays are lists of just values.

I actually disagree with Scott here about changing {} to [] in the grok/match filters. If we look at the documentation then your original lines were correct. You can change the "message" in match to be an array, but not the match itself.

And third using the following as an example:

Code: Select all

result=\"%{WORD:result}

what does each term in that line do?

I would read that as

Code: Select all

match the literal characters result="
match according to the 'WORD' pattern, store it in the field 'result'

The backslash escapes the double quote since that's a reserved character, and the %{} tells it to match a regular expression. I'm having trouble finding all of the defaults (edit: Craig found them, check the end of this doc), but the left side of the colon is the pattern and the right side is the field it goes into.

I'll also point out that this is a weird use of %{}. Most of the time, %{} is used to reference a particular field. For instance, I can do

Code: Select all

exec {
    command => "bash /path/to/do/something.sh %{IP}"
}

as an output and this will execute /path/to/do/something.sh with the IP address as $1.

Let us know if that all made sense, or if there are other questions.

[cdienger]

You can save the IP to a field called Ipaddress with:

ip=\"%{IP:Ipaddress}

The different between line 1 and line 2 is the extra space and colon versus equal sign. This it to account for differences in logs where the log line may contain:

Username: jdoe

vs

Username=jdoe

The lines could be made better with some regex to account for both in a single pattern:

match => [ "message" => "result=\"%{WORD:result}\" ip=\"%{IP:IP}\" action=\"%{WORD:action}\" params=\"Username((: ){1})*((=){1})*%{USER:params}\" user=\"%{USER:user}\" tenant=\"%{WORD:tenant}\""]

Using [ vs } is a matter of what exactly you're try to do. This for example:

Code: Select all

grok {
match => { "message" => "patternpatternpattern..."}
}

tells Logstash to take the message field and try to match it against the pattern. And:

Code: Select all

grok {
match => ["patternpatternpattern1...","patternpatternpattern2..."]
}

tells Logstash to try and match the event(typically the contents of the messages field) against a list of patterns.

Code: Select all

result=\"%{WORD:result}

this searches for the string "result=" followed by a word that is then stored in a field called "result"

Grok and logstash configs can be ...fun... and can take a moment to get used to. For more examples I would check out:

https://support.nagios.com/kb/article/n ... ew-98.html
https://assets.nagios.com/downloads/nag ... ilters.pdf

[benhank]

THANKS GUYS!
Thanks to everyone who pitched in on this! Im still working on it and will keep you all updated!

[cdienger]

Sound good!

[benhank]

hqdefault.jpg

this allows me to apply the config and seems to parse the data correctly, I think I have to wait for someone to connect with an external ip in order to get the geoip data, but looking good so far:

Code: Select all

if [host] == '172.30.100.226' {

grok {

match => { "message" => "result=\"%{WORD:result}\" ip=\"%{IP:IP}\" action=\"%{WORD:action}\" params=\"Username: %{USER:params}\" user=\"%{USER:user}\" tenant=\"%{WORD:tenant}\""}
match => { "message" => "result=\"%{WORD:result}\" ip=\"%{IP:IP}\" action=\"%{WORD:action}\" params=\"Username=%{USER:params}\" user=\"%{USER:user}\" tenant=\"%{WORD:tenant}\""}
}
geoip {
database => "/usr/share/GeoIP/GeoLiteCity.dat"
source => "IP"
}
}

[cdienger]

Glad to hear!

[benhank]

i have another question.
is there a way to force my newly created filter to work on logfiles created before I implemented the newly created filter?
That is If I create a new filter today, I want it to work on logfiles that were sent to NLS from say a month ago.