Logstash Filter using Ruby Code

[SteveBeauchemin]

I have everything working perfectly from the command line. Everything = configuration related to using Internal IP to geoip mapping.

Added the filters to NLS and committed the change to all 4 systems.

Now I'm just waiting for the servers to notice that they have a new configuration. Maybe they will update by tomorrow when I come back in to work. I'll let you know. Like I said - from command line it's awesome...

Will touch base tomorrow and let you know if the configs are running. They are listed in the GUI when I open and look.

Thanks

Steve B

[scottwilkerson]

Nice, you sometimes you have to wait for the index to roll over to the next day for the fields to map properly as a geo point

[SteveBeauchemin]

I guess I broke something. The data stopped coming in completely. Today I had to remove some filters to get the IIS logs back in the GUI.

I am now looking at the filter syntax to see what is not right.

I have a basic question about the logstash language versus ruby language.

Is it better to have the ruby do the if, elsif, else. Or is logstash the place to so that.
Specifically this piece of code. I am not getting the clientip fields consistently. And for the c-ip branch, not at all.

Code: Select all

# All ip need to be put in this variable 'clientipdec' to be 
# considered in the geoip generator
if [x-forwarded-for] {
  ruby {
    code => "require 'ipaddr'
      decimalip = event['x-forwarded-for']
      event['clientipdec'] = IPAddr.new(decimalip,Socket::AF_INET).to_i
      event['clientipnew'] = decimalip
    "
  }
} else if [c-ip] {
  ruby {
    code => "require 'ipaddr'
      decimalip = event['c-ip']
      event['clientipdec'] = IPAddr.new(decimalip,Socket::AF_INET).to_i
      event['clientipnew'] = decimalip
    "
  }
}

The code should prefer the x-forwarded-for first and only use c-ip if the other is not found.

I'll try putting the decision logic in ruby, but what runs good from the command line is not okay in the GUI. Meaning that the first clientip field shows up in the GUI, but the second one does not.

Thanks

Steve B

[cdienger]

I would rely on Logstash to handle the logic first and only use the Ruby filter if necessary. I would think handing the logic before having to pass it to another plugin to handle it would be a little better for performance and probably a bit easier to debug as well.

Do you have any consistent test data to reproduce the issue with the clientip field? Try enabling Logstash debug logging and importing that data again if so and we can take a look at it.

[SteveBeauchemin]

I have been watching the logstash log file and what I see looks like good stuff being processed.

I think that I need to wait for the new index to get created, like Scott suggested.

Maybe tomorrow I will see what I expect. But, the logstash logs do look good. I have my fingers crossed.

We will see tomorrow.

Thanks

Steve B

[SteveBeauchemin]

Okay, it is 'tomorrow'

!! There is data !!

I have geoip data for all my IP. Both private internal, and public external.

I can now see on the map my Internal IP which is most of our systems. The internal IP matter to us more than the external from an application support perspective. Internally, we have LAN, WiFi, and IP-Phones.

Before this effort, I had one dot on the map where our company Class B address was registered. All the 10.x 172.x 192.168.x were dropped. Now I see those all over the globe.

This is really awesome.

If anyone wants to know the setup, I am willing to share.

Thanks for all the help.

Steve B

[cdienger]

Very glad to hear! This sounds is a pretty neat configuration and I would appreciate any details about the setup you can share.