Logstash Filter using Ruby Code

[SteveBeauchemin]

Now I am looking for Logstash > Filter > Ruby help

Just to test some basic functionality, I have this in a filter file.
It does not make a file in /tmp ... nor does it add the field named clientip

Code: Select all

  ruby {
    code => "
      File.open('/tmp/mydebug.log','a') { |f| f.puts event['x-forwarded-for'] }
      event['clientip'] = 'testing'
    "
  }

The field x-forwarded-for exists and has an IP address in it. I wanted to make a list in a file. Just as a temporary test.
The field clientip does not exist but I am attempting to add it. Just as a temporary test.

I wanted to take small steps and prove that I can make a field. I wanted to see ruby do anything. Or something.

I am basically working up to getting the code below in place. Ultimately, I need to get the following work.

Code: Select all

  ruby {
    code => "
      require 'ipaddr'
      event['clientip'] = IPAddr.new(event['c-ip']).strip.to_i
    "
  }

Please help with this. Is it syntax? Do I need to do something special to make ruby work?
Do I need to install ipaddr in ruby?

Thanks

Steve B

[cdienger]

Check the permissions on /tmp - I was able to get your code working without any issue so it may be running into problems with writing to that location.

For the second part, I don't think 'strip' is a valid function for ipaddr.new and was able to get it to work by dropping it.

Attached is a screenshot showing my simple test setup. I would telnet 10 the NLS machine on port 2059 and send a simple string like "1.2.3.4 test".

[SteveBeauchemin]

You know... Something weird is happening on my system when I make updates to the Configs.

When I run the test config from the command line, it works just fine.
I use
/usr/local/nagioslogserver/logstash/bin/logstash -f IPtest01.conf
and that file is basic. I paste the line item in and see the results. then hit ctrl-d to bail out
File is here - works great - hit ctrl-d to quit.

Code: Select all

cat IPtest01.conf
input {
  stdin { }
}

# The following data is the input - pasted in
# 02:36.01 /var/log/Service1/myapp.log 131.198.86.206 Ruby is great
filter {

  # test things here
  grok {
    match => { "message" => "%{DATA:justtime} %{DATA:logsource} %{IPORHOST:c-ip} %{GREEDYDATA:msg}" }
  } #
  # incorrectly autopopulates to first day of year
  date {
    match => [ "justtime", "HH:mm.ss" ]
    target => "incorrectfulldatetime"
    timezone => "America/Los_Angeles"
  } # date

  # use ruby to augment with current day
  ruby {
    code => "
      event['fulldatetime'] = Time.now.strftime('%Y-%m-%d') + ' ' + event['justtime']
    "
  }
  date {
    match => [ "fulldatetime", "YYYY-MM-dd HH:mm.ss" ]
    target => "correctfulldatetime"
    timezone => "America/Los_Angeles"
  } # date

  # split apart log source to extract service name
  ruby {
    code => "
      fpath = event['logsource'].split('/')
      event['serviceName'] = fpath[fpath.length-2].downcase
    "
  }
  ruby {
    code => "
      require 'ipaddr'
      decimalip = event['c-ip']
      event['clientip'] = IPAddr.new(decimalip,Socket::AF_INET).to_i
    "
  }

}

output {
  stdout {
    codec => rubydebug
  }
}

Once I add it to the GUI and Save / Apply the new filter, it is not working right.
This is the filter in the GUI

Code: Select all

if [x-forwarded-for] {
  ruby {
    code => "require 'ipaddr'
      decimalip = event['x-forwarded-for']
      event['clientip'] = IPAddr.new(decimalip,Socket::AF_INET).to_i"
  }
  geoip {
    database => "/usr/share/GeoIP/GeoLite2-City.mmdb"
    source => "x-forwarded-for"
  }
}
else if [c-ip] {
  ruby {
    code => "require 'ipaddr'
      decimalip = event['c-ip']
      event['clientip'] = IPAddr.new(decimalip,Socket::AF_INET).to_i"
  }
  geoip {
    database => "/usr/share/GeoIP/GeoLite2-City.mmdb"
    source => "c-ip"
  }
}

I just now did an update, and I am getting the
event['clientip'] = 'testing'

But that has not been in the config for an hour. I made changes since it didn't show up. I moved on... But NLS is stuck on the old config. The config that did not work and caused me to post in the first place.

Now I have the filed "clientip = testing" in the NLS event log screen as part of the data. But it is not in the logstash config file at the OS level.
I even manually copied the 3 logstash files to conf.d and restarted Logstash on all 4 systems.

Do the /usr/local/nagioslogserver/logstash/etc/conf.d files provide the configuration that is run? Or are they copied somewhere else and used from there.

This is confusing. I can manually run and get the results I want.

Please advise.

Thanks

Steve B

[cdienger]

The config is part of the database and then writes to the conf.d/ location so it sounds like there may be an issue replicating the database. Are all the machines in the same location or are they spread out? Try making a change to the config using the web UI and then login to each of the other 3 machines and see if the change shows up in their web UI.

The Elasticsearch logs may have something of interest - check these out under /var/log/elasticsearch/ and attach or PM me copies so we can review them if needed.

[SteveBeauchemin]

I just walked through all 4 servers, one at a time. The configurations look good. The way I expect them to.

I looked in the line items being created for IIS and there is a field named clientip there with the content of 'testing'

The field has no reason to be there. I manually added it before as a test of the 'add a field' stuff. But that configuration no longer exists.

The machines are not all located on the same LAN. They are spread out to 3 locations.

2 are in the Primary site. The other 2 are basically in our MPLS cloud.

We have 11 'Network Hub' locations in a home grown MPLS Cloud setup. It works great. All my mod_gearman servers are there, one at each of the 6 USA hub sites, and 2 in the primary location in the CST time zone. It is very fault tolerant. Transparent failovers.

I think the replication from server to server works good. If I stop the logstash and elasticsearch processes for 5 minutes or so, and turn it on again, it takes about 10 minutes and the cluster changes from yellow to green. Approximately.

I just don't grok how the OS files look good, the GUI filters look good, but I get a field that I should not see.

I'm almost where I want to be, but this is kicking my butt.

Steve B

(Heinlein grok, not the geek one)

[cdienger]

What is the output if you run 'curl -XGET 'http://localhost:9200/nagioslogserver/_ ... ode&pretty'' on each machine?

[SteveBeauchemin]

I sent raw text as PM to @cdienger as requested previously.

I think time heals all "NLS config errors" or wounds - today the field is gone.

I'm still going to need other help. For example...

I am trying to use a filter of jdbc_static

It installed clean when I did this:

Code: Select all

cd /usr/local/nagioslogserver/logstash
bin/logstash-plugin install logstash-filter-jdbc_static

When I run "bin/logstash-plugin list" it is shown in the list

This is a snip of the filter text, sanitized a little.

Code: Select all

  jdbc_static {
    loaders => [
      {
        id => "remote-geoips"
        query => "SELECT
                  startrange, endrange, building,
                  geoiplongitude, geoiplatitude, geoiplocation,
                  geoipcity, geoiptime_zone, geoipcontinent_code,
                  geoipcountry_code3, geoipcountry_code2, geoipcountry_name
                  FROM mydatabase ORDER BY startrange"
        local_table => "local-geoips"
      }
    ]
    local_db_objects => [
      {
        name => "local-geoips"
        index_columns => ["startrange"]
        columns => [
          ["startrange", "int(10) unsigned"],
          ["endrange", "int(10) unsigned"],
          ["building", "varchar(8)"],
          ["geoiplongitude", "decimal(11,8)"],
          ["geoiplatitude", "decimal(10,8)"],
          ["geoiplocation", "varchar(64)"],
          ["geoipcity", "varchar(64)"],
          ["geoiptime_zone", "varchar(64)"],
          ["geoipcontinent_code", "varchar(2)"],
          ["geoipcountry_code3", "varchar(3)"],
          ["geoipcountry_code2", "varchar(2)"],
          ["geoipcountry_name", "varchar(64)"]
        ]
      }
    ]
    local_lookups => [
      {
        query => "SELECT
                  geoiplongitude, geoiplatitude, geoiplocation,
                  geoipcity, geoiptime_zone, geoipcontinent_code,
                  geoipcountry_code3, geoipcountry_code2, geoipcountry_name
                  FROM local-geoips
                  WHERE :clientipnumber BETWEEN startrange AND endrange"
        parameters => {clientipnumber => "[c-ip]"}
        target => "sql_geoip"
      }
    ]
    staging_directory => "/tmp/logstash/jdbc_static/import_data"
    # run loaders every 2 hours
    loader_schedule => "* */2 * * *"
    jdbc_user => "user"
    jdbc_password => "password"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_driver_library => "/usr/share/java/mysql-connector-java.jar"
    jdbc_connection_string => "jdbc:mysql://some-address:3306/DB"
  }

When I try to use it I get at the top:
"fetched an invalid config" with lots of red and... at the bottom.

Code: Select all

:reason=>"Couldn't find any filter plugin named 'jdbc_static'. Are you sure this is correct? Trying to load the jdbc_static filter plugin resulted in this error: no such file to load -- logstash/util/loggable", :level=>:error}

I cannot even start to debug my syntax issues until Logserver says it is okay to play with the jdbc stuff.

So, if anyone understands this one I'm all ears. I'm hoping to develop the required filter and test from the command line. Everything I have done from the command line has helped a ton figuring out what I needed to change. This one is nasty though. I see the plugin files installed where all the other plugins are. It just isn't happy.

Please advise.

I can PM the actual config file and describe the process I am using to test, if that helps.

Thanks

Steve B

[scottwilkerson]

I've not used this plugin and it actually id for a newer version of Logstash, however I do see a similar post with possible solution here
https://github.com/logstash-plugins/log ... -413254098

In our world this path would be

Code: Select all

/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-java/lib/logstash/util/loggable.rb

I do want to say you should try this on a test server as I have not actually verified this, but wanted to offer it up as a suggestion.

[SteveBeauchemin]

Scott,

Very Nice... Thank you.

Now I can start debugging the rest of the filter. That post got me to a new level where I have to now solve SQL issues.

This is a good thing.

This error is something I can actually work with. Very cool.

Code: Select all

Pipeline aborted due to error {:exception=>"LogStash::Filters::Jdbc::ConnectionJdbcException", :error=>"Java::JavaSql::SQLException: The server time zone value 'CDT' is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the serverTimezone configuration property) to use a more specifc time zone value if you want to utilize time zone support."

This is what I wanted. Not a show stopper like that missing file was. This looks like it just needs a setting changed.

Have a good weekend!

Thanks

Steve B

[scottwilkerson]

You have a good weekend as well sir!