installing beats plugin

[meerkat2u]

Hi all,

I'm evaluating the Log Server using the downloaded ovm and installed as a virtual on my network. One of the needs I'm looking at is to capture large numbers of application logs from a number of linux hosts. The beats plugin with the filebeat log stash forwarder seems appropriate from my reading so far. I have installed filebeat successfully on a separate test host.

I'm having problems installing the beats plugin logstash-input-beats on my log server as follows, there are version conflicts, dependencies on other components. I've upgraded to latest version of the log server 1.3.0 but still the same issue. Any ideas? I don't see the command "bundle update" available at the command-line.

Code: Select all

[root@localhost logstash]# bin/plugin install logstash-input-beats
Validating logstash-input-beats
Installing logstash-input-beats
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-codec-multiline":
  In snapshot (Gemfile.lock):
    logstash-codec-multiline (= 0.1.9)

  In Gemfile:
    logstash-input-beats (>= 0) java depends on
      logstash-codec-multiline (~> 2.0.3) java

    logstash-codec-multiline (>= 0) java

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Bundler could not find compatible versions for gem "concurrent-ruby":
  In snapshot (Gemfile.lock):
    concurrent-ruby (= 0.8.0)

  In Gemfile:
    logstash-input-lumberjack (>= 0) java depends on
      concurrent-ruby (>= 0) java

    logstash-input-lumberjack (>= 0) java depends on
      concurrent-ruby (>= 0) java

    logstash-input-lumberjack (>= 0) java depends on
      concurrent-ruby (>= 0) java

    logstash-input-lumberjack (>= 0) java depends on
      concurrent-ruby (>= 0) java

    logstash-input-beats (>= 0) java depends on
      concurrent-ruby (= 0.9.1) java

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Bundler could not find compatible versions for gem "logstash-core":
  In snapshot (Gemfile.lock):
    logstash-core (= 1.5.1)

  In Gemfile:
    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core (< 2.0.0, >= 1.4.0) java
.
.
.
    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core (< 2.0.0, >= 1.4.0) java

    logstash-input-beats (>= 0) java depends on
      logstash-core (< 3.0.0, >= 1.5.4) java

    logstash-core (= 1.5.1) java

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

Thanks
Des

[jolson]

Try this out:

Code: Select all

bin/plugin update logstash-codec-multiline
bin/plugin install logstash-input-beats

[meerkat2u]

Closer but no go. It upgrades logstash-codec-multiline from current 0.1.9 to 1.0.0 only but not to the >2.0.3 that beats requires.

Code: Select all

[root@localhost logstash]# bin/plugin update logstash-codec-multiline
Updating logstash-codec-multiline
Updated logstash-codec-multiline 0.1.9 to 1.0.0
Updated logstash-patterns-core 0.1.10 to 0.4.0

Code: Select all

[root@localhost logstash]# bin/plugin install logstash-input-beats
Validating logstash-input-beats
Installing logstash-input-beats
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-codec-multiline":
  In snapshot (Gemfile.lock):
    logstash-codec-multiline (= 1.0.0)

  In Gemfile:
    logstash-input-beats (>= 0) java depends on
      logstash-codec-multiline (~> 2.0.3) java

    logstash-codec-multiline (>= 0) java

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Bundler could not find compatible versions for gem "concurrent-ruby":
  In snapshot (Gemfile.lock):
    concurrent-ruby (= 0.8.0)

  In Gemfile:
    logstash-input-lumberjack (>= 0) java depends on
      concurrent-ruby (>= 0) java

[jolson]

It looks like the version of Logstash needs to be updated to 2.1. I'll request this update in our next revision - I have it mostly working on a test box here. If you want test instructions, I'd be happy to give them to you (thought at that point your installation would be unsupported). Thank you!

[meerkat2u]

Thanks that will be great, I appreciate you following it up. If you could provide me with test instructions, I understand the conditions they will run under. This is a proof of concept install here so there will be a re-install anyway.

Cheers,
Des

[jolson]

Note: This procedure is _extremely_ experimental and will cause things to break.

Test instructions:

First, download the latest version of logstash and store it somewhere:

Code: Select all

wget https://download.elastic.co/logstash/logstash/logstash-2.1.0.tar.gz

Extract it and change directory into it:

Code: Select all

tar zxf logstash-2.1.0.tar.gz
cd logstash-2.1.0

Copy all files to the standard install directory:

Code: Select all

cp --force -R -p * /usr/local/nagioslogserver/logstash/

Modify the default output file, and make it immutable:

Code: Select all

vi /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf

Code: Select all

output {
    elasticsearch {
        document_type => '%{type}'
        workers => 4
    }
}

Code: Select all

chattr +i /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf

Restart Logstash:

Code: Select all

service logstash restart

That should do it - your logs will enter the system at this point, but there is still work that needs to be done to get this new version to work with the GUI properly. I would not expect 'verify' or 'apply config' to work properly if at all - modify the logstash configs from the command line and restart logstash to test out your changes. Thanks!

[meerkat2u]

Thanks for your help so far. I've followed the test instructions and the beats plugin has installed ok but I've hit a problem, the logstash service crashes on start up. This is the only error I see in /var/log/logstash/logstash.log even with debug option in the command line. Any ideas?

Code: Select all

{:timestamp=>"2015-12-02T21:38:40.509000+1100", :message=>"Error: The setting `host` in plugin `elasticsearch` is obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask."}
{:timestamp=>"2015-12-02T21:38:40.519000+1100", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}

The only additional change I have made beyond the test instructions supplied was to add the beats plugin to the /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf as follows

Code: Select all

[root@localhost conf.d]# cat 000_inputs.conf 
# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 26 Nov 2015 12:11:51 -0500
#

#
# Global inputs
#

input {
    syslog {
        type => 'syslog'
        port => 5544
    }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
    beats {
        type => 'beats'
        port => 5545
    }
}

[meerkat2u]

I thought perhaps adding the 'hosts' parameter to the output file 999_outputs.conf as there was a 'host' parameter in the original version but no change, it still fails to start with the same error message in the log file

Code: Select all

output {
    elasticsearch {
        hosts => 'localhost'
        document_type => '%{type}'
        workers => 4
    }
}

[jolson]

I hit that problem as well, and changes 'host' to 'hosts' before deciding that the 'hosts' directive was unecessary in general. Are you certain that you restarted logstash after removing the directive?

[meerkat2u]

I've definitely restarted the logstash service, a few times, to double check that I've followed the same process you did.
Latest run:

I've removed the 'hosts' parameter as follows

Code: Select all

[root@localhost conf.d]# vi 999_outputs.conf
as follows:

output {
    elasticsearch {
        document_type => '%{type}'
        workers => 4
    }
}

and restarted the logstash daemon

Code: Select all

[root@localhost conf.d]# chattr +i 999_outputs.conf
[root@localhost conf.d]# service logstash restart

[root@localhost conf.d]# service logstash status
Logstash Daemon (pid  7145) is running...

# within a minute daemon dies
[root@localhost conf.d]# service logstash status
Logstash Daemon dead but pid file exists

Output of /var/log/logstash/logstash.log:

Code: Select all

{:timestamp=>"2015-12-03T13:49:57.792000+1100", :message=>"Error: The setting `host` in plugin `elasticsearch` is obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask."}
{:timestamp=>"2015-12-03T13:49:57.812000+1100", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}

is there somewhere else it could possible pick up this plugin config from other than /usr/local/nagioslogserver/logstash/etc/conf.d? I'm thinking a runtime location?