Can't start logstash after 2.1 Upgrade

[stecino]

Logstash won't start on the node that I did the upgrade on


/etc/init.d/logstash restart
Restarting Logstash Daemon: [FAILED]
WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environment. Environment JAVA_OPTS are -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp
[ OK ]

what should I do?

[jolson]

Are you certain that logstash isn't already running?

Code: Select all

ps -ef | grep logstash

If not, check the logstash logs for some more detail regarding the failure.

Code: Select all

cat /var/log/logstash/logstash.log

Let me know if there's any good information in that log. Thanks!

[stecino]

tail -f /var/log/logstash/logstash.log
{:timestamp=>"2015-08-05T19:01:59.293000+0000", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}
{:timestamp=>"2015-08-05T19:05:42.610000+0000", :message=>"Invalid setting for geoip filter plugin:\n\n filter {\n geoip {\n # This setting must be a path\n # File does not exist or cannot be opened /usr/local/nagioslogserver/logstash/vendor/geoip/GeoLiteCity.dat\n database => \"/usr/local/nagioslogserver/logstash/vendor/geoip/GeoLiteCity.dat\"\n ...\n }\n }", :level=>:error}
{:timestamp=>"2015-08-05T19:05:42.617000+0000", :message=>"Error: Something is wrong with your configuration."}
{:timestamp=>"2015-08-05T19:05:42.619000+0000", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}
{:timestamp=>"2015-08-05T19:06:41.729000+0000", :message=>"Invalid setting for geoip filter plugin:\n\n filter {\n geoip {\n # This setting must be a path\n # File does not exist or cannot be opened /usr/local/nagioslogserver/logstash/vendor/geoip/GeoLiteCity.dat\n database => \"/usr/local/nagioslogserver/logstash/vendor/geoip/GeoLiteCity.dat\"\n ...\n }\n }", :level=>:error}
{:timestamp=>"2015-08-05T19:06:41.736000+0000", :message=>"Error: Something is wrong with your configuration."}
{:timestamp=>"2015-08-05T19:06:41.738000+0000", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}
{:timestamp=>"2015-08-05T19:13:08.263000+0000", :message=>"Invalid setting for geoip filter plugin:\n\n filter {\n geoip {\n # This setting must be a path\n # File does not exist or cannot be opened /usr/local/nagioslogserver/logstash/vendor/geoip/GeoLiteCity.dat\n database => \"/usr/local/nagioslogserver/logstash/vendor/geoip/GeoLiteCity.dat\"\n ...\n }\n }", :level=>:error}
{:timestamp=>"2015-08-05T19:13:08.271000+0000", :message=>"Error: Something is wrong with your configuration."}
{:timestamp=>"2015-08-05T19:13:08.272000+0000", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}
{:timestamp=>"2015-08-05T19:13:41.409000+0000", :message=>"Invalid setting for geoip filter plugin:\n\n filter {\n geoip {\n # This setting must be a path\n # File does not exist or cannot be opened /usr/local/nagioslogserver/logstash/vendor/geoip/GeoLiteCity.dat\n database => \"/usr/local/nagioslogserver/logstash/vendor/geoip/GeoLiteCity.dat\"\n ...\n }\n }", :level=>:error}
{:timestamp=>"2015-08-05T19:13:41.417000+0000", :message=>"Error: Something is wrong with your configuration."}
{:timestamp=>"2015-08-05T19:13:41.418000+0000", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}

[jolson]

Lets take a look at your filter configuration - it's possible that the syntax differs between the old version of logstash and the new:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf

[stecino]

Looks like when I did the upgrade it didn't preserve the directory structure

On the node without an upgrade

pwd
/usr/local/nagioslogserver/logstash/vendor
ls -al
total 32
drwxrwxr-x 8 nagios nagios 4096 Nov 21 2014 .
drwxrwxr-x 9 nagios nagios 4096 Nov 21 2014 ..
drwxrwxr-x 3 nagios nagios 4096 Nov 21 2014 bundle
drwxrwxr-x 2 nagios nagios 4096 Nov 21 2014 collectd
drwxrwxr-x 2 nagios nagios 4096 Nov 21 2014 geoip
drwxrwxr-x 3 nagios nagios 4096 Nov 21 2014 jar
drwxrwxr-x 7 nagios nagios 4096 Nov 21 2014 kibana
drwxrwxr-x 2 nagios nagios 4096 Nov 21 2014 ua-parser

Node with upgrade

pwd
/usr/local/nagioslogserver/logstash/vendor
ls -al
total 16
drwxrwxr-x 4 nagios nagios 4096 Aug 5 18:54 .
drwxrwxr-x 9 nagios nagios 4096 Aug 5 18:54 ..
drwxrwxr-x 4 nagios nagios 4096 Aug 5 18:54 bundle
drwxrwxr-x 6 nagios nagios 4096 Aug 5 18:54 jruby

[stecino]

Lets take a look at your filter configuration - it's possible that the syntax differs between the old version of logstash and the new:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf

Here is a snippet

filter {
ruby {
code => "begin; if !event['message'].nil?; event['message'] = event['message'].force_encoding('ASCII-8BIT').encode('UTF-8', :invalid => :replace, :undef => :replace, :replace => '?'); end; rescue; end;"
}
geoip {
add_tag => [ 'geoip' ]
database => '/usr/local/nagioslogserver/logstash/vendor/geoip/GeoLiteCity.dat'
source => 'host'
}

It's looking for this

[jolson]

Logstash now uses ruby gems for plugins - this is why the path you mentioned has been overwritten.

Regarding how to approach the issue:

database
edit

Value type is path
There is no default value for this setting.

The path to the GeoIP database file which Logstash should use. Country, City, ASN, ISP and organization databases are supported.

If not specified, this will default to the GeoLiteCity database that ships with Logstash.

You can either remove the database setting from your filter and restart logstash (using the included GeoLiteCity Database), or you can re-add your custom database in a location less likely to be overwritten. Does that make sense?

[stecino]

Logstash now uses ruby gems for plugins - this is why the path you mentioned has been overwritten.

Regarding how to approach the issue:

database
edit

Value type is path
There is no default value for this setting.

The path to the GeoIP database file which Logstash should use. Country, City, ASN, ISP and organization databases are supported.

If not specified, this will default to the GeoLiteCity database that ships with Logstash.

You can either remove the database setting from your filter and restart logstash (using the included GeoLiteCity Database), or you can re-add your custom database in a location less likely to be overwritten. Does that make sense?

Ok so for a quick fix, I did create geoip forlder and copied the .dat file. It fixed the issue. I will probably put it somewhere it won't get overwritten. Logstash is now up and running. Thanks. I will let you know if I see any other issues

[jolson]

Sounds good to me - I'm happy to hear that this is back up and running.

[stecino]

Sounds good to me - I'm happy to hear that this is back up and running.

I actually have another problem: Cluster status shows the indecies, but I click on them it shown nothing