Page 1 of 1

Logstash crashes upon launch

Posted: Thu Aug 25, 2016 9:46 am
by jsmith.mim
I just finished up a manual install of Nagios Log Server on a fresh CentOS 7 VM according to the official instructions. After configuring a couple of our other VMs to send log data to Nagios Log Server, I noticed that nothing was showing up in the UI.

It turns out Logstash is crashing when I try to launch it:

Code: Select all

Aug 25 09:34:28 logs.mimsoftware.com systemd[1]: Started LSB: Logstash.
Aug 25 09:34:28 logs.mimsoftware.com systemd[1]: Started LSB: Logstash.
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: LoadError: load error: jopenssl/load -- java.lang.InternalError: null
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: require at org/jruby/RubyKernel.java:1072
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: require at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: (root) at /usr/local/nagioslogserver/logstash/vendor/jruby/lib/ruby/shared/openssl.rb:1
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: require at org/jruby/RubyKernel.java:1072
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: require at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: (root) at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:1
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: register at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:65
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: each at org/jruby/RubyArray.java:1613
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: start_inputs at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:148
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: start_inputs at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:147
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: synchronize at org/jruby/ext/thread/Mutex.java:149
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:80
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:80
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: execute at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/agent.rb:150
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: call at org/jruby/RubyProc.java:271
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/runner.rb:87
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: call at org/jruby/RubyProc.java:271
Aug 25 09:34:45 logs.mimsoftware.com logstash[4345]: run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/runner.rb:92
Aug 25 10:27:25 logs.mimsoftware.com systemd[1]: Stopping LSB: Logstash...
Aug 25 10:27:25 logs.mimsoftware.com logstash[50939]: Stopping Logstash Daemon: [FAILED]
Any ideas?

Re: Logstash crashes upon launch

Posted: Thu Aug 25, 2016 11:46 am
by mcapra
Can you show me your java -version output?

Can you also share the contents of your /var/log/logstash/logstash.log file?

Re: Logstash crashes upon launch

Posted: Thu Aug 25, 2016 12:27 pm
by jsmith.mim
java -version:

Code: Select all

java version "1.7.0_111"
OpenJDK Runtime Environment (rhel-2.6.7.2.el7_2-x86_64 u111-b01)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)
The logstash.log file is blank.

Re: Logstash crashes upon launch

Posted: Thu Aug 25, 2016 1:26 pm
by mcapra
Do you know if your organization uses a custom repository for their Java distributions? This looks like something specific to the Java environment.

In the file /usr/local/nagioslogserver/logstash/bin/logstash.lib.sh

Around line 35 try adding the following line:

Code: Select all

JAVA_OPTS="$JAVA_OPTS -Djava.ext.dirs=/some/dummy/path"
Save the file and try to start the logstash service. Share a fresh copy of the relevant syslog information also.

Re: Logstash crashes upon launch

Posted: Thu Aug 25, 2016 1:52 pm
by jsmith.mim
No, no custom repositories.

I installed OpenJDK 1.8 from the official repositories, and it works now.

Problem solved.

Re: Logstash crashes upon launch

Posted: Thu Aug 25, 2016 1:56 pm
by mcapra
Very strange. Thanks for sharing your solution though!

Is it alright if I lock this thread and mark the issue as resolved?