Logstash keeps switching to Active (exited)

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
rferebee
Posts: 733
Joined: Wed Jul 11, 2018 11:37 am

Logstash keeps switching to Active (exited)

Post by rferebee »

Good morning,

We experienced a major issue with our vSphere environment yesterday and since then one of our Log Server nodes has logstash service issues.

For some reason, it keeps switching it's status to Active (exited) despite my best efforts to keep it running. Everything in the environment looks okay except for this one problem.

Code: Select all

root@nagioslscc3:/root>systemctl status logstash
● logstash.service - LSB: Logstash
   Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
   Active: active (exited) since Wed 2020-09-16 07:24:17 PDT; 6min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 104280 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
  Process: 104293 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)

Sep 16 07:24:30 nagioslscc3 logstash[104293]: _log at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/cabin-0.8.1/lib/cabin/mixins/logger.rb:113
Sep 16 07:24:30 nagioslscc3 logstash[104293]: log_with_level at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/cabin-0.8.1/lib/cabin/mixins/logger.rb:90
Sep 16 07:24:30 nagioslscc3 logstash[104293]: warn at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/cabin-0.8.1/lib/cabin/mixins/logger.rb:75
Sep 16 07:24:30 nagioslscc3 logstash[104293]: execute at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/agent.rb:248
Sep 16 07:24:30 nagioslscc3 logstash[104293]: run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/runner.rb:94
Sep 16 07:24:30 nagioslscc3 logstash[104293]: call at org/jruby/RubyProc.java:281
Sep 16 07:24:30 nagioslscc3 logstash[104293]: run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/runner.rb:99
Sep 16 07:24:30 nagioslscc3 logstash[104293]: call at org/jruby/RubyProc.java:281
Sep 16 07:24:30 nagioslscc3 logstash[104293]: initialize at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:24
Sep 16 07:24:30 nagioslscc3 runuser[104303]: pam_unix(runuser:session): session closed for user nagios
I appreciate your assistance.
Top
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Logstash keeps switching to Active (exited)

Post by cdienger »

Is there anything else logged in /var/log/logstash? Please provide a copy of the recent logs if possible.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Top
rferebee
Posts: 733
Joined: Wed Jul 11, 2018 11:37 am

Re: Logstash keeps switching to Active (exited)

Post by rferebee »

So, it's really strange. There's a logstash.log file from today but it has no size (0 kb).

I tried stopping and starting logstash which changes the date of the logstash.log file, but still the system does not write anything to it.

I'll PM you the one from yesterday. Maybe that will tell you something?
Top
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Logstash keeps switching to Active (exited)

Post by cdienger »

Nothing obvious is jumping out that would cause this. Try running logstash on the command line of the machine having issues with this:

Code: Select all

/usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
This may give us some more details to work with. Please also PM me a profile from this system.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Top
rferebee
Posts: 733
Joined: Wed Jul 11, 2018 11:37 am

Re: Logstash keeps switching to Active (exited)

Post by rferebee »

I ran the command you provided:

Code: Select all

root@nagioslscc3:/var>/usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
Sending logstash logs to /var/log/logstash/logstash.log.
Settings: User set pipeline workers: 4, Default pipeline workers: 12
Exception in thread "pipeline.main" java.nio.BufferOverflowException
        at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:194)
        at org.jruby.util.io.ChannelStream.bufferedWrite(ChannelStream.java:1100)
        at org.jruby.util.io.ChannelStream.fwrite(ChannelStream.java:1277)
        at org.jruby.RubyIO.fwrite(RubyIO.java:1560)
        at org.jruby.RubyIO.write(RubyIO.java:1431)
        at org.jruby.RubyIO$INVOKER$i$1$0$write.call(RubyIO$INVOKER$i$1$0$write.gen)
        at org.jruby.RubyClass.finvoke(RubyClass.java:743)
        at org.jruby.runtime.Helpers.invoke(Helpers.java:505)
        at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:364)
        at org.jruby.RubyIO.write(RubyIO.java:2509)
        at org.jruby.RubyIO.putsSingle(RubyIO.java:2497)
        at org.jruby.RubyIO.puts1(RubyIO.java:2426)
        at org.jruby.RubyIO.puts(RubyIO.java:2399)
        at org.jruby.RubyIO$INVOKER$i$puts.call(RubyIO$INVOKER$i$puts.gen)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.IfNode.interpret(IfNode.java:116)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
        at org.jruby.runtime.Block.yield(Block.java:142)
        at org.jruby.ext.thread.Mutex.synchronize(Mutex.java:149)
        at org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at org.jruby.runtime.callsite.ShiftLeftCallSite.call(ShiftLeftCallSite.java:24)
        at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at org.jruby.runtime.callsite.ShiftLeftCallSite.call(ShiftLeftCallSite.java:24)
        at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.IfNode.interpret(IfNode.java:116)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
        at org.jruby.runtime.Block.yield(Block.java:142)
        at org.jruby.RubyHash$13.visit(RubyHash.java:1355)
        at org.jruby.RubyHash.visitLimited(RubyHash.java:648)
        at org.jruby.RubyHash.visitAll(RubyHash.java:634)
        at org.jruby.RubyHash.iteratorVisitAll(RubyHash.java:1306)
        at org.jruby.RubyHash.each_pairCommon(RubyHash.java:1351)
        at org.jruby.RubyHash.each19(RubyHash.java:1342)
        at org.jruby.RubyHash$INVOKER$i$0$0$each19.call(RubyHash$INVOKER$i$0$0$each19.gen)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
        at org.jruby.runtime.Block.yield(Block.java:142)
        at org.jruby.ext.thread.Mutex.synchronize(Mutex.java:149)
        at org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:225)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:219)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:346)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:204)
        at org.jruby.ast.FCallTwoArgNode.interpret(FCallTwoArgNode.java:38)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:268)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:235)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:366)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:238)
        at org.jruby.ast.FCallThreeArgNode.interpret(FCallThreeArgNode.java:40)
        at org.jruby.ast.IfNode.interpret(IfNode.java:116)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:300)
        at org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:64)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:214)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:346)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:204)
        at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.RescueBodyNode.interpret(RescueBodyNode.java:108)
        at org.jruby.ast.RescueNode.handleException(RescueNode.java:174)
        at org.jruby.ast.RescueNode.interpret(RescueNode.java:120)
        at org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:300)
        at org.jruby.RubyProc.call(RubyProc.java:230)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
        at java.lang.Thread.run(Thread.java:748)
When I ran it the first time, there was an I/O error saying that /var was out of space. I'm trying to figure out what's causing that as it does appear to be the case:

Code: Select all

/dev/mapper/centos7-var        3.9G  3.8G     0 100% /var
Any ideas on what is safe to clear out? Thank you.
Top
rferebee
Posts: 733
Joined: Wed Jul 11, 2018 11:37 am

Re: Logstash keeps switching to Active (exited)

Post by rferebee »

I went through the various directories inside of /var and the one thing that jumped out at me is the current elasticseach .log file on the node I'm having trouble with is 2.8GBs while the other nodes are only about 4MBs.

I think if we can clear out that log file and start a new one, that might correct the issue with the logstash service.

I can't get a system profile, because the node won't let me login to it.
You do not have the required permissions to view the files attached to this post.
Top
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Logstash keeps switching to Active (exited)

Post by cdienger »

Stop the elasticsearch service and clear its log directory:

Code: Select all

systemctl stop elasticsearch
rm -rf /var/log/elasticsearch/*
systemctl start elasticsearch
Wait a few minutes and then try running logstash again.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Top
rferebee
Posts: 733
Joined: Wed Jul 11, 2018 11:37 am

Re: Logstash keeps switching to Active (exited)

Post by rferebee »

You can lock this thread, I resolved the issue myself. Thank you.
Top
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:
Contact scottwilkerson

Re: Logstash keeps switching to Active (exited)

Post by scottwilkerson »

rferebee wrote:You can lock this thread, I resolved the issue myself. Thank you.
Great!

Locking thread
Former Nagios employee
Creator:
Human Design Website
Get Your Human Design Chart
Top
Locked

Return to “Nagios Log Server”