Page 1 of 1
Logstash keeps switching to Active (exited)
Posted: Wed Sep 16, 2020 9:34 am
by rferebee
Good morning,
We experienced a major issue with our vSphere environment yesterday and since then one of our Log Server nodes has logstash service issues.
For some reason, it keeps switching it's status to Active (exited) despite my best efforts to keep it running. Everything in the environment looks okay except for this one problem.
Code: Select all
root@nagioslscc3:/root>systemctl status logstash
● logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
Active: active (exited) since Wed 2020-09-16 07:24:17 PDT; 6min ago
Docs: man:systemd-sysv-generator(8)
Process: 104280 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 104293 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
Sep 16 07:24:30 nagioslscc3 logstash[104293]: _log at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/cabin-0.8.1/lib/cabin/mixins/logger.rb:113
Sep 16 07:24:30 nagioslscc3 logstash[104293]: log_with_level at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/cabin-0.8.1/lib/cabin/mixins/logger.rb:90
Sep 16 07:24:30 nagioslscc3 logstash[104293]: warn at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/cabin-0.8.1/lib/cabin/mixins/logger.rb:75
Sep 16 07:24:30 nagioslscc3 logstash[104293]: execute at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/agent.rb:248
Sep 16 07:24:30 nagioslscc3 logstash[104293]: run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/runner.rb:94
Sep 16 07:24:30 nagioslscc3 logstash[104293]: call at org/jruby/RubyProc.java:281
Sep 16 07:24:30 nagioslscc3 logstash[104293]: run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/runner.rb:99
Sep 16 07:24:30 nagioslscc3 logstash[104293]: call at org/jruby/RubyProc.java:281
Sep 16 07:24:30 nagioslscc3 logstash[104293]: initialize at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:24
Sep 16 07:24:30 nagioslscc3 runuser[104303]: pam_unix(runuser:session): session closed for user nagios
I appreciate your assistance.
Re: Logstash keeps switching to Active (exited)
Posted: Thu Sep 17, 2020 2:17 pm
by cdienger
Is there anything else logged in /var/log/logstash? Please provide a copy of the recent logs if possible.
Re: Logstash keeps switching to Active (exited)
Posted: Thu Sep 17, 2020 5:04 pm
by rferebee
So, it's really strange. There's a logstash.log file from today but it has no size (0 kb).
I tried stopping and starting logstash which changes the date of the logstash.log file, but still the system does not write anything to it.
I'll PM you the one from yesterday. Maybe that will tell you something?
Re: Logstash keeps switching to Active (exited)
Posted: Fri Sep 18, 2020 3:54 pm
by cdienger
Nothing obvious is jumping out that would cause this. Try running logstash on the command line of the machine having issues with this:
Code: Select all
/usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
This may give us some more details to work with. Please also PM me a profile from this system.
Re: Logstash keeps switching to Active (exited)
Posted: Mon Sep 21, 2020 9:24 am
by rferebee
I ran the command you provided:
Code: Select all
root@nagioslscc3:/var>/usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
Sending logstash logs to /var/log/logstash/logstash.log.
Settings: User set pipeline workers: 4, Default pipeline workers: 12
Exception in thread "pipeline.main" java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:194)
at org.jruby.util.io.ChannelStream.bufferedWrite(ChannelStream.java:1100)
at org.jruby.util.io.ChannelStream.fwrite(ChannelStream.java:1277)
at org.jruby.RubyIO.fwrite(RubyIO.java:1560)
at org.jruby.RubyIO.write(RubyIO.java:1431)
at org.jruby.RubyIO$INVOKER$i$1$0$write.call(RubyIO$INVOKER$i$1$0$write.gen)
at org.jruby.RubyClass.finvoke(RubyClass.java:743)
at org.jruby.runtime.Helpers.invoke(Helpers.java:505)
at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:364)
at org.jruby.RubyIO.write(RubyIO.java:2509)
at org.jruby.RubyIO.putsSingle(RubyIO.java:2497)
at org.jruby.RubyIO.puts1(RubyIO.java:2426)
at org.jruby.RubyIO.puts(RubyIO.java:2399)
at org.jruby.RubyIO$INVOKER$i$puts.call(RubyIO$INVOKER$i$puts.gen)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.ast.IfNode.interpret(IfNode.java:116)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
at org.jruby.runtime.Block.yield(Block.java:142)
at org.jruby.ext.thread.Mutex.synchronize(Mutex.java:149)
at org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at org.jruby.runtime.callsite.ShiftLeftCallSite.call(ShiftLeftCallSite.java:24)
at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at org.jruby.runtime.callsite.ShiftLeftCallSite.call(ShiftLeftCallSite.java:24)
at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.IfNode.interpret(IfNode.java:116)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
at org.jruby.runtime.Block.yield(Block.java:142)
at org.jruby.RubyHash$13.visit(RubyHash.java:1355)
at org.jruby.RubyHash.visitLimited(RubyHash.java:648)
at org.jruby.RubyHash.visitAll(RubyHash.java:634)
at org.jruby.RubyHash.iteratorVisitAll(RubyHash.java:1306)
at org.jruby.RubyHash.each_pairCommon(RubyHash.java:1351)
at org.jruby.RubyHash.each19(RubyHash.java:1342)
at org.jruby.RubyHash$INVOKER$i$0$0$each19.call(RubyHash$INVOKER$i$0$0$each19.gen)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
at org.jruby.runtime.Block.yield(Block.java:142)
at org.jruby.ext.thread.Mutex.synchronize(Mutex.java:149)
at org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:225)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:219)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:346)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:204)
at org.jruby.ast.FCallTwoArgNode.interpret(FCallTwoArgNode.java:38)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:268)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:235)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:366)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:238)
at org.jruby.ast.FCallThreeArgNode.interpret(FCallThreeArgNode.java:40)
at org.jruby.ast.IfNode.interpret(IfNode.java:116)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
at org.jruby.runtime.Block.call(Block.java:101)
at org.jruby.RubyProc.call(RubyProc.java:300)
at org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:64)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:214)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:346)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:204)
at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.ast.RescueBodyNode.interpret(RescueBodyNode.java:108)
at org.jruby.ast.RescueNode.handleException(RescueNode.java:174)
at org.jruby.ast.RescueNode.interpret(RescueNode.java:120)
at org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
at org.jruby.runtime.Block.call(Block.java:101)
at org.jruby.RubyProc.call(RubyProc.java:300)
at org.jruby.RubyProc.call(RubyProc.java:230)
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
at java.lang.Thread.run(Thread.java:748)
When I ran it the first time, there was an I/O error saying that /var was out of space. I'm trying to figure out what's causing that as it does appear to be the case:
Code: Select all
/dev/mapper/centos7-var 3.9G 3.8G 0 100% /var
Any ideas on what is safe to clear out? Thank you.
Re: Logstash keeps switching to Active (exited)
Posted: Mon Sep 21, 2020 9:35 am
by rferebee
I went through the various directories inside of /var and the one thing that jumped out at me is the current elasticseach .log file on the node I'm having trouble with is 2.8GBs while the other nodes are only about 4MBs.
I think if we can clear out that log file and start a new one, that might correct the issue with the logstash service.
I can't get a system profile, because the node won't let me login to it.
Re: Logstash keeps switching to Active (exited)
Posted: Mon Sep 21, 2020 4:01 pm
by cdienger
Stop the elasticsearch service and clear its log directory:
Code: Select all
systemctl stop elasticsearch
rm -rf /var/log/elasticsearch/*
systemctl start elasticsearch
Wait a few minutes and then try running logstash again.
Re: Logstash keeps switching to Active (exited)
Posted: Mon Sep 21, 2020 4:35 pm
by rferebee
You can lock this thread, I resolved the issue myself. Thank you.
Re: Logstash keeps switching to Active (exited)
Posted: Mon Sep 21, 2020 4:42 pm
by scottwilkerson
rferebee wrote:You can lock this thread, I resolved the issue myself. Thank you.
Great!
Locking thread