Logstash service not running. Reboot did not correct

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
krobertson71
Posts: 444
Joined: Tue Feb 11, 2014 10:16 pm

Re: Logstash service not running. Reboot did not correct

Post by krobertson71 »

/etc/init.d/logstash start

Code: Select all

sudo /etc/init.d/logstash stop
Stopping Logstash Daemon: Errno::EBADF: Bad file descriptor - Bad file descriptor
          each at org/jruby/RubyIO.java:3542
  tcp_receiver at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:175
  tcp_listener at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:161
                                                           [  OK  ]

/etc/init.d/logstash stop

Code: Select all

sudo /etc/init.d/logstash start
Starting Logstash Daemon: WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environment. Environment JAVA_OPTS are -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp
                                                           [  OK  ]
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Logstash service not running. Reboot did not correct

Post by mcapra »

Might have been a one-time issue. Everything seems functional now. Definitely let us know if the same issue pops up again!
Former Nagios employee
https://www.mcapra.com/
Top
sgiworks
Posts: 197
Joined: Mon Mar 21, 2016 11:38 am

Re: Logstash service not running. Reboot did not correct

Post by sgiworks »

I have similar issue, service starts and stops after sometime. What was the fix applied.

Starting Logstash Daemon: WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environment. Environment JAVA_OPTS are -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp
[ OK ]
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Logstash service not running. Reboot did not correct

Post by mcapra »

This command produced the "green" logstash status by bypassing the init.d script:

Code: Select all

/usr/local/nagioslogserver/logstash/bin/logstash -f /usr/local/nagioslogserver/logstash/etc/conf.d
It will also give more meaningful output if the service is unable to start or crashes immediately.
Former Nagios employee
https://www.mcapra.com/
Top
sgiworks
Posts: 197
Joined: Mon Mar 21, 2016 11:38 am

Re: Logstash service not running. Reboot did not correct

Post by sgiworks »

[root@ip-10-1-252-201 ec2-user]# /usr/local/nagioslogserver/logstash/bin/logstash -f /usr/local/nagioslogserver/logstash/etc/conf.d
Sep 07, 2016 7:22:32 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [2f526f36-8df4-4c76-b884-13a523776c3b] loaded [], sites []
Sep 07, 2016 7:22:33 PM org.elasticsearch.common.network.NetworkUtils <clinit>
WARNING: failed to resolve local host, fallback to loopback
java.net.UnknownHostException: ip-10-1-252-201.imcs.loca: ip-10-1-252-201.imcs.loca: unknown error
at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
at org.elasticsearch.common.network.NetworkUtils.<clinit>(NetworkUtils.java:55)
at org.elasticsearch.transport.netty.NettyTransport.createClientBootstrap(NettyTransport.java:329)
at org.elasticsearch.transport.netty.NettyTransport.doStart(NettyTransport.java:224)
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
at org.elasticsearch.transport.TransportService.doStart(TransportService.java:153)
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:189)
at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:123)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.jruby.javasupport.JavaConstructor.newInstanceDirect(JavaConstructor.java:268)
at org.jruby.java.invokers.ConstructorInvoker.call(ConstructorInvoker.java:83)
at org.jruby.java.invokers.ConstructorInvoker.call(ConstructorInvoker.java:174)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
at org.jruby.java.proxies.ConcreteJavaProxy$InitializeMethod.call(ConcreteJavaProxy.java:56)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
at org.jruby.RubyClass.newInstance(RubyClass.java:856)
at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroOrOneOrNBlock.call(JavaMethod.java:301)
at org.jruby.java.proxies.ConcreteJavaProxy$NewMethod.call(ConcreteJavaProxy.java:155)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)
at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
at org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)
at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
at org.jruby.ast.InstAsgnNode.interpret(InstAsgnNode.java:95)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:187)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
at org.jruby.ast.VCallNode.interpret(VCallNode.java:88)
at org.jruby.ast.InstAsgnNode.interpret(InstAsgnNode.java:95)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
at org.jruby.RubyClass.newInstance(RubyClass.java:856)
at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroOrOneOrNBlock.call(JavaMethod.java:301)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)
at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
at org.jruby.runtime.Block.yield(Block.java:142)
at org.jruby.RubyArray.collect(RubyArray.java:2399)
at org.jruby.RubyArray.map19(RubyArray.java:2412)
at org.jruby.RubyArray$INVOKER$i$0$0$map19.call(RubyArray$INVOKER$i$0$0$map19.gen)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:316)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:145)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
at org.jruby.ast.InstAsgnNode.interpret(InstAsgnNode.java:95)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.IfNode.interpret(IfNode.java:118)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:112)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:169)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:286)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:85)
at org.jruby.RubySymbol$1.yieldInner(RubySymbol.java:461)
at org.jruby.RubySymbol$1.yield(RubySymbol.java:481)
at org.jruby.runtime.Block.yield(Block.java:142)
at org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
at org.jruby.RubyArray.each(RubyArray.java:1613)
at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:316)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:145)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:149)
at org.jruby.ast.CallNoArgBlockPassNode.interpret(CallNoArgBlockPassNode.java:53)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:187)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
at org.jruby.ast.VCallNode.interpret(VCallNode.java:88)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:187)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
at org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
at org.jruby.ast.EnsureNode.interpret(EnsureNode.java:96)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:187)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
at org.jruby.ast.ReturnNode.interpret(ReturnNode.java:92)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
at org.jruby.runtime.Block.call(Block.java:101)
at org.jruby.RubyProc.call(RubyProc.java:290)
at org.jruby.RubyProc.call19(RubyProc.java:271)
at org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
at org.jruby.runtime.Block.call(Block.java:101)
at org.jruby.RubyProc.call(RubyProc.java:290)
at org.jruby.RubyProc.call19(RubyProc.java:271)
at org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
at org.jruby.ast.CallSpecialArgNode.interpret(CallSpecialArgNode.java:65)
at org.jruby.ast.DAsgnNode.interpret(DAsgnNode.java:110)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
at org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
at org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
at org.jruby.runtime.Block.call(Block.java:101)
at org.jruby.RubyProc.call(RubyProc.java:290)
at org.jruby.RubyProc.call(RubyProc.java:228)
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: ip-10-1-252-201.imcs.loca: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
... 169 more

Logstash startup completed
Logstash shutdown completed
Sep 07, 2016 7:22:34 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [2f526f36-8df4-4c76-b884-13a523776c3b] loaded [], sites []
Sep 07, 2016 7:22:34 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [2f526f36-8df4-4c76-b884-13a523776c3b] loaded [], sites []
[root@ip-10-1-252-201 ec2-user]#
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Logstash service not running. Reboot did not correct

Post by mcapra »

We will continue this in the other thread and see if OP comes back :)
Former Nagios employee
https://www.mcapra.com/
Top
Locked

Return to “Nagios Log Server”