Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

All shards failed for phase: [query_fetch]

https://support.nagios.com/forum/viewtopic.php?t=41291

Page 1 of 1

All shards failed for phase: [query_fetch]

Posted: Wed Nov 23, 2016 8:15 am
by _asp_
Probelm on production:

after restarting elasticsearch I get follwoing errors during startup.

Code: Select all

[2016-11-23 14:02:17,056][DEBUG][action.search.type       ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] All shards failed for phase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:160)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:57)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:47)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:104)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)
        at org.elasticsearch.client.FilterClient.execute(FilterClient.java:66)
        at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.execute(BaseRestHandler.java:92)
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338)
        at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:84)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:53)
        at org.elasticsearch.rest.RestController.executeHandler(RestController.java:225)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:170)
        at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
        at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
        at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:327)
        at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
        at org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
[2016-11-23 14:02:17,059][DEBUG][action.search.type       ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] All shards failed for phase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:160)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:57)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:47)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:104)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)
        at org.elasticsearch.client.FilterClient.execute(FilterClient.java:66)
        at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.execute(BaseRestHandler.java:92)
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338)
        at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:84)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:53)
        at org.elasticsearch.rest.RestController.executeHandler(RestController.java:225)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:170)
        at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
        at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
        at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:327)
        at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
        at org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

searched the net and found that I should check: curl "localhost:9200/_cat/shards?v?"
Here is the output, but I don't know what I can read there.

Code: Select all

curl GET "localhost:9200/_cat/shards?v?"
curl: (6) Couldn't resolve host 'GET'
nagioslogserver     0 p STARTED        107 530.6kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
nagioslogserver     0 r UNASSIGNED
logstash-2016.11.23 2 p STARTED     991114 983.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.23 2 r UNASSIGNED
logstash-2016.11.23 0 p STARTED     990652 964.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.23 0 r UNASSIGNED
logstash-2016.11.23 3 p STARTED     991001 979.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.23 3 r UNASSIGNED
logstash-2016.11.23 1 p STARTED     990492 973.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.23 1 r UNASSIGNED
logstash-2016.11.23 4 p STARTED     990992 978.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.23 4 r UNASSIGNED
logstash-2016.10.10 2 p STARTED     584886 457.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.10 2 r UNASSIGNED
logstash-2016.10.10 0 p STARTED     584653   453mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.10 0 r UNASSIGNED
logstash-2016.10.10 3 p STARTED     585535 455.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.10 3 r UNASSIGNED
logstash-2016.10.10 1 p STARTED     584790 452.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.10 1 r UNASSIGNED
logstash-2016.10.10 4 p STARTED     585088 451.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.10 4 r UNASSIGNED
nagioslogserver_log 4 p STARTED     212698    22mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
nagioslogserver_log 4 r UNASSIGNED
nagioslogserver_log 0 p STARTED     212470    24mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
nagioslogserver_log 0 r UNASSIGNED
nagioslogserver_log 3 p STARTED     213852  24.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
nagioslogserver_log 3 r UNASSIGNED
nagioslogserver_log 1 p STARTED     212652    22mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
nagioslogserver_log 1 r UNASSIGNED
nagioslogserver_log 2 p STARTED     213521  24.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
nagioslogserver_log 2 r UNASSIGNED
logstash-2016.10.05 4 p STARTED     632328 508.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.05 4 r UNASSIGNED
logstash-2016.10.05 0 p STARTED     632029 511.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.05 0 r UNASSIGNED
logstash-2016.10.05 3 p STARTED     632356 511.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.05 3 r UNASSIGNED
logstash-2016.10.05 1 p STARTED     632727 498.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.05 1 r UNASSIGNED
logstash-2016.10.05 2 p STARTED     632259 506.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.05 2 r UNASSIGNED
logstash-2016.10.04 2 p STARTED     598932 479.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.04 2 r UNASSIGNED
logstash-2016.10.04 0 p STARTED     599390 475.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.04 0 r UNASSIGNED
logstash-2016.10.04 3 p STARTED     599622 479.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.04 3 r UNASSIGNED
logstash-2016.10.04 1 p STARTED     598917   482mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.04 1 r UNASSIGNED
logstash-2016.10.04 4 p STARTED     598647 479.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.04 4 r UNASSIGNED
logstash-2016.10.07 4 p STARTED     619176 492.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.07 4 r UNASSIGNED
logstash-2016.10.07 0 p STARTED     618366 498.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.07 0 r UNASSIGNED
logstash-2016.10.07 3 p STARTED     618650 497.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.07 3 r UNASSIGNED
logstash-2016.10.07 1 p STARTED     619161 499.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.07 1 r UNASSIGNED
logstash-2016.10.07 2 p STARTED     618758 498.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.07 2 r UNASSIGNED
logstash-2016.10.06 4 p STARTED     857430 555.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.06 4 r UNASSIGNED
logstash-2016.10.06 0 p STARTED     857346 551.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.06 0 r UNASSIGNED
logstash-2016.10.06 3 p STARTED     857098 555.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.06 3 r UNASSIGNED
logstash-2016.10.06 1 p STARTED     856821 555.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.06 1 r UNASSIGNED
logstash-2016.10.06 2 p STARTED     857339 556.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.06 2 r UNASSIGNED
logstash-2016.10.01 2 p STARTED     428190   352mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.01 2 r UNASSIGNED
logstash-2016.10.01 0 p STARTED     428479 352.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.01 0 r UNASSIGNED
logstash-2016.10.01 3 p STARTED     428613 349.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.01 3 r UNASSIGNED
logstash-2016.10.01 1 p STARTED     427966 349.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.01 1 r UNASSIGNED
logstash-2016.10.01 4 p STARTED     428519 348.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.01 4 r UNASSIGNED
logstash-2016.10.03 2 p STARTED     471890 390.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.03 2 r UNASSIGNED
logstash-2016.10.03 0 p STARTED     472540 388.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.03 0 r UNASSIGNED
logstash-2016.10.03 3 p STARTED     472313 390.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.03 3 r UNASSIGNED
logstash-2016.10.03 1 p STARTED     472110 390.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.03 1 r UNASSIGNED
logstash-2016.10.03 4 p STARTED     472926 387.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.03 4 r UNASSIGNED
logstash-2016.10.02 2 p STARTED     404732 323.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.02 2 r UNASSIGNED
logstash-2016.10.02 0 p STARTED     404955 327.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.02 0 r UNASSIGNED
logstash-2016.10.02 3 p STARTED     404431   326mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.02 3 r UNASSIGNED
logstash-2016.10.02 1 p STARTED     404625 327.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.02 1 r UNASSIGNED
logstash-2016.10.02 4 p STARTED     404622 324.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.02 4 r UNASSIGNED
logstash-2016.10.09 4 p STARTED     432993 338.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.09 4 r UNASSIGNED
logstash-2016.10.09 0 p STARTED     433000 342.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.09 0 r UNASSIGNED
logstash-2016.10.09 3 p STARTED     432935 340.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.09 3 r UNASSIGNED
logstash-2016.10.09 1 p STARTED     433364 341.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.09 1 r UNASSIGNED
logstash-2016.10.09 2 p STARTED     432896 339.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.09 2 r UNASSIGNED
logstash-2016.10.08 2 p STARTED     461355 370.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.08 2 r UNASSIGNED
logstash-2016.10.08 0 p STARTED     461765 370.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.08 0 r UNASSIGNED
logstash-2016.10.08 3 p STARTED     461992 368.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.08 3 r UNASSIGNED
logstash-2016.10.08 1 p STARTED     461715 374.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.08 1 r UNASSIGNED
logstash-2016.10.08 4 p STARTED     461905 371.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.08 4 r UNASSIGNED
logstash-2016.11.15 4 p STARTED     490626   412mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.15 4 r UNASSIGNED
logstash-2016.11.15 0 p STARTED     490576 411.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.15 0 r UNASSIGNED
logstash-2016.11.15 3 p STARTED     490219 411.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.15 3 r UNASSIGNED
logstash-2016.11.15 1 p STARTED     490620 410.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.15 1 r UNASSIGNED
logstash-2016.11.15 2 p STARTED     490657 410.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.15 2 r UNASSIGNED
logstash-2016.10.20 4 p STARTED     630280 490.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.20 4 r UNASSIGNED
logstash-2016.10.20 0 p STARTED     630413 493.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.20 0 r UNASSIGNED
logstash-2016.10.20 3 p STARTED     630406 494.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.20 3 r UNASSIGNED
logstash-2016.10.20 1 p STARTED     630587 495.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.20 1 r UNASSIGNED
logstash-2016.10.20 2 p STARTED     630674 493.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.20 2 r UNASSIGNED
logstash-2016.11.14 2 p STARTED     522863 412.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.14 2 r UNASSIGNED
logstash-2016.11.14 0 p STARTED     522341   411mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.14 0 r UNASSIGNED
logstash-2016.11.14 3 p STARTED     522879 410.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.14 3 r UNASSIGNED
logstash-2016.11.14 1 p STARTED     522742 411.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.14 1 r UNASSIGNED
logstash-2016.11.14 4 p STARTED     522255 412.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.14 4 r UNASSIGNED
logstash-2016.10.21 2 p STARTED     594008 488.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.21 2 r UNASSIGNED
logstash-2016.10.21 0 p STARTED     594120   477mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.21 0 r UNASSIGNED
logstash-2016.10.21 3 p STARTED     593912 488.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.21 3 r UNASSIGNED
logstash-2016.10.21 1 p STARTED     594363 490.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.21 1 r UNASSIGNED
logstash-2016.10.21 4 p STARTED     594727 488.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.21 4 r UNASSIGNED
logstash-2016.11.13 4 p STARTED     304017 231.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.13 4 r UNASSIGNED
logstash-2016.11.13 0 p STARTED     304019 231.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.13 0 r UNASSIGNED
logstash-2016.11.13 3 p STARTED     303884 228.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.13 3 r UNASSIGNED
logstash-2016.11.13 1 p STARTED     303771 231.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.13 1 r UNASSIGNED
logstash-2016.11.13 2 p STARTED     303659 230.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.13 2 r UNASSIGNED
logstash-2016.11.12 4 p STARTED     611076 375.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.12 4 r UNASSIGNED
logstash-2016.11.12 0 p STARTED     610917 375.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.12 0 r UNASSIGNED
logstash-2016.11.12 3 p STARTED     610498 374.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.12 3 r UNASSIGNED
logstash-2016.11.12 1 p STARTED     611000 377.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.12 1 r UNASSIGNED
logstash-2016.11.12 2 p STARTED     610887 374.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.12 2 r UNASSIGNED
logstash-2016.11.19 2 p STARTED     506809 357.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.19 2 r UNASSIGNED
logstash-2016.11.19 0 p STARTED     506485 356.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.19 0 r UNASSIGNED
logstash-2016.11.19 3 p STARTED     506379 356.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.19 3 r UNASSIGNED
logstash-2016.11.19 1 p STARTED     506510 355.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.19 1 r UNASSIGNED
logstash-2016.11.19 4 p STARTED     506590 357.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.19 4 r UNASSIGNED
logstash-2016.11.18 2 p STARTED     984280 573.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.18 2 r UNASSIGNED
logstash-2016.11.18 0 p STARTED     984410 580.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.18 0 r UNASSIGNED
logstash-2016.11.18 3 p STARTED     983546 572.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.18 3 r UNASSIGNED
logstash-2016.11.18 1 p STARTED     984117 575.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.18 1 r UNASSIGNED
logstash-2016.11.18 4 p STARTED     984486 567.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.18 4 r UNASSIGNED
logstash-2016.11.17 4 p STARTED     665129 488.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.17 4 r UNASSIGNED
logstash-2016.11.17 0 p STARTED     664247 484.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.17 0 r UNASSIGNED
logstash-2016.11.17 3 p STARTED     665167 488.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.17 3 r UNASSIGNED
logstash-2016.11.17 1 p STARTED     664654 484.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.17 1 r UNASSIGNED
logstash-2016.11.17 2 p STARTED     665061 489.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.17 2 r UNASSIGNED
logstash-2016.11.16 4 p STARTED     496628 410.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.16 4 r UNASSIGNED
logstash-2016.11.16 0 p STARTED     496402 408.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.16 0 r UNASSIGNED
logstash-2016.11.16 3 p STARTED     496509 410.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.16 3 r UNASSIGNED
logstash-2016.11.16 1 p STARTED     496662 409.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.16 1 r UNASSIGNED
logstash-2016.11.16 2 p STARTED     496834 409.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.16 2 r UNASSIGNED
logstash-2016.10.18 2 p STARTED     672879 516.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.18 2 r UNASSIGNED
logstash-2016.10.18 0 p STARTED     672238   512mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.18 0 r UNASSIGNED
logstash-2016.10.18 3 p STARTED     672470   509mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.18 3 r UNASSIGNED
logstash-2016.10.18 1 p STARTED     673198 503.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.18 1 r UNASSIGNED
logstash-2016.10.18 4 p STARTED     672383 512.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.18 4 r UNASSIGNED
logstash-2016.10.17 4 p STARTED     734148 515.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.17 4 r UNASSIGNED
logstash-2016.10.17 0 p STARTED     734693 513.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.17 0 r UNASSIGNED
logstash-2016.10.17 3 p STARTED     734551 507.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.17 3 r UNASSIGNED
logstash-2016.10.17 1 p STARTED     734131 513.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.17 1 r UNASSIGNED
logstash-2016.10.17 2 p STARTED     733505 501.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.17 2 r UNASSIGNED
logstash-2016.10.16 2 p STARTED     466473 351.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.16 2 r UNASSIGNED
logstash-2016.10.16 0 p STARTED     466141 351.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.16 0 r UNASSIGNED
logstash-2016.10.16 3 p STARTED     466327 350.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.16 3 r UNASSIGNED
logstash-2016.10.16 1 p STARTED     466069   352mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.16 1 r UNASSIGNED
logstash-2016.10.16 4 p STARTED     466484 351.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.16 4 r UNASSIGNED
logstash-2016.10.15 4 p STARTED     455885 358.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.15 4 r UNASSIGNED
logstash-2016.10.15 0 p STARTED     455834 354.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.15 0 r UNASSIGNED
logstash-2016.10.15 3 p STARTED     455809 355.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.15 3 r UNASSIGNED
logstash-2016.10.15 1 p STARTED     455648 354.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.15 1 r UNASSIGNED
logstash-2016.10.15 2 p STARTED     455590 352.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.15 2 r UNASSIGNED
logstash-2016.11.21 4 p STARTED    1331608   1.1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.21 4 r UNASSIGNED
logstash-2016.11.21 0 p STARTED    1331627   1.1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.21 0 r UNASSIGNED
logstash-2016.11.21 3 p STARTED    1331269   1.1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.21 3 r UNASSIGNED
logstash-2016.11.21 1 p STARTED    1331292   1.1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.21 1 r UNASSIGNED
logstash-2016.11.21 2 p STARTED    1331313   1.1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.21 2 r UNASSIGNED
logstash-2016.10.14 2 p STARTED     576615 449.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.14 2 r UNASSIGNED
logstash-2016.10.14 0 p STARTED     576538 452.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.14 0 r UNASSIGNED
logstash-2016.10.14 3 p STARTED     576827 449.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.14 3 r UNASSIGNED
logstash-2016.10.14 1 p STARTED     577297 450.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.14 1 r UNASSIGNED
logstash-2016.10.14 4 p STARTED     576277   451mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.14 4 r UNASSIGNED
logstash-2016.11.22 4 p STARTED    1701021   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.22 4 r UNASSIGNED
logstash-2016.11.22 0 p STARTED    1701135   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.22 0 r UNASSIGNED
logstash-2016.11.22 3 p STARTED    1701586   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.22 3 r UNASSIGNED
logstash-2016.11.22 1 p STARTED    1701583   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.22 1 r UNASSIGNED
logstash-2016.11.22 2 p STARTED    1700772   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.22 2 r UNASSIGNED
logstash-2016.10.13 2 p STARTED    1146166 592.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.13 2 r UNASSIGNED
logstash-2016.10.13 0 p STARTED    1146506 589.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.13 0 r UNASSIGNED
logstash-2016.10.13 3 p STARTED    1146025   588mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.13 3 r UNASSIGNED
logstash-2016.10.13 1 p STARTED    1146565 591.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.13 1 r UNASSIGNED
logstash-2016.10.13 4 p STARTED    1146851 591.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.13 4 r UNASSIGNED
logstash-2016.10.12 4 p STARTED    1132509 576.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.12 4 r UNASSIGNED
logstash-2016.10.12 0 p STARTED    1131922 582.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.12 0 r UNASSIGNED
logstash-2016.10.12 3 p STARTED    1131681 578.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.12 3 r UNASSIGNED
logstash-2016.10.12 1 p STARTED    1131503 572.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.12 1 r UNASSIGNED
logstash-2016.10.12 2 p STARTED    1131506   579mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.12 2 r UNASSIGNED
logstash-2016.11.20 2 p STARTED     403048 301.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.20 2 r UNASSIGNED
logstash-2016.11.20 0 p STARTED     402788 300.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.20 0 r UNASSIGNED
logstash-2016.11.20 3 p STARTED     403309 303.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.20 3 r UNASSIGNED
logstash-2016.11.20 1 p STARTED     403070 299.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.20 1 r UNASSIGNED
logstash-2016.11.20 4 p STARTED     403134 301.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.20 4 r UNASSIGNED
logstash-2016.10.11 4 p STARTED     650677 509.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.11 4 r UNASSIGNED
logstash-2016.10.11 0 p STARTED     650849 516.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.11 0 r UNASSIGNED
logstash-2016.10.11 3 p STARTED     650492 515.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.11 3 r UNASSIGNED
logstash-2016.10.11 1 p STARTED     650619 515.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.11 1 r UNASSIGNED
logstash-2016.10.11 2 p STARTED     651144 516.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.11 2 r UNASSIGNED
logstash-2016.10.19 2 p STARTED     767870 535.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.19 2 r UNASSIGNED
logstash-2016.10.19 0 p STARTED     767651 532.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.19 0 r UNASSIGNED
logstash-2016.10.19 3 p STARTED     766873 535.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.19 3 r UNASSIGNED
logstash-2016.10.19 1 p STARTED     768105 538.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.19 1 r UNASSIGNED
logstash-2016.10.19 4 p STARTED     767133 530.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.19 4 r UNASSIGNED
logstash-2016.11.05 4 p STARTED     497207 393.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.05 4 r UNASSIGNED
logstash-2016.11.05 0 p STARTED     497375 391.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.05 0 r UNASSIGNED
logstash-2016.11.05 3 p STARTED     497058 384.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.05 3 r UNASSIGNED
logstash-2016.11.05 1 p STARTED     497598 393.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.05 1 r UNASSIGNED
logstash-2016.11.05 2 p STARTED     497391 392.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.05 2 r UNASSIGNED
logstash-2016.11.06 4 p STARTED     456113 358.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.06 4 r UNASSIGNED
logstash-2016.11.06 0 p STARTED     456394   360mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.06 0 r UNASSIGNED
logstash-2016.11.06 3 p STARTED     456506   358mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.06 3 r UNASSIGNED
logstash-2016.11.06 1 p STARTED     456541 359.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.06 1 r UNASSIGNED
logstash-2016.11.06 2 p STARTED     456409 358.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.06 2 r UNASSIGNED
logstash-2016.11.07 4 p STARTED     689901 557.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.07 4 r UNASSIGNED
logstash-2016.11.07 0 p STARTED     689458 565.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.07 0 r UNASSIGNED
logstash-2016.11.07 3 p STARTED     689861 560.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.07 3 r UNASSIGNED
logstash-2016.11.07 1 p STARTED     689828 559.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.07 1 r UNASSIGNED
logstash-2016.11.07 2 p STARTED     690574 565.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.07 2 r UNASSIGNED
logstash-2016.11.08 4 p STARTED     724440 584.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.08 4 r UNASSIGNED
logstash-2016.11.08 0 p STARTED     723225 592.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.08 0 r UNASSIGNED
logstash-2016.11.08 3 p STARTED     724208 588.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.08 3 r UNASSIGNED
logstash-2016.11.08 1 p STARTED     723162 584.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.08 1 r UNASSIGNED
logstash-2016.11.08 2 p STARTED     723005 588.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.08 2 r UNASSIGNED
logstash-2016.11.01 2 p STARTED     587197 475.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.01 2 r UNASSIGNED
logstash-2016.11.01 0 p STARTED     586477 475.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.01 0 r UNASSIGNED
logstash-2016.11.01 3 p STARTED     587219 473.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.01 3 r UNASSIGNED
logstash-2016.11.01 1 p STARTED     587276 475.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.01 1 r UNASSIGNED
logstash-2016.11.01 4 p STARTED     586842 477.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.01 4 r UNASSIGNED
logstash-2016.11.02 2 p STARTED     667486 535.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.02 2 r UNASSIGNED
logstash-2016.11.02 0 p STARTED     667600 534.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.02 0 r UNASSIGNED
logstash-2016.11.02 3 p STARTED     667894 535.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.02 3 r UNASSIGNED
logstash-2016.11.02 1 p STARTED     667717 534.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.02 1 r UNASSIGNED
logstash-2016.11.02 4 p STARTED     666992 529.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.02 4 r UNASSIGNED
logstash-2016.11.03 2 p STARTED     670906 547.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.03 2 r UNASSIGNED
logstash-2016.11.03 0 p STARTED     670556 542.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.03 0 r UNASSIGNED
logstash-2016.11.03 3 p STARTED     670239 540.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.03 3 r UNASSIGNED
logstash-2016.11.03 1 p STARTED     670376 545.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.03 1 r UNASSIGNED
logstash-2016.11.03 4 p STARTED     670384   535mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.03 4 r UNASSIGNED
logstash-2016.11.04 4 p STARTED     658493 526.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.04 4 r UNASSIGNED
logstash-2016.11.04 0 p STARTED     658437 534.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.04 0 r UNASSIGNED
logstash-2016.11.04 3 p STARTED     658137 532.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.04 3 r UNASSIGNED
logstash-2016.11.04 1 p STARTED     658598 526.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.04 1 r UNASSIGNED
logstash-2016.11.04 2 p STARTED     658339 531.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.04 2 r UNASSIGNED
logstash-2016.11.09 4 p STARTED     714942 584.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.09 4 r UNASSIGNED
logstash-2016.11.09 0 p STARTED     714754 582.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.09 0 r UNASSIGNED
logstash-2016.11.09 3 p STARTED     714362   582mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.09 3 r UNASSIGNED
logstash-2016.11.09 1 p STARTED     715058 577.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.09 1 r UNASSIGNED
logstash-2016.11.09 2 p STARTED     714291 576.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.09 2 r UNASSIGNED
logstash-2016.10.30 2 p STARTED     527542 394.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.30 2 r UNASSIGNED
logstash-2016.10.30 0 p STARTED     527752 395.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.30 0 r UNASSIGNED
logstash-2016.10.30 3 p STARTED     527476 393.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.30 3 r UNASSIGNED
logstash-2016.10.30 1 p STARTED     527393 394.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.30 1 r UNASSIGNED
logstash-2016.10.30 4 p STARTED     528661 392.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.30 4 r UNASSIGNED
logstash-2016.10.31 2 p STARTED     618237 501.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.31 2 r UNASSIGNED
logstash-2016.10.31 0 p STARTED     619290 498.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.31 0 r UNASSIGNED
logstash-2016.10.31 3 p STARTED     619356 503.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.31 3 r UNASSIGNED
logstash-2016.10.31 1 p STARTED     619200   499mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.31 1 r UNASSIGNED
logstash-2016.10.31 4 p STARTED     619380 500.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.31 4 r UNASSIGNED
logstash-2016.10.23 4 p STARTED     428113 350.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.23 4 r UNASSIGNED
logstash-2016.10.23 0 p STARTED     428355   348mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.23 0 r UNASSIGNED
logstash-2016.10.23 3 p STARTED     428416   351mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.23 3 r UNASSIGNED
logstash-2016.10.23 1 p STARTED     427966 348.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.23 1 r UNASSIGNED
logstash-2016.10.23 2 p STARTED     428086 348.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.23 2 r UNASSIGNED
logstash-2016.10.22 2 p STARTED     445397 366.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.22 2 r UNASSIGNED
logstash-2016.10.22 0 p STARTED     446070 367.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.22 0 r UNASSIGNED
logstash-2016.10.22 3 p STARTED     445990 365.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.22 3 r UNASSIGNED
logstash-2016.10.22 1 p STARTED     445543   367mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.22 1 r UNASSIGNED
logstash-2016.10.22 4 p STARTED     445873 366.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.22 4 r UNASSIGNED
logstash-2016.10.25 2 p STARTED     681841 549.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.25 2 r UNASSIGNED
logstash-2016.10.25 0 p STARTED     682524 542.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.25 0 r UNASSIGNED
logstash-2016.10.25 3 p STARTED     682036 541.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.25 3 r UNASSIGNED
logstash-2016.10.25 1 p STARTED     681968 550.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.25 1 r UNASSIGNED
logstash-2016.10.25 4 p STARTED     682802 555.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.25 4 r UNASSIGNED
logstash-2016.10.24 4 p STARTED     634974   513mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.24 4 r UNASSIGNED
logstash-2016.10.24 0 p STARTED     634917 512.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.24 0 r UNASSIGNED
logstash-2016.10.24 3 p STARTED     634763 511.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.24 3 r UNASSIGNED
logstash-2016.10.24 1 p STARTED     635108 507.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.24 1 r UNASSIGNED
logstash-2016.10.24 2 p STARTED     635086 507.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.24 2 r UNASSIGNED
logstash-2016.10.27 4 p STARTED     738606 554.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.27 4 r UNASSIGNED
logstash-2016.10.27 0 p STARTED     737826 550.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.27 0 r UNASSIGNED
logstash-2016.10.27 3 p STARTED     738928 561.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.27 3 r UNASSIGNED
logstash-2016.10.27 1 p STARTED     737967 561.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.27 1 r UNASSIGNED
logstash-2016.10.27 2 p STARTED     738174 552.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.27 2 r UNASSIGNED
logstash-2016.10.26 4 p STARTED     668137 540.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.26 4 r UNASSIGNED
logstash-2016.10.26 0 p STARTED     668823 526.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.26 0 r UNASSIGNED
logstash-2016.10.26 3 p STARTED     668540 537.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.26 3 r UNASSIGNED
logstash-2016.10.26 1 p STARTED     668281 536.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.26 1 r UNASSIGNED
logstash-2016.10.26 2 p STARTED     668584 534.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.26 2 r UNASSIGNED
logstash-2016.10.29 2 p STARTED     489580 396.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.29 2 r UNASSIGNED
logstash-2016.10.29 0 p STARTED     489944 395.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.29 0 r UNASSIGNED
logstash-2016.10.29 3 p STARTED     490810 397.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.29 3 r UNASSIGNED
logstash-2016.10.29 1 p STARTED     489872 396.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.29 1 r UNASSIGNED
logstash-2016.10.29 4 p STARTED     490002 396.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.29 4 r UNASSIGNED
logstash-2016.10.28 4 p STARTED     639300 513.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.28 4 r UNASSIGNED
logstash-2016.10.28 0 p STARTED     639887 513.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.28 0 r UNASSIGNED
logstash-2016.10.28 3 p STARTED     640285 513.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.28 3 r UNASSIGNED
logstash-2016.10.28 1 p STARTED     639985 511.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.28 1 r UNASSIGNED
logstash-2016.10.28 2 p STARTED     639700 512.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.10.28 2 r UNASSIGNED
logstash-2016.11.11 2 p STARTED    3100821     1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.11 2 r UNASSIGNED
logstash-2016.11.11 0 p STARTED    3101154     1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.11 0 r UNASSIGNED
logstash-2016.11.11 3 p STARTED    3101064     1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.11 3 r UNASSIGNED
logstash-2016.11.11 1 p STARTED    3101013     1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.11 1 r UNASSIGNED
logstash-2016.11.11 4 p STARTED    3100164     1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.11 4 r UNASSIGNED
logstash-2016.11.10 2 p STARTED     800065 816.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.10 2 r UNASSIGNED
logstash-2016.11.10 0 p STARTED     799295 823.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.10 0 r UNASSIGNED
logstash-2016.11.10 3 p STARTED     799862 821.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.10 3 r UNASSIGNED
logstash-2016.11.10 1 p STARTED     799901 822.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.10 1 r UNASSIGNED
logstash-2016.11.10 4 p STARTED     799770 828.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
logstash-2016.11.10 4 r UNASSIGNED
kibana-int          4 p STARTED         31 216.2kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
kibana-int          4 r UNASSIGNED
kibana-int          0 p STARTED         33 337.2kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
kibana-int          0 r UNASSIGNED
kibana-int          3 p STARTED         30 324.1kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
kibana-int          3 r UNASSIGNED
kibana-int          1 p STARTED         19 170.4kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
kibana-int          1 r UNASSIGNED
kibana-int          2 p STARTED         24 256.9kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
kibana-int          2 r UNASSIGNED

Re: All shards failed for phase: [query_fetch]

Posted: Wed Nov 23, 2016 8:21 am
by _asp_

Code: Select all

curl "localhost:9200/_cluster/health?level=indices&pretty"

{
  "cluster_name" : "a94658e4-9cbe-4f72-8cc8-9a276f7853d1",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 281,
  "active_shards" : 281,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 281,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "indices" : {
    "nagioslogserver" : {
      "status" : "yellow",
      "number_of_shards" : 1,
      "number_of_replicas" : 1,
      "active_primary_shards" : 1,
      "active_shards" : 1,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 1
    },
    "logstash-2016.11.23" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.10" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "nagioslogserver_log" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.05" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.04" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.07" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.06" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.01" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.03" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.02" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.09" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.08" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.15" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.20" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.14" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.21" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.13" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.12" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.19" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.18" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.17" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.16" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.18" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.17" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.16" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.15" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.21" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.14" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.13" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.22" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.12" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.11" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.20" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.19" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.05" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.06" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.07" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.08" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.01" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.02" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.03" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.04" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.09" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.30" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.31" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.23" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.22" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.25" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.24" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.27" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.26" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.29" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.10.28" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.11" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.10" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "kibana-int" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    }
  }
}

Re: All shards failed for phase: [query_fetch]

Posted: Wed Nov 23, 2016 10:26 am
by mcapra
Hmm, did the log message and your execution of curl "localhost:9200/_cat/shards?v?" occur at roughly the same time?

This exception implies that the nagioslogserver index has no shards allocated:

Code: Select all

org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null
However, the output of your _cat/shards clearly shows a primary assigned to the nagioslogserver index:

Code: Select all

nagioslogserver     0 p STARTED        107 530.6kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87
It may have been a one-off issue during elasticsearch startup. Are you able to login and navigate the Nagios Log Server GUI? Does the Administration -> Command Subsystem show 5 System Jobs?

Re: All shards failed for phase: [query_fetch]

Posted: Mon Dec 05, 2016 4:02 am
by _asp_
Hi,

today we are facing the same problem again.

Accessing logservers webpage throws http 503.
restarting elasticsearch (logstash still stopped) doesn't help. 503 stays.

here is the log of elasticsearch after startup:

Code: Select all

[2016-12-05 09:43:27,781][INFO ][node                     ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] version[1.6.0], pid[37979], build[cdd3ac4/2015-06-09T13:36:34Z]
[2016-12-05 09:43:27,782][INFO ][node                     ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] initializing ...
[2016-12-05 09:43:27,793][INFO ][plugins                  ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] loaded [knapsack-1.5.2.0-f340ad1], sites []
[2016-12-05 09:43:27,831][INFO ][env                      ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] using [1] data paths, mounts [[/data (/dev/mapper/vg_data-lv_data)]], net usable_space [44.2gb], net total_space [196.6gb], types [ext4]
[2016-12-05 09:43:31,053][INFO ][node                     ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] initialized
[2016-12-05 09:43:31,053][INFO ][node                     ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] starting ...
[2016-12-05 09:43:31,175][INFO ][transport                ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/139.1.117.45:9300]}
[2016-12-05 09:43:31,183][INFO ][discovery                ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] a94658e4-9cbe-4f72-8cc8-9a276f7853d1/S1evn6GHQKyWlFoHrJNqww
[2016-12-05 09:43:34,207][INFO ][cluster.service          ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] new_master [f12d79cf-a699-42b8-b6c2-47d36407ac87][S1evn6GHQKyWlFoHrJNqww][logiprodcontrol][inet[/139.1.117.45:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2016-12-05 09:43:34,276][INFO ][http                     ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[localhost/127.0.0.1:9200]}
[2016-12-05 09:43:34,276][INFO ][node                     ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] started
[2016-12-05 09:43:34,414][INFO ][gateway                  ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] recovered [38] indices into cluster_state
[2016-12-05 09:43:36,488][DEBUG][action.search.type       ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] All shards failed for phase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null
	at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:160)
	at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:57)
	at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:47)
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:104)
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
	at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)
	at org.elasticsearch.client.FilterClient.execute(FilterClient.java:66)
	at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.execute(BaseRestHandler.java:92)
	at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338)
	at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:84)
	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:53)
	at org.elasticsearch.rest.RestController.executeHandler(RestController.java:225)
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:170)
	at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
	at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
	at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:327)
	at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
	at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
	at org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
	at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
	at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
	at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
	at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
	at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
[2016-12-05 09:43:36,499][DEBUG][action.search.type       ] [f12d79cf-a699-42b8-b6c2-47d36407ac87] All shards failed for phase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null
	at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:160)
	at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:57)
	at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:47)
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:104)
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
	at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)
	at org.elasticsearch.client.FilterClient.execute(FilterClient.java:66)
	at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.execute(BaseRestHandler.java:92)
	at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338)
	at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:84)
	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:53)
	at org.elasticsearch.rest.RestController.executeHandler(RestController.java:225)
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:170)
	at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
	at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
	at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:327)
	at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
	at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
	at org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
	at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
	at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
	at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
	at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
	at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
I requested cluster health:

Code: Select all

curl "localhost:9200/_cluster/health?level=indices&pretty"

{
  "cluster_name" : "a94658e4-9cbe-4f72-8cc8-9a276f7853d1",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 186,
  "active_shards" : 186,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 186,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "indices" : {
    "logstash-2016.11.05" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.06" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.07" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.08" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.01" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.02" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "nagioslogserver" : {
      "status" : "yellow",
      "number_of_shards" : 1,
      "number_of_replicas" : 1,
      "active_primary_shards" : 1,
      "active_shards" : 1,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 1
    },
    "logstash-2016.11.03" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.04" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.24" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.23" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.26" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.25" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.09" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.28" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.27" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.29" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "nagioslogserver_log" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.30" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.11" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.10" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.12.05" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.12.04" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.12.03" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.12.02" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.12.01" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.15" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.14" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.13" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.12" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.19" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.18" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.17" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.16" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.21" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.22" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "logstash-2016.11.20" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    "kibana-int" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    }
  }
}

Re: All shards failed for phase: [query_fetch]

Posted: Mon Dec 05, 2016 4:05 am
by _asp_
additionally I requested the shard overview some seconds later:

Code: Select all

logstash-2016.11.05 2 p STARTED     497391 392.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.05 2 r UNASSIGNED                                                                
logstash-2016.11.05 0 p STARTED     497375 391.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.05 0 r UNASSIGNED                                                                
logstash-2016.11.05 3 p STARTED     497058 384.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.05 3 r UNASSIGNED                                                                
logstash-2016.11.05 1 p STARTED     497598 393.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.05 1 r UNASSIGNED                                                                
logstash-2016.11.05 4 p STARTED     497207 393.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.05 4 r UNASSIGNED                                                                
logstash-2016.11.06 2 p STARTED     456409 358.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.06 2 r UNASSIGNED                                                                
logstash-2016.11.06 0 p STARTED     456394   360mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.06 0 r UNASSIGNED                                                                
logstash-2016.11.06 3 p STARTED     456506   358mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.06 3 r UNASSIGNED                                                                
logstash-2016.11.06 1 p STARTED     456541 359.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.06 1 r UNASSIGNED                                                                
logstash-2016.11.06 4 p STARTED     456113 358.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.06 4 r UNASSIGNED                                                                
logstash-2016.11.07 4 p STARTED     689901 557.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.07 4 r UNASSIGNED                                                                
logstash-2016.11.07 0 p STARTED     689458 565.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.07 0 r UNASSIGNED                                                                
logstash-2016.11.07 3 p STARTED     689861 560.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.07 3 r UNASSIGNED                                                                
logstash-2016.11.07 1 p STARTED     689828 559.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.07 1 r UNASSIGNED                                                                
logstash-2016.11.07 2 p STARTED     690574 565.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.07 2 r UNASSIGNED                                                                
logstash-2016.11.08 2 p STARTED     723005 588.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.08 2 r UNASSIGNED                                                                
logstash-2016.11.08 0 p STARTED     723225 592.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.08 0 r UNASSIGNED                                                                
logstash-2016.11.08 3 p STARTED     724208 588.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.08 3 r UNASSIGNED                                                                
logstash-2016.11.08 1 p STARTED     723162 584.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.08 1 r UNASSIGNED                                                                
logstash-2016.11.08 4 p STARTED     724440 584.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.08 4 r UNASSIGNED                                                                
logstash-2016.11.01 4 p STARTED     586842 477.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.01 4 r UNASSIGNED                                                                
logstash-2016.11.01 0 p STARTED     586477 475.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.01 0 r UNASSIGNED                                                                
logstash-2016.11.01 3 p STARTED     587219 473.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.01 3 r UNASSIGNED                                                                
logstash-2016.11.01 1 p STARTED     587276 475.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.01 1 r UNASSIGNED                                                                
logstash-2016.11.01 2 p STARTED     587197 475.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.01 2 r UNASSIGNED                                                                
logstash-2016.11.02 2 p STARTED     667486 535.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.02 2 r UNASSIGNED                                                                
logstash-2016.11.02 0 p STARTED     667600 534.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.02 0 r UNASSIGNED                                                                
logstash-2016.11.02 3 p STARTED     667894 535.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.02 3 r UNASSIGNED                                                                
logstash-2016.11.02 1 p STARTED     667717 534.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.02 1 r UNASSIGNED                                                                
logstash-2016.11.02 4 p STARTED     666992 529.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.02 4 r UNASSIGNED                                                                
nagioslogserver     0 p STARTED        109 251.4kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
nagioslogserver     0 r UNASSIGNED                                                                
logstash-2016.11.03 2 p STARTED     670906 547.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.03 2 r UNASSIGNED                                                                
logstash-2016.11.03 0 p STARTED     670556 542.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.03 0 r UNASSIGNED                                                                
logstash-2016.11.03 3 p STARTED     670239 540.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.03 3 r UNASSIGNED                                                                
logstash-2016.11.03 1 p STARTED     670376 545.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.03 1 r UNASSIGNED                                                                
logstash-2016.11.03 4 p STARTED     670384   535mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.03 4 r UNASSIGNED                                                                
logstash-2016.11.04 4 p STARTED     658493 526.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.04 4 r UNASSIGNED                                                                
logstash-2016.11.04 0 p STARTED     658437 534.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.04 0 r UNASSIGNED                                                                
logstash-2016.11.04 3 p STARTED     658137 532.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.04 3 r UNASSIGNED                                                                
logstash-2016.11.04 1 p STARTED     658598 526.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.04 1 r UNASSIGNED                                                                
logstash-2016.11.04 2 p STARTED     658339 531.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.04 2 r UNASSIGNED                                                                
logstash-2016.11.24 4 p STARTED    1691127   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.24 4 r UNASSIGNED                                                                
logstash-2016.11.24 0 p STARTED    1691339   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.24 0 r UNASSIGNED                                                                
logstash-2016.11.24 3 p STARTED    1690766   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.24 3 r UNASSIGNED                                                                
logstash-2016.11.24 1 p STARTED    1691555   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.24 1 r UNASSIGNED                                                                
logstash-2016.11.24 2 p STARTED    1690578   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.24 2 r UNASSIGNED                                                                
logstash-2016.11.23 2 p STARTED    1787935   1.7gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.23 2 r UNASSIGNED                                                                
logstash-2016.11.23 0 p STARTED    1787476   1.7gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.23 0 r UNASSIGNED                                                                
logstash-2016.11.23 3 p STARTED    1787775   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.23 3 r UNASSIGNED                                                                
logstash-2016.11.23 1 p STARTED    1787272   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.23 1 r UNASSIGNED                                                                
logstash-2016.11.23 4 p STARTED    1787833   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.23 4 r UNASSIGNED                                                                
logstash-2016.11.26 2 p STARTED     738890 690.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.26 2 r UNASSIGNED                                                                
logstash-2016.11.26 0 p STARTED     738524 694.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.26 0 r UNASSIGNED                                                                
logstash-2016.11.26 3 p STARTED     739053 698.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.26 3 r UNASSIGNED                                                                
logstash-2016.11.26 1 p STARTED     738498 701.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.26 1 r UNASSIGNED                                                                
logstash-2016.11.26 4 p STARTED     738843 690.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.26 4 r UNASSIGNED                                                                
logstash-2016.11.25 2 p STARTED    1627784   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.25 2 r UNASSIGNED                                                                
logstash-2016.11.25 0 p STARTED    1626358   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.25 0 r UNASSIGNED                                                                
logstash-2016.11.25 3 p STARTED    1627255   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.25 3 r UNASSIGNED                                                                
logstash-2016.11.25 1 p STARTED    1627775   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.25 1 r UNASSIGNED                                                                
logstash-2016.11.25 4 p STARTED    1628312   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.25 4 r UNASSIGNED                                                                
logstash-2016.11.28 2 p STARTED    1653158   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.28 2 r UNASSIGNED                                                                
logstash-2016.11.28 0 p STARTED    1653183   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.28 0 r UNASSIGNED                                                                
logstash-2016.11.28 3 p STARTED    1652715   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.28 3 r UNASSIGNED                                                                
logstash-2016.11.28 1 p STARTED    1652787   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.28 1 r UNASSIGNED                                                                
logstash-2016.11.28 4 p STARTED    1653556   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.28 4 r UNASSIGNED                                                                
logstash-2016.11.09 4 p STARTED     714942 584.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.09 4 r UNASSIGNED                                                                
logstash-2016.11.09 0 p STARTED     714754 582.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.09 0 r UNASSIGNED                                                                
logstash-2016.11.09 3 p STARTED     714362   582mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.09 3 r UNASSIGNED                                                                
logstash-2016.11.09 1 p STARTED     715058 577.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.09 1 r UNASSIGNED                                                                
logstash-2016.11.09 2 p STARTED     714291 576.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.09 2 r UNASSIGNED                                                                
logstash-2016.11.27 4 p STARTED     570577 517.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.27 4 r UNASSIGNED                                                                
logstash-2016.11.27 0 p STARTED     570077   517mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.27 0 r UNASSIGNED                                                                
logstash-2016.11.27 3 p STARTED     570118 513.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.27 3 r UNASSIGNED                                                                
logstash-2016.11.27 1 p STARTED     570149 512.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.27 1 r UNASSIGNED                                                                
logstash-2016.11.27 2 p STARTED     570158 511.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.27 2 r UNASSIGNED                                                                
logstash-2016.11.29 2 p STARTED    1678069   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.29 2 r UNASSIGNED                                                                
logstash-2016.11.29 0 p STARTED    1677995   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.29 0 r UNASSIGNED                                                                
logstash-2016.11.29 3 p STARTED    1677980   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.29 3 r UNASSIGNED                                                                
logstash-2016.11.29 1 p STARTED    1678367   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.29 1 r UNASSIGNED                                                                
logstash-2016.11.29 4 p STARTED    1677731   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.29 4 r UNASSIGNED                                                                
nagioslogserver_log 4 p STARTED     252610  26.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
nagioslogserver_log 4 r UNASSIGNED                                                                
nagioslogserver_log 0 p STARTED     252229  26.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
nagioslogserver_log 0 r UNASSIGNED                                                                
nagioslogserver_log 3 p STARTED     253396  26.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
nagioslogserver_log 3 r UNASSIGNED                                                                
nagioslogserver_log 1 p STARTED     251903  26.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
nagioslogserver_log 1 r UNASSIGNED                                                                
nagioslogserver_log 2 p STARTED     253038  26.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
nagioslogserver_log 2 r UNASSIGNED                                                                
logstash-2016.11.30 4 p STARTED    1766600   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.30 4 r UNASSIGNED                                                                
logstash-2016.11.30 0 p STARTED    1766774   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.30 0 r UNASSIGNED                                                                
logstash-2016.11.30 3 p STARTED    1767441   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.30 3 r UNASSIGNED                                                                
logstash-2016.11.30 1 p STARTED    1767340   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.30 1 r UNASSIGNED                                                                
logstash-2016.11.30 2 p STARTED    1767881   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.30 2 r UNASSIGNED                                                                
logstash-2016.11.11 4 p STARTED    3100164     1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.11 4 r UNASSIGNED                                                                
logstash-2016.11.11 0 p STARTED    3101154     1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.11 0 r UNASSIGNED                                                                
logstash-2016.11.11 3 p STARTED    3101064     1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.11 3 r UNASSIGNED                                                                
logstash-2016.11.11 1 p STARTED    3101013     1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.11 1 r UNASSIGNED                                                                
logstash-2016.11.11 2 p STARTED    3100821     1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.11 2 r UNASSIGNED                                                                
logstash-2016.11.10 2 p STARTED     800065 816.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.10 2 r UNASSIGNED                                                                
logstash-2016.11.10 0 p STARTED     799295 823.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.10 0 r UNASSIGNED                                                                
logstash-2016.11.10 3 p STARTED     799862 821.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.10 3 r UNASSIGNED                                                                
logstash-2016.11.10 1 p STARTED     799901 822.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.10 1 r UNASSIGNED                                                                
logstash-2016.11.10 4 p STARTED     799770 828.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.10 4 r UNASSIGNED                                                                
logstash-2016.12.05 2 p STARTED     648238 523.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.05 2 r UNASSIGNED                                                                
logstash-2016.12.05 0 p STARTED     648449 522.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.05 0 r UNASSIGNED                                                                
logstash-2016.12.05 3 p STARTED     648657 522.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.05 3 r UNASSIGNED                                                                
logstash-2016.12.05 1 p STARTED     648327 523.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.05 1 r UNASSIGNED                                                                
logstash-2016.12.05 4 p STARTED     648925   524mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.05 4 r UNASSIGNED                                                                
logstash-2016.12.04 2 p STARTED     397376 146.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.04 2 r UNASSIGNED                                                                
logstash-2016.12.04 0 p STARTED     396524 149.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.04 0 r UNASSIGNED                                                                
logstash-2016.12.04 3 p STARTED     396668 147.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.04 3 r UNASSIGNED                                                                
logstash-2016.12.04 1 p STARTED     396626   148mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.04 1 r UNASSIGNED                                                                
logstash-2016.12.04 4 p STARTED     397058 148.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.04 4 r UNASSIGNED                                                                
logstash-2016.12.03 2 p STARTED     766407 687.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.03 2 r UNASSIGNED                                                                
logstash-2016.12.03 0 p STARTED     766247 688.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.03 0 r UNASSIGNED                                                                
logstash-2016.12.03 3 p STARTED     765946 680.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.03 3 r UNASSIGNED                                                                
logstash-2016.12.03 1 p STARTED     765964 688.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.03 1 r UNASSIGNED                                                                
logstash-2016.12.03 4 p STARTED     765999 686.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.03 4 r UNASSIGNED                                                                
logstash-2016.12.02 2 p STARTED    1649969   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.02 2 r UNASSIGNED                                                                
logstash-2016.12.02 0 p STARTED    1650490   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.02 0 r UNASSIGNED                                                                
logstash-2016.12.02 3 p STARTED    1650813   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.02 3 r UNASSIGNED                                                                
logstash-2016.12.02 1 p STARTED    1651074   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.02 1 r UNASSIGNED                                                                
logstash-2016.12.02 4 p STARTED    1651122   1.5gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.02 4 r UNASSIGNED                                                                
logstash-2016.12.01 4 p STARTED    1772610   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.01 4 r UNASSIGNED                                                                
logstash-2016.12.01 0 p STARTED    1773079   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.01 0 r UNASSIGNED                                                                
logstash-2016.12.01 3 p STARTED    1772146   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.01 3 r UNASSIGNED                                                                
logstash-2016.12.01 1 p STARTED    1772329   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.01 1 r UNASSIGNED                                                                
logstash-2016.12.01 2 p STARTED    1772432   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.12.01 2 r UNASSIGNED                                                                
logstash-2016.11.15 2 p STARTED     490657 410.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.15 2 r UNASSIGNED                                                                
logstash-2016.11.15 0 p STARTED     490576 411.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.15 0 r UNASSIGNED                                                                
logstash-2016.11.15 3 p STARTED     490219 411.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.15 3 r UNASSIGNED                                                                
logstash-2016.11.15 1 p STARTED     490620 410.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.15 1 r UNASSIGNED                                                                
logstash-2016.11.15 4 p STARTED     490626   412mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.15 4 r UNASSIGNED                                                                
logstash-2016.11.14 4 p STARTED     522255 412.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.14 4 r UNASSIGNED                                                                
logstash-2016.11.14 0 p STARTED     522341   411mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.14 0 r UNASSIGNED                                                                
logstash-2016.11.14 3 p STARTED     522879 410.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.14 3 r UNASSIGNED                                                                
logstash-2016.11.14 1 p STARTED     522742 411.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.14 1 r UNASSIGNED                                                                
logstash-2016.11.14 2 p STARTED     522863 412.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.14 2 r UNASSIGNED                                                                
logstash-2016.11.13 4 p STARTED     304017 231.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.13 4 r UNASSIGNED                                                                
logstash-2016.11.13 0 p STARTED     304019 231.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.13 0 r UNASSIGNED                                                                
logstash-2016.11.13 3 p STARTED     303884 228.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.13 3 r UNASSIGNED                                                                
logstash-2016.11.13 1 p STARTED     303771 231.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.13 1 r UNASSIGNED                                                                
logstash-2016.11.13 2 p STARTED     303659 230.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.13 2 r UNASSIGNED                                                                
logstash-2016.11.12 2 p STARTED     610887 374.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.12 2 r UNASSIGNED                                                                
logstash-2016.11.12 0 p STARTED     610917 375.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.12 0 r UNASSIGNED                                                                
logstash-2016.11.12 3 p STARTED     610498 374.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.12 3 r UNASSIGNED                                                                
logstash-2016.11.12 1 p STARTED     611000 377.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.12 1 r UNASSIGNED                                                                
logstash-2016.11.12 4 p STARTED     611076 375.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.12 4 r UNASSIGNED                                                                
logstash-2016.11.19 4 p STARTED     506590 357.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.19 4 r UNASSIGNED                                                                
logstash-2016.11.19 0 p STARTED     506485 356.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.19 0 r UNASSIGNED                                                                
logstash-2016.11.19 3 p STARTED     506379 356.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.19 3 r UNASSIGNED                                                                
logstash-2016.11.19 1 p STARTED     506510 355.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.19 1 r UNASSIGNED                                                                
logstash-2016.11.19 2 p STARTED     506809 357.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.19 2 r UNASSIGNED                                                                
logstash-2016.11.18 4 p STARTED     984486 567.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.18 4 r UNASSIGNED                                                                
logstash-2016.11.18 0 p STARTED     984410 580.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.18 0 r UNASSIGNED                                                                
logstash-2016.11.18 3 p STARTED     983546 572.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.18 3 r UNASSIGNED                                                                
logstash-2016.11.18 1 p STARTED     984117 575.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.18 1 r UNASSIGNED                                                                
logstash-2016.11.18 2 p STARTED     984280 573.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.18 2 r UNASSIGNED                                                                
logstash-2016.11.17 2 p STARTED     665061 489.1mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.17 2 r UNASSIGNED                                                                
logstash-2016.11.17 0 p STARTED     664247 484.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.17 0 r UNASSIGNED                                                                
logstash-2016.11.17 3 p STARTED     665167 488.2mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.17 3 r UNASSIGNED                                                                
logstash-2016.11.17 1 p STARTED     664654 484.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.17 1 r UNASSIGNED                                                                
logstash-2016.11.17 4 p STARTED     665129 488.6mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.17 4 r UNASSIGNED                                                                
logstash-2016.11.16 4 p STARTED     496628 410.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.16 4 r UNASSIGNED                                                                
logstash-2016.11.16 0 p STARTED     496402 408.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.16 0 r UNASSIGNED                                                                
logstash-2016.11.16 3 p STARTED     496509 410.7mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.16 3 r UNASSIGNED                                                                
logstash-2016.11.16 1 p STARTED     496662 409.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.16 1 r UNASSIGNED                                                                
logstash-2016.11.16 2 p STARTED     496834 409.9mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.16 2 r UNASSIGNED                                                                
logstash-2016.11.21 2 p STARTED    1331313   1.1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.21 2 r UNASSIGNED                                                                
logstash-2016.11.21 0 p STARTED    1331627   1.1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.21 0 r UNASSIGNED                                                                
logstash-2016.11.21 3 p STARTED    1331269   1.1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.21 3 r UNASSIGNED                                                                
logstash-2016.11.21 1 p STARTED    1331292   1.1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.21 1 r UNASSIGNED                                                                
logstash-2016.11.21 4 p STARTED    1331608   1.1gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.21 4 r UNASSIGNED                                                                
logstash-2016.11.22 2 p STARTED    1700772   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.22 2 r UNASSIGNED                                                                
logstash-2016.11.22 0 p STARTED    1701135   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.22 0 r UNASSIGNED                                                                
logstash-2016.11.22 3 p STARTED    1701586   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.22 3 r UNASSIGNED                                                                
logstash-2016.11.22 1 p STARTED    1701583   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.22 1 r UNASSIGNED                                                                
logstash-2016.11.22 4 p STARTED    1701021   1.6gb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.22 4 r UNASSIGNED                                                                
logstash-2016.11.20 2 p STARTED     403048 301.3mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.20 2 r UNASSIGNED                                                                
logstash-2016.11.20 0 p STARTED     402788 300.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.20 0 r UNASSIGNED                                                                
logstash-2016.11.20 3 p STARTED     403309 303.5mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.20 3 r UNASSIGNED                                                                
logstash-2016.11.20 1 p STARTED     403070 299.8mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.20 1 r UNASSIGNED                                                                
logstash-2016.11.20 4 p STARTED     403134 301.4mb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
logstash-2016.11.20 4 r UNASSIGNED                                                                
kibana-int          2 p STARTED         27   234kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
kibana-int          2 r UNASSIGNED                                                                
kibana-int          0 p STARTED         33 426.5kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
kibana-int          0 r UNASSIGNED                                                                
kibana-int          3 p STARTED         31 334.4kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
kibana-int          3 r UNASSIGNED                                                                
kibana-int          1 p STARTED         20 194.7kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
kibana-int          1 r UNASSIGNED                                                                
kibana-int          4 p STARTED         33 241.5kb 127.0.0.1 f12d79cf-a699-42b8-b6c2-47d36407ac87 
kibana-int          4 r UNASSIGNED
CPU load is very low. Last time we faced the issue CPU for elasticsearch was very high during startup and it just takes very long for starting up. That differs this time.

Code: Select all

top - 10:04:19 up 21 days, 22:51,  3 users,  load average: 0.00, 0.03, 0.66
Tasks: 252 total,   1 running, 251 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.4%us,  0.5%sy,  0.0%ni, 98.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  20462820k total, 17407280k used,  3055540k free,   230472k buffers
Swap:  8388320k total,    59736k used,  8328584k free,  5501832k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
37979 nagios    20   0 23.6g  10g 135m S 14.3 53.6   1:49.57 java
 1913 root      20   0  246m 9836  844 S  0.3  0.0   2:12.27 rsyslogd
    1 root      20   0 19360 1312 1096 S  0.0  0.0   0:01.75 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.05 kthreadd
    3 root      RT   0     0    0    0 S  0.0  0.0   0:11.07 migration/0
    4 root      20   0     0    0    0 S  0.0  0.0   0:05.54 ksoftirqd/0
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0
    6 root      RT   0     0    0    0 S  0.0  0.0   1:09.98 watchdog/0
    7 root      RT   0     0    0    0 S  0.0  0.0   0:12.24 migration/1
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/1
    9 root      20   0     0    0    0 S  0.0  0.0   0:08.23 ksoftirqd/1
   10 root      RT   0     0    0    0 S  0.0  0.0   1:09.06 watchdog/1
   11 root      RT   0     0    0    0 S  0.0  0.0   0:10.40 migration/2
   12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/2
   13 root      20   0     0    0    0 S  0.0  0.0   0:05.61 ksoftirqd/2
   14 root      RT   0     0    0    0 S  0.0  0.0   1:08.63 watchdog/2
   15 root      RT   0     0    0    0 S  0.0  0.0   0:08.92 migration/3
   16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/3
   17 root      20   0     0    0    0 S  0.0  0.0   0:05.46 ksoftirqd/3
   18 root      RT   0     0    0    0 S  0.0  0.0   1:08.34 watchdog/3
   19 root      RT   0     0    0    0 S  0.0  0.0   0:09.88 migration/4
   20 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/4
   21 root      20   0     0    0    0 S  0.0  0.0   0:05.33 ksoftirqd/4
   22 root      RT   0     0    0    0 S  0.0  0.0   1:14.91 watchdog/4
   23 root      RT   0     0    0    0 S  0.0  0.0   0:10.38 migration/5
   24 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/5
   25 root      20   0     0    0    0 S  0.0  0.0   0:05.22 ksoftirqd/5
   26 root      RT   0     0    0    0 S  0.0  0.0   1:12.14 watchdog/5
   27 root      RT   0     0    0    0 S  0.0  0.0   0:09.00 migration/6
   28 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/6
   29 root      20   0     0    0    0 S  0.0  0.0   0:05.18 ksoftirqd/6
   30 root      RT   0     0    0    0 S  0.0  0.0   1:09.63 watchdog/6
   31 root      RT   0     0    0    0 S  0.0  0.0   0:09.11 migration/7
   32 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/7
   33 root      20   0     0    0    0 S  0.0  0.0   0:04.89 ksoftirqd/7
   34 root      RT   0     0    0    0 S  0.0  0.0   1:11.66 watchdog/7
currently we are unable to use / startup logserver.

Re: All shards failed for phase: [query_fetch]

Posted: Mon Dec 05, 2016 4:14 am
by _asp_
restarting httpd does not help anything.

Re: All shards failed for phase: [query_fetch]

Posted: Mon Dec 05, 2016 4:23 am
by _asp_
seems to be a frontend problem. after restarting logstash again, I can see high cpu usage on logstash and some cpu on elasticsearch

Code: Select all

top - 10:19:55 up 21 days, 23:07,  3 users,  load average: 1.55, 2.75, 1.69
Tasks: 251 total,   1 running, 250 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.8%us,  0.6%sy, 62.7%ni, 34.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  20462820k total, 18964672k used,  1498148k free,   232148k buffers
Swap:  8388320k total,    59592k used,  8328728k free,  6308060k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
44727 nagios    39  19 5321m 762m  14m S 689.4  3.8  29:31.08 java
37979 nagios    20   0 23.6g  10g 135m S 22.3 53.6   9:07.18 java
 2098 nxlog     20   0  266m 1188  896 S  1.0  0.0  84:55.40 nxlog
 6631 root      20   0  195m 1268  784 S  0.3  0.0   8:39.86 snmpd
    1 root      20   0 19360 1312 1096 S  0.0  0.0   0:01.75 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.05 kthreadd
    3 root      RT   0     0    0    0 S  0.0  0.0   0:11.07 migration/0
    4 root      20   0     0    0    0 S  0.0  0.0   0:05.54 ksoftirqd/0
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0
    6 root      RT   0     0    0    0 S  0.0  0.0   1:10.01 watchdog/0
    7 root      RT   0     0    0    0 S  0.0  0.0   0:12.25 migration/1
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/1
    9 root      20   0     0    0    0 S  0.0  0.0   0:08.24 ksoftirqd/1
   10 root      RT   0     0    0    0 S  0.0  0.0   1:09.07 watchdog/1
   11 root      RT   0     0    0    0 S  0.0  0.0   0:10.40 migration/2
   12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/2
   13 root      20   0     0    0    0 S  0.0  0.0   0:05.62 ksoftirqd/2
   14 root      RT   0     0    0    0 S  0.0  0.0   1:08.65 watchdog/2
   15 root      RT   0     0    0    0 S  0.0  0.0   0:08.92 migration/3
   16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/3
   17 root      20   0     0    0    0 S  0.0  0.0   0:05.53 ksoftirqd/3
   18 root      RT   0     0    0    0 S  0.0  0.0   1:08.38 watchdog/3
   19 root      RT   0     0    0    0 S  0.0  0.0   0:09.89 migration/4
   20 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/4
   21 root      20   0     0    0    0 S  0.0  0.0   0:05.41 ksoftirqd/4
   22 root      RT   0     0    0    0 S  0.0  0.0   1:15.46 watchdog/4
   23 root      RT   0     0    0    0 S  0.0  0.0   0:10.38 migration/5
   24 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/5
   25 root      20   0     0    0    0 S  0.0  0.0   0:05.22 ksoftirqd/5
   26 root      RT   0     0    0    0 S  0.0  0.0   1:12.31 watchdog/5
   27 root      RT   0     0    0    0 S  0.0  0.0   0:09.00 migration/6
   28 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/6
   29 root      20   0     0    0    0 S  0.0  0.0   0:05.19 ksoftirqd/6
   30 root      RT   0     0    0    0 S  0.0  0.0   1:09.74 watchdog/6
   31 root      RT   0     0    0    0 S  0.0  0.0   0:09.11 migration/7
but how can I restart the kibana service? I dont find a service in init.d called kibana* or nagios* or logser*

Re: All shards failed for phase: [query_fetch]

Posted: Mon Dec 05, 2016 4:46 am
by _asp_
after rebooting the whole server logserver webfrontent is available again.

But that is not a solution.
1. What can we do to identify the root cause?
- during the weekend log processing stopped suddenly
- after restarting logstash and elasticsearch this morning, the log processing started to work again, but login page was not available.
- why did it crash?
2. How can I restart all components of the web interface without rebooting the whole server?

Re: All shards failed for phase: [query_fetch]

Posted: Mon Dec 05, 2016 10:26 am
by mcapra
_asp_ wrote:but how can I restart the kibana service? I dont find a service in init.d called kibana* or nagios* or logser*
Kibana does not have a dedicated service in this case. Practically everything relating to the front-end runs under httpd.
_asp_ wrote: What can we do to identify the root cause?
- during the weekend log processing stopped suddenly
That's either logstash or elasticsearch failing most likely. Viewing the logs of both applications (/var/log/logstash/logstash.log and /var/log/elasticsearch/*.log) can usually help determine which one is causing problems.
_asp_ wrote:- after restarting logstash and elasticsearch this morning, the log processing started to work again, but login page was not available.
- why did it crash?
Something still appears to be causing failures within the nagioslogserver index:

Code: Select all

org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null
Which may or may not be indicative of deeper problems within elasticsearch. It is certainly odd when one specific index continues to fail after several restarts to the elasticsearch service.
_asp_ wrote:2. How can I restart all components of the web interface without rebooting the whole server?
The only major components used by the web interface are the httpd and elasticsearch services.

In the interest of time, it might be best to move this to an email ticket if possible and schedule a remote assistance session.
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited