Node timeout and cannot join cluster

CFT6Server · Post by **CFT6Server** » Wed Dec 30, 2015 3:50 pm

I am seeing the following message and I can't get the node to join back to the cluster.....

[2015-12-30 12:47:47,850][DEBUG][action.bulk              ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]

This happened after this main disk ran out of space. The space is freed up now but doesn't seem to be able to get ES to run.

jolson · Post by **jolson** » Wed Dec 30, 2015 4:11 pm

Is elasticsearch capable of shutting down cleanly and without errors (besides the occasional java stack trace)?

Code: Select all

service elasticsearch stop
service elasticsearch status
tail /var/log/elasticsearch/*.log

If so, that's good news. How long has it been since you started elasticsearch up? It may take a second to re-join the cluster.

Check on the following on the failed node while elasticsearch is running:

Code: Select all

curl -XGET 'localhost:9200/_cat/recovery?v'
curl 'localhost:9200/_cat/plugins?v'
curl 'localhost:9200/_cluster/health?level=indices&pretty'
cat /usr/local/nagioslogserver/var/cluster_*
curl 'localhost:9200/_cat/nodes?v'
curl 'localhost:9200/_cat/master?v'

Let me know what you find out. Thanks!

CFT6Server · Post by **CFT6Server** » Wed Dec 30, 2015 5:25 pm

So ES stops cleanly

Code: Select all

[2015-12-30 14:13:23,582][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] stopping ...
[2015-12-30 14:13:24,740][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] stopped
[2015-12-30 14:13:24,741][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] closing ...
[2015-12-30 14:13:24,756][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] closed

I had to reboot the node, so I was doing a clean shutdown and reboot, but afterwards started having this. The node has been up for over an hour and doesn't want to join. No network issues and it can ping other nodes.
(Previously it had stopped due to drive space issue on /)

on Startup...

Code: Select all

# service elasticsearch start
Starting elasticsearch:                                    [  OK  ]
[root@kdcnagls1n3 ~]# tail -f /var/log/elasticsearch/80e9022e-f73f-429e-8927-f23d0d88dfd2.log
[2015-12-30 14:13:24,741][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] closing ...
[2015-12-30 14:13:24,756][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] closed
[2015-12-30 14:15:44,510][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] version[1.6.0], pid[855                                                                                      2], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-12-30 14:15:44,511][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] initializing ...
[2015-12-30 14:15:44,524][INFO ][plugins                  ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] loaded [knapsack-1.5.2.                                                                                      0-f340ad1], sites []
[2015-12-30 14:15:44,587][INFO ][env                      ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] using [1] data paths, m                                                                                      ounts [[/nfs/logdata (10.242.145.237:/vol/v_kdcnagiosnfs1_kdcnagls1n3_logs)]], net usable_space [1.4tb], net total_space [                                                                                      2.4tb], types [nfs]
[2015-12-30 14:15:49,375][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] initialized
[2015-12-30 14:15:49,375][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] starting ...
[2015-12-30 14:15:49,501][INFO ][transport                ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] bound_address {inet[/0:                                                                                      0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.242.102.109:9300]}
[2015-12-30 14:15:49,511][INFO ][discovery                ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] 80e9022e-f73f-429e-8927                                                                                      -f23d0d88dfd2/JEOqkDrhRwSBJG84nTHzvA


[2015-12-30 14:15:59,538][INFO ][cluster.service          ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] new_master [e63648a3-d9                                                                                      12-4f5d-a867-1b99282a5e7c][JEOqkDrhRwSBJG84nTHzvA][kdcnagls1n3.bchydro.bc.ca][inet[/10.242.102.109:9300]]{max_local_storag                                                                                      e_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-12-30 14:15:59,935][INFO ][http                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] bound_address {inet[/12                                                                                      7.0.0.1:9200]}, publish_address {inet[localhost/127.0.0.1:9200]}
[2015-12-30 14:15:59,936][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] started
[2015-12-30 14:16:00,176][INFO ][discovery.zen            ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] updating discovery.zen.                                                                                      minimum_master_nodes from [-1] to [1]
[2015-12-30 14:16:00,630][INFO ][gateway                  ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] recovered [327] indices                                                                                       into cluster_state
[2015-12-30 14:16:01,641][DEBUG][action.search.type       ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] All shards failed for p                                                                                      hase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null

        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.                                                                                      java:160)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchA                                                                                      ction.java:57)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchA                                                                                      ction.java:47)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:104)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)
        at org.elasticsearch.client.FilterClient.execute(FilterClient.java:66)
        at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.execute(BaseRestHandler.java:92)
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338)
        at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:84)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:53)
        at org.elasticsearch.rest.RestController.executeHandler(RestController.java:225)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:170)
        at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
        at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
        at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:327)
        at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
        at org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:                                                                                      145)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:10                                                                                      8)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:                                                                                      459)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
[2015-12-30 14:16:01,641][DEBUG][action.search.type       ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] All shards failed for p                                                                                      hase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.                                                                                      java:160)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchA                                                                                      ction.java:57)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchA                                                                                      ction.java:47)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:104)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)
        at org.elasticsearch.client.FilterClient.execute(FilterClient.java:66)
        at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.execute(BaseRestHandler.java:92)
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338)
        at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:84)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:53)
        at org.elasticsearch.rest.RestController.executeHandler(RestController.java:225)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:170)
        at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
        at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
        at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:327)
        at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
        at org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:                                                                                      145)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:10                                                                                      8)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:                                                                                      459)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
[2015-12-30 14:16:01,661][DEBUG][action.search.type       ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] All shards failed for p                                                                                      hase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.                                                                                      java:160)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchA                                                                                      ction.java:57)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchA                                                                                      ction.java:47)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:104)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)
        at org.elasticsearch.client.FilterClient.execute(FilterClient.java:66)
        at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.execute(BaseRestHandler.java:92)
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338)
        at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:84)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:53)
        at org.elasticsearch.rest.RestController.executeHandler(RestController.java:225)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:170)
        at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
        at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
        at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:327)
        at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
        at org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:                                                                                      145)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:10                                                                                      8)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:                                                                                      459)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

So it think itself is the new master?

# curl 'localhost:9200/_cat/nodes?v'
host ip heap.percent ram.percent load node.role master name
kdcnagls1n3a 10.242.102.109 16 71 0.07 d * e63648a3-d912-4f5d-a867-1b99282a5e7c

# curl 'localhost:9200/_cat/master?v'
id host ip node
JEOqkDrhRwSBJG84nTHzvA kdcnagls1n3 10.242.102.109 e63648a3-d912-4f5d-a867-1b99282a5e7c

Other nodes...
# curl 'localhost:9200/_cat/nodes?v'
host ip heap.percent ram.percent load node.role master name
kdcnagls2n1 10.242.144.212 70 72 0.84 d m 9a92d6ef-d554-49d8-9191-dcf886382926
kdcnagls2n2 10.242.144.213 66 72 0.73 d * c424515a-16b3-43f9-866e-19daedef8a63
kdcnagls2n3 10.242.144.214 72 76 0.81 d m 2db4ce89-4c01-4a30-9bc8-66e987b7d613
kdcnagls1n1 10.242.102.107 48 78 4.51 d m 4521585a-88af-47c9-81e5-c4d13cffb148
kdcnagls1n2 10.242.102.108 66 74 0.30 d m 30ab2b2c-439f-4bcc-977d-7c0e9a90f3a5

# curl 'localhost:9200/_cat/master?v'
id host ip node
OcC8_UCIR5qjNEIYYTIzfA kdcnagls2n2 10.242.144.213 c424515a-16b3-43f9-866e-19daedef8a63

CFT6Server · Post by **CFT6Server** » Wed Dec 30, 2015 5:45 pm

ok think I got it fixed. I had to take the current real master offline. The minute I've done that, the one that won't join become the master and all the other nodes joined. Did a rolling restart just in case and now waiting for the shards to 'recover' and reinitialize. At least I am back to 6 nodes and yellow status.... will keep an eye on it.

bwallace · Post by **bwallace** » Mon Jan 04, 2016 10:28 am

Any changes since your previous post? Let us know of the status whenever you have a moment, thanks.

CFT6Server · Post by **CFT6Server** » Mon Jan 04, 2016 4:34 pm

The nodes are back to normal and issues is resolved. Thanks.

rkennedy · Post by **rkennedy** » Mon Jan 04, 2016 4:50 pm

Thanks for the update, and glad to see things are working! I'll go ahead and close this thread out, but feel free to open a new one if you ever need assistance!

Nagios Support Forum

Node timeout and cannot join cluster

Node timeout and cannot join cluster

Re: Node timeout and cannot join cluster

Re: Node timeout and cannot join cluster

Re: Node timeout and cannot join cluster

Re: Node timeout and cannot join cluster

Re: Node timeout and cannot join cluster

Re: Node timeout and cannot join cluster