Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Node timeout and cannot join cluster

https://support.nagios.com/forum/viewtopic.php?t=36392

Page 1 of 1

Node timeout and cannot join cluster

Posted: Wed Dec 30, 2015 3:50 pm
by CFT6Server
I am seeing the following message and I can't get the node to join back to the cluster.....

Code: Select all

[2015-12-30 12:47:47,850][DEBUG][action.bulk              ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
This happened after this main disk ran out of space. The space is freed up now but doesn't seem to be able to get ES to run.

Re: Node timeout and cannot join cluster

Posted: Wed Dec 30, 2015 4:11 pm
by jolson
Is elasticsearch capable of shutting down cleanly and without errors (besides the occasional java stack trace)?

Code: Select all

service elasticsearch stop
service elasticsearch status
tail /var/log/elasticsearch/*.log
If so, that's good news. How long has it been since you started elasticsearch up? It may take a second to re-join the cluster.

Check on the following on the failed node while elasticsearch is running:

Code: Select all

curl -XGET 'localhost:9200/_cat/recovery?v'
curl 'localhost:9200/_cat/plugins?v'
curl 'localhost:9200/_cluster/health?level=indices&pretty'
cat /usr/local/nagioslogserver/var/cluster_*
curl 'localhost:9200/_cat/nodes?v'
curl 'localhost:9200/_cat/master?v'
Let me know what you find out. Thanks!

Re: Node timeout and cannot join cluster

Posted: Wed Dec 30, 2015 5:25 pm
by CFT6Server
So ES stops cleanly

Code: Select all

[2015-12-30 14:13:23,582][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] stopping ...
[2015-12-30 14:13:24,740][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] stopped
[2015-12-30 14:13:24,741][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] closing ...
[2015-12-30 14:13:24,756][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] closed
I had to reboot the node, so I was doing a clean shutdown and reboot, but afterwards started having this. The node has been up for over an hour and doesn't want to join. No network issues and it can ping other nodes.
(Previously it had stopped due to drive space issue on /)

on Startup...

Code: Select all

# service elasticsearch start
Starting elasticsearch:                                    [  OK  ]
[root@kdcnagls1n3 ~]# tail -f /var/log/elasticsearch/80e9022e-f73f-429e-8927-f23d0d88dfd2.log
[2015-12-30 14:13:24,741][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] closing ...
[2015-12-30 14:13:24,756][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] closed
[2015-12-30 14:15:44,510][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] version[1.6.0], pid[855                                                                                      2], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-12-30 14:15:44,511][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] initializing ...
[2015-12-30 14:15:44,524][INFO ][plugins                  ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] loaded [knapsack-1.5.2.                                                                                      0-f340ad1], sites []
[2015-12-30 14:15:44,587][INFO ][env                      ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] using [1] data paths, m                                                                                      ounts [[/nfs/logdata (10.242.145.237:/vol/v_kdcnagiosnfs1_kdcnagls1n3_logs)]], net usable_space [1.4tb], net total_space [                                                                                      2.4tb], types [nfs]
[2015-12-30 14:15:49,375][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] initialized
[2015-12-30 14:15:49,375][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] starting ...
[2015-12-30 14:15:49,501][INFO ][transport                ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] bound_address {inet[/0:                                                                                      0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.242.102.109:9300]}
[2015-12-30 14:15:49,511][INFO ][discovery                ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] 80e9022e-f73f-429e-8927                                                                                      -f23d0d88dfd2/JEOqkDrhRwSBJG84nTHzvA


[2015-12-30 14:15:59,538][INFO ][cluster.service          ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] new_master [e63648a3-d9                                                                                      12-4f5d-a867-1b99282a5e7c][JEOqkDrhRwSBJG84nTHzvA][kdcnagls1n3.bchydro.bc.ca][inet[/10.242.102.109:9300]]{max_local_storag                                                                                      e_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-12-30 14:15:59,935][INFO ][http                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] bound_address {inet[/12                                                                                      7.0.0.1:9200]}, publish_address {inet[localhost/127.0.0.1:9200]}
[2015-12-30 14:15:59,936][INFO ][node                     ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] started
[2015-12-30 14:16:00,176][INFO ][discovery.zen            ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] updating discovery.zen.                                                                                      minimum_master_nodes from [-1] to [1]
[2015-12-30 14:16:00,630][INFO ][gateway                  ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] recovered [327] indices                                                                                       into cluster_state
[2015-12-30 14:16:01,641][DEBUG][action.search.type       ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] All shards failed for p                                                                                      hase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null

        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.                                                                                      java:160)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchA                                                                                      ction.java:57)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchA                                                                                      ction.java:47)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:104)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)
        at org.elasticsearch.client.FilterClient.execute(FilterClient.java:66)
        at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.execute(BaseRestHandler.java:92)
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338)
        at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:84)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:53)
        at org.elasticsearch.rest.RestController.executeHandler(RestController.java:225)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:170)
        at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
        at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
        at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:327)
        at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
        at org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:                                                                                      145)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:10                                                                                      8)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:                                                                                      459)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
[2015-12-30 14:16:01,641][DEBUG][action.search.type       ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] All shards failed for p                                                                                      hase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.                                                                                      java:160)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchA                                                                                      ction.java:57)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchA                                                                                      ction.java:47)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:104)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)
        at org.elasticsearch.client.FilterClient.execute(FilterClient.java:66)
        at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.execute(BaseRestHandler.java:92)
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338)
        at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:84)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:53)
        at org.elasticsearch.rest.RestController.executeHandler(RestController.java:225)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:170)
        at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
        at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
        at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:327)
        at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
        at org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:                                                                                      145)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:10                                                                                      8)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:                                                                                      459)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
[2015-12-30 14:16:01,661][DEBUG][action.search.type       ] [e63648a3-d912-4f5d-a867-1b99282a5e7c] All shards failed for p                                                                                      hase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.                                                                                      java:160)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchA                                                                                      ction.java:57)
        at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchA                                                                                      ction.java:47)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:104)
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)
        at org.elasticsearch.client.FilterClient.execute(FilterClient.java:66)
        at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.execute(BaseRestHandler.java:92)
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338)
        at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:84)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:53)
        at org.elasticsearch.rest.RestController.executeHandler(RestController.java:225)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:170)
        at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
        at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
        at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:327)
        at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
        at org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:                                                                                      145)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:10                                                                                      8)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:                                                                                      459)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler                                                                                      .java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Default                                                                                      ChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
So it think itself is the new master?

# curl 'localhost:9200/_cat/nodes?v'
host ip heap.percent ram.percent load node.role master name
kdcnagls1n3a 10.242.102.109 16 71 0.07 d * e63648a3-d912-4f5d-a867-1b99282a5e7c

# curl 'localhost:9200/_cat/master?v'
id host ip node
JEOqkDrhRwSBJG84nTHzvA kdcnagls1n3 10.242.102.109 e63648a3-d912-4f5d-a867-1b99282a5e7c

Other nodes...
# curl 'localhost:9200/_cat/nodes?v'
host ip heap.percent ram.percent load node.role master name
kdcnagls2n1 10.242.144.212 70 72 0.84 d m 9a92d6ef-d554-49d8-9191-dcf886382926
kdcnagls2n2 10.242.144.213 66 72 0.73 d * c424515a-16b3-43f9-866e-19daedef8a63
kdcnagls2n3 10.242.144.214 72 76 0.81 d m 2db4ce89-4c01-4a30-9bc8-66e987b7d613
kdcnagls1n1 10.242.102.107 48 78 4.51 d m 4521585a-88af-47c9-81e5-c4d13cffb148
kdcnagls1n2 10.242.102.108 66 74 0.30 d m 30ab2b2c-439f-4bcc-977d-7c0e9a90f3a5

# curl 'localhost:9200/_cat/master?v'
id host ip node
OcC8_UCIR5qjNEIYYTIzfA kdcnagls2n2 10.242.144.213 c424515a-16b3-43f9-866e-19daedef8a63

Re: Node timeout and cannot join cluster

Posted: Wed Dec 30, 2015 5:45 pm
by CFT6Server
ok think I got it fixed. I had to take the current real master offline. The minute I've done that, the one that won't join become the master and all the other nodes joined. Did a rolling restart just in case and now waiting for the shards to 'recover' and reinitialize. At least I am back to 6 nodes and yellow status.... will keep an eye on it.

Re: Node timeout and cannot join cluster

Posted: Mon Jan 04, 2016 10:28 am
by bwallace
Any changes since your previous post? Let us know of the status whenever you have a moment, thanks.

Re: Node timeout and cannot join cluster

Posted: Mon Jan 04, 2016 4:34 pm
by CFT6Server
The nodes are back to normal and issues is resolved. Thanks.

Re: Node timeout and cannot join cluster

Posted: Mon Jan 04, 2016 4:50 pm
by rkennedy
Thanks for the update, and glad to see things are working! I'll go ahead and close this thread out, but feel free to open a new one if you ever need assistance!
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited