Slave became master and won't go back

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
2evanowen
Posts: 35
Joined: Fri Jan 16, 2015 1:24 pm

Slave became master and won't go back

Post by 2evanowen »

Hey guys,
After an os update on my master, the slave became a master and won't go back. I have tried rebooting the slave, and stopping the elastic search service on both hosts but it is pretty positive it is the master. Looking at the Instance Status on each host, it states that Elasticsearch + Logstash are down on the other host. Both have the same cluster id as well.

How do I get my slave to become a slave again?
(real master)

Code: Select all

[owen@calcium ~]$ curl 'localhost:9200/_cat/master?v'
id                     host                     ip           node
O09GtvlHQdiP5NwoE_InCA calcium.colo.seagate.com 10.30.216.20 707a55e2-e8cb-4073-a800-2fbc5bd4c6df

[owen@calcium ~]$ curl 'localhost:9200/_cat/nodes?v'
host                     ip           heap.percent ram.percent load node.role master name
calcium.colo.seagate.com 10.30.216.20           13          59 0.32 d         *      707a55e2-e8cb-4073-a800-2fbc5bd4c6df

[owen@calcium ~]$ curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "76900ee2-f769-413c-9948-850204a96b32",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2166,
  "active_shards" : 2166,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 2166,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}
(slave)

Code: Select all

[owen@barium ~]$ curl 'localhost:9200/_cat/master?v'
id                     host                    ip           node
L-9RRltKSvivOmEVB6l-RA barium.colo.seagate.com 10.30.216.56 c69070c3-90bb-49f3-9199-9037ff45d3bc

[owen@barium ~]$ curl 'localhost:9200/_cat/nodes?v'
host                    ip           heap.percent ram.percent load node.role master name
barium.colo.seagate.com 10.30.216.56           11          58 0.02 d         *      c69070c3-90bb-49f3-9199-9037ff45d3bc

[owen@barium ~]$ curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "76900ee2-f769-413c-9948-850204a96b32",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2166,
  "active_shards" : 2166,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 2166,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Slave became master and won't go back

Post by jolson »

Run the following on both nodes:

Code: Select all

cat /usr/local/nagioslogserver/var/cluster_*
ip a
tail -n200 /var/log/elasticsearch/*.log
cat /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml
I hope the above will be adequate to move this case forward.

Jesse
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
2evanowen
Posts: 35
Joined: Fri Jan 16, 2015 1:24 pm

Re: Slave became master and won't go back

Post by 2evanowen »

(master)

Code: Select all

[owen@calcium ~]$ cat /usr/local/nagioslogserver/var/cluster_*
localhost

10.30.216.20
10.30.216.5676900ee2-f769-413c-9948-850204a96b32[owen@calcium ~]$
[owen@calcium ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether f0:1f:af:d5:4a:25 brd ff:ff:ff:ff:ff:ff
    inet 10.30.216.20/20 brd 10.30.223.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::f21f:afff:fed5:4a25/64 scope link
       valid_lft forever preferred_lft forever
3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether f0:1f:af:d5:4a:26 brd ff:ff:ff:ff:ff:ff
[owen@calcium ~]$ tail -n 200 /var/log/elasticsearch/*.log
==> /var/log/elasticsearch/76900ee2-f769-413c-9948-850204a96b32_index_indexing_slowlog.log <==

==> /var/log/elasticsearch/76900ee2-f769-413c-9948-850204a96b32_index_search_slowlog.log <==

==> /var/log/elasticsearch/76900ee2-f769-413c-9948-850204a96b32.log <==
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:466)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [Jan 27 12:36:41], tried both date format [dateOptionalTime], and timestamp number with locale []
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:617)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:535)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Jan 27 12:36:41"
	at org.elasticsearch.common.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
	at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:612)
	... 16 more
[2016-01-27 12:36:42,770][WARN ][transport.netty          ] [707a55e2-e8cb-4073-a800-2fbc5bd4c6df] exception caught on transport layer [[id: 0x8d4c6b30]], closing connection
java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
[2016-01-27 12:36:43,533][DEBUG][action.bulk              ] [707a55e2-e8cb-4073-a800-2fbc5bd4c6df] [logstash-2016.01.27][4] failed to execute bulk item (index) index {[logstash-2016.01.27][syslog][AVKElocMXTchF9UHZ_CP], source[{"message":"Accepted publickey for postgres from 10.30.216.131 port 51390 ssh2\n","@version":"1","@timestamp":"2016-01-27T19:36:43.000Z","type":"syslog","host":"10.30.216.132","priority":86,"timestamp":"Jan 27 12:36:43","logsource":"cyclonus","program":"sshd","pid":"20388","severity":6,"facility":10,"facility_label":"security/authorization","severity_label":"Informational"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [timestamp]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:466)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [Jan 27 12:36:43], tried both date format [dateOptionalTime], and timestamp number with locale []
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:617)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:535)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Jan 27 12:36:43"
	at org.elasticsearch.common.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
	at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:612)
	... 16 more
[2016-01-27 12:36:43,725][DEBUG][action.bulk              ] [707a55e2-e8cb-4073-a800-2fbc5bd4c6df] [logstash-2016.01.27][4] failed to execute bulk item (index) index {[logstash-2016.01.27][syslog][AVKElofNXTchF9UHZ_CQ], source[{"message":"pam_unix(sshd:session): session opened for user postgres by (uid=0)\n","@version":"1","@timestamp":"2016-01-27T19:36:43.000Z","type":"syslog","host":"10.30.216.132","priority":86,"timestamp":"Jan 27 12:36:43","logsource":"cyclonus","program":"sshd","pid":"20388","severity":6,"facility":10,"facility_label":"security/authorization","severity_label":"Informational"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [timestamp]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:466)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [Jan 27 12:36:43], tried both date format [dateOptionalTime], and timestamp number with locale []
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:617)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:535)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Jan 27 12:36:43"
	at org.elasticsearch.common.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
	at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:612)
	... 16 more
[2016-01-27 12:36:44,217][DEBUG][action.bulk              ] [707a55e2-e8cb-4073-a800-2fbc5bd4c6df] [logstash-2016.01.27][1] failed to execute bulk item (index) index {[logstash-2016.01.27][syslog][AVKElom4XTchF9UHZ_CS], source[{"message":"Compiled catalog for straxus.colo.seagate.com in environment production in 0.19 seconds\n","@version":"1","@timestamp":"2016-01-27T19:36:43.000Z","type":"syslog","host":"10.30.217.78","priority":29,"timestamp":"Jan 27 12:36:43","logsource":"platinum","program":"puppet-master","pid":"11875","severity":5,"facility":3,"facility_label":"system","severity_label":"Notice"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [timestamp]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:466)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [Jan 27 12:36:43], tried both date format [dateOptionalTime], and timestamp number with locale []
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:617)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:535)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Jan 27 12:36:43"
	at org.elasticsearch.common.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
	at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:612)
	... 16 more
[2016-01-27 12:36:44,217][DEBUG][action.bulk              ] [707a55e2-e8cb-4073-a800-2fbc5bd4c6df] [logstash-2016.01.27][2] failed to execute bulk item (index) index {[logstash-2016.01.27][syslog][AVKElom4XTchF9UHZ_CR], source[{"message":"Received disconnect from 10.30.216.131: 11: disconnected by user\n","@version":"1","@timestamp":"2016-01-27T19:36:43.000Z","type":"syslog","host":"10.30.216.132","priority":86,"timestamp":"Jan 27 12:36:43","logsource":"cyclonus","program":"sshd","pid":"20391","severity":6,"facility":10,"facility_label":"security/authorization","severity_label":"Informational"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [timestamp]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:466)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [Jan 27 12:36:43], tried both date format [dateOptionalTime], and timestamp number with locale []
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:617)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:535)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Jan 27 12:36:43"
	at org.elasticsearch.common.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
	at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:612)
	... 16 more
[2016-01-27 12:36:44,262][DEBUG][action.bulk              ] [707a55e2-e8cb-4073-a800-2fbc5bd4c6df] [logstash-2016.01.27][3] failed to execute bulk item (index) index {[logstash-2016.01.27][syslog][AVKElonmXTchF9UHZ_CY], source[{"message":"Ignoring invalid UTF-8 byte sequences in data to be sent to PuppetDB\n","@version":"1","@timestamp":"2016-01-27T19:36:44.000Z","type":"syslog","host":"10.30.217.78","priority":28,"timestamp":"Jan 27 12:36:44","logsource":"platinum","program":"puppet-master","pid":"11875","severity":4,"facility":3,"facility_label":"system","severity_label":"Warning"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [timestamp]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:466)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [Jan 27 12:36:44], tried both date format [dateOptionalTime], and timestamp number with locale []
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:617)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:535)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Jan 27 12:36:44"
	at org.elasticsearch.common.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
	at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:612)
	... 16 more
[2016-01-27 12:36:44,262][DEBUG][action.bulk              ] [707a55e2-e8cb-4073-a800-2fbc5bd4c6df] [logstash-2016.01.27][0] failed to execute bulk item (index) index {[logstash-2016.01.27][syslog][AVKElonmXTchF9UHZ_CV], source[{"message":"pam_unix(sshd:session): session closed for user postgres\n","@version":"1","@timestamp":"2016-01-27T19:36:43.000Z","type":"syslog","host":"10.30.216.132","priority":86,"timestamp":"Jan 27 12:36:43","logsource":"cyclonus","program":"sshd","pid":"20388","severity":6,"facility":10,"facility_label":"security/authorization","severity_label":"Informational"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [timestamp]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:466)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [Jan 27 12:36:43], tried both date format [dateOptionalTime], and timestamp number with locale []
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:617)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:535)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Jan 27 12:36:43"
	at org.elasticsearch.common.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
	at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:612)
	... 16 more

==> /var/log/elasticsearch/b2c9806c-b9cf-425f-aa05-fae191e8d33b_index_indexing_slowlog.log <==

==> /var/log/elasticsearch/b2c9806c-b9cf-425f-aa05-fae191e8d33b_index_search_slowlog.log <==

==> /var/log/elasticsearch/b2c9806c-b9cf-425f-aa05-fae191e8d33b.log <==
[owen@calcium ~]$ cat /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml
##################### Elasticsearch Configuration Example #####################

# This file contains an overview of various configuration settings,
# targeted at operations staff. Application developers should
# consult the guide at <http://elasticsearch.org/guide>.
#
# The installation procedure is covered at
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.
#
# Elasticsearch comes with reasonable defaults for most settings,
# so you can try it out without bothering with configuration.
#
# Most of the time, these defaults are just fine for running a production
# cluster. If you're fine-tuning your cluster, or wondering about the
# effect of certain configuration option, please _do ask_ on the
# mailing list or IRC channel [http://elasticsearch.org/community].

# Any element in the configuration can be replaced with environment variables
# by placing them in ${...} notation. For example:
#
# node.rack: ${RACK_ENV_VAR}

# For information on supported formats and syntax for the config file, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>


################################### Cluster ###################################

# Cluster name identifies your cluster for auto-discovery. If you're running
# multiple clusters on the same network, make sure you're using unique names.
#
cluster.name: nagios_elasticsearch


#################################### Node #####################################

# Node names are generated dynamically on startup, so you're relieved
# from configuring them manually. You can tie this node to a specific name:
#
# node.name: "Franz Kafka"

# Every node can be configured to allow or deny being eligible as the master,
# and to allow or deny to store the data.
#
# Allow this node to be eligible as a master node (enabled by default):
#
# node.master: true
#
# Allow this node to store data (enabled by default):
#
# node.data: true

# You can exploit these settings to design advanced cluster topologies.
#
# 1. You want this node to never become a master node, only to hold data.
#    This will be the "workhorse" of your cluster.
#
# node.master: false
# node.data: true
#
# 2. You want this node to only serve as a master: to not store any data and
#    to have free resources. This will be the "coordinator" of your cluster.
#
# node.master: true
# node.data: false
#
# 3. You want this node to be neither master nor data node, but
#    to act as a "search load balancer" (fetching data from nodes,
#    aggregating results, etc.)
#
# node.master: false
# node.data: false

# Use the Cluster Health API [http://localhost:9200/_cluster/health], the
# Node Info API [http://localhost:9200/_nodes] or GUI tools
# such as <http://www.elasticsearch.org/overview/marvel/>,
# <http://github.com/karmi/elasticsearch-paramedic>,
# <http://github.com/lukas-vlcek/bigdesk> and
# <http://mobz.github.com/elasticsearch-head> to inspect the cluster state.

# A node can have generic attributes associated with it, which can later be used
# for customized shard allocation filtering, or allocation awareness. An attribute
# is a simple key value pair, similar to node.key: value, here is an example:
#
# node.rack: rack314

# By default, multiple nodes are allowed to start from the same installation location
# to disable it, set the following:
node.max_local_storage_nodes: 1


#################################### Index ####################################

# You can set a number of options (such as shard/replica options, mapping
# or analyzer definitions, translog settings, ...) for indices globally,
# in this file.
#
# Note, that it makes more sense to configure index settings specifically for
# a certain index, either when creating it or by using the index templates API.
#
# See <http://elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules.html> and
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/indices-create-index.html>
# for more information.

# Set the number of shards (splits) of an index (5 by default):
#
# index.number_of_shards: 5

# Set the number of replicas (additional copies) of an index (1 by default):
#
# index.number_of_replicas: 1

# Note, that for development on a local machine, with small indices, it usually
# makes sense to "disable" the distributed features:
#
# index.number_of_shards: 1
# index.number_of_replicas: 0

# These settings directly affect the performance of index and search operations
# in your cluster. Assuming you have enough machines to hold shards and
# replicas, the rule of thumb is:
#
# 1. Having more *shards* enhances the _indexing_ performance and allows to
#    _distribute_ a big index across machines.
# 2. Having more *replicas* enhances the _search_ performance and improves the
#    cluster _availability_.
#
# The "number_of_shards" is a one-time setting for an index.
#
# The "number_of_replicas" can be increased or decreased anytime,
# by using the Index Update Settings API.
#
# Elasticsearch takes care about load balancing, relocating, gathering the
# results from nodes, etc. Experiment with different settings to fine-tune
# your setup.

# Use the Index Status API (<http://localhost:9200/A/_status>) to inspect
# the index status.


#################################### Paths ####################################

# Path to directory containing configuration (this file and logging.yml):
#
# path.conf: /path/to/conf

# Path to directory where to store index data allocated for this node.
#
# path.data: /path/to/data
#
# Can optionally include more than one location, causing data to be striped across
# the locations (a la RAID 0) on a file level, favouring locations with most free
# space on creation. For example:
#
# path.data: /path/to/data1,/path/to/data2

# Path to temporary files:
#
# path.work: /path/to/work

# Path to log files:
#
# path.logs: /path/to/logs

# Path to where plugins are installed:
#
# path.plugins: /path/to/plugins


#################################### Plugin ###################################

# If a plugin listed here is not installed for current node, the node will not start.
#
# plugin.mandatory: mapper-attachments,lang-groovy


################################### Memory ####################################

# Elasticsearch performs poorly when JVM starts swapping: you should ensure that
# it _never_ swaps.
#
# Set this property to true to lock the memory:
#
bootstrap.mlockall: true

# Make sure that the ES_MIN_MEM and ES_MAX_MEM environment variables are set
# to the same value, and that the machine has enough memory to allocate
# for Elasticsearch, leaving enough memory for the operating system itself.
#
# You should also make sure that the Elasticsearch process is allowed to lock
# the memory, eg. by using `ulimit -l unlimited`.


############################## Network And HTTP ###############################

# Elasticsearch, by default, binds itself to the 0.0.0.0 address, and listens
# on port [9200-9300] for HTTP traffic and on port [9300-9400] for node-to-node
# communication. (the range means that if the port is busy, it will automatically
# try the next port).

# Set the bind address specifically (IPv4 or IPv6):
#
# network.bind_host: 192.168.0.1

# Set the address other nodes will use to communicate with this node. If not
# set, it is automatically derived. It must point to an actual IP address.
#
# network.publish_host: 192.168.0.1

# Set both 'bind_host' and 'publish_host':
#
# network.host: 192.168.0.1

# Set a custom port for the node to node communication (9300 by default):
#
# transport.tcp.port: 9300

# Enable compression for all communication between nodes (disabled by default):
#
transport.tcp.compress: true

# Set a custom port to listen for HTTP traffic:
#
# http.port: 9200

# Set a custom allowed content length:
#
# http.max_content_length: 100mb

# Disable HTTP completely:
#
# http.enabled: false

# Set the HTTP host to listen to
#
http.host: "localhost"

################################### Gateway ###################################

# The gateway allows for persisting the cluster state between full cluster
# restarts. Every change to the state (such as adding an index) will be stored
# in the gateway, and when the cluster starts up for the first time,
# it will read its state from the gateway.

# There are several types of gateway implementations. For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html>.

# The default gateway type is the "local" gateway (recommended):
#
# gateway.type: local

# Settings below control how and when to start the initial recovery process on
# a full cluster restart (to reuse as much local data as possible when using shared
# gateway).

# Allow recovery process after N nodes in a cluster are up:
#
# gateway.recover_after_nodes: 1

# Set the timeout to initiate the recovery process, once the N nodes
# from previous setting are up (accepts time value):
#
# gateway.recover_after_time: 5m

# Set how many nodes are expected in this cluster. Once these N nodes
# are up (and recover_after_nodes is met), begin recovery process immediately
# (without waiting for recover_after_time to expire):
#
# gateway.expected_nodes: 2


############################# Recovery Throttling #############################

# These settings allow to control the process of shards allocation between
# nodes during initial recovery, replica allocation, rebalancing,
# or when adding and removing nodes.

# Set the number of concurrent recoveries happening on a node:
#
# 1. During the initial recovery
#
# cluster.routing.allocation.node_initial_primaries_recoveries: 4
#
# 2. During adding/removing nodes, rebalancing, etc
#
# cluster.routing.allocation.node_concurrent_recoveries: 2

# Set to throttle throughput when recovering (eg. 100mb, by default 20mb):
#
# indices.recovery.max_bytes_per_sec: 20mb

# Set to limit the number of open concurrent streams when
# recovering a shard from a peer:
#
# indices.recovery.concurrent_streams: 5


################################## Discovery ##################################

# Discovery infrastructure ensures nodes can be found within a cluster
# and master node is elected. Multicast discovery is the default.

# Set to ensure a node sees N other master eligible nodes to be considered
# operational within the cluster. Its recommended to set it to a higher value
# than 1 when running more than 2 nodes in the cluster.
#
# discovery.zen.minimum_master_nodes: 1

# Set the time to wait for ping responses from other nodes when discovering.
# Set this option to a higher value on a slow or congested network
# to minimize discovery failures:
#
# discovery.zen.ping.timeout: 3s

# For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>

# Unicast discovery allows to explicitly control which nodes will be used
# to discover the cluster. It can be used when multicast is not present,
# or to restrict the cluster communication-wise.
#
# 1. Disable multicast discovery (enabled by default):
#
discovery.zen.ping.multicast.enabled: false
#
# 2. Configure an initial list of master nodes in the cluster
#    to perform discovery when new nodes (master or data) are started:
#
discovery.zen.ping.unicast.hosts: ["localhost"]

# EC2 discovery allows to use AWS EC2 API in order to perform discovery.
#
# You have to install my butt-aws plugin for enabling the EC2 discovery.
#
# For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>
#
# See <http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>
# for a step-by-step tutorial.

# GCE discovery allows to use Google Compute Engine API in order to perform discovery.
#
# You have to install my butt-gce plugin for enabling the GCE discovery.
#
# For more information, see <https://github.com/elasticsearch/elasticsearch-butt-gce>.

# Azure discovery allows to use Azure API in order to perform discovery.
#
# You have to install my butt-azure plugin for enabling the Azure discovery.
#
# For more information, see <https://github.com/elasticsearch/elasticsearch-butt-azure>.

################################## Slow Log ##################################

# Shard level query and fetch threshold logging.

#index.search.slowlog.threshold.query.warn: 10s
#index.search.slowlog.threshold.query.info: 5s
#index.search.slowlog.threshold.query.debug: 2s
#index.search.slowlog.threshold.query.trace: 500ms

#index.search.slowlog.threshold.fetch.warn: 1s
#index.search.slowlog.threshold.fetch.info: 800ms
#index.search.slowlog.threshold.fetch.debug: 500ms
#index.search.slowlog.threshold.fetch.trace: 200ms

#index.indexing.slowlog.threshold.index.warn: 10s
#index.indexing.slowlog.threshold.index.info: 5s
#index.indexing.slowlog.threshold.index.debug: 2s
#index.indexing.slowlog.threshold.index.trace: 500ms

################################## GC Logging ################################

#monitor.jvm.gc.young.warn: 1000ms
#monitor.jvm.gc.young.info: 700ms
#monitor.jvm.gc.young.debug: 400ms

#monitor.jvm.gc.old.warn: 10s
#monitor.jvm.gc.old.info: 5s
#monitor.jvm.gc.old.debug: 2s
[owen@calcium ~]$
Top
2evanowen
Posts: 35
Joined: Fri Jan 16, 2015 1:24 pm

Re: Slave became master and won't go back

Post by 2evanowen »

(slave)

Code: Select all

[owen@barium ~]$ cat /usr/local/nagioslogserver/var/cluster_*
localhost

10.30.216.56
10.30.216.2076900ee2-f769-413c-9948-850204a96b32[owen@barium ~]$
[owen@barium ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether f0:1f:af:d5:21:5a brd ff:ff:ff:ff:ff:ff
    inet 10.30.216.56/20 brd 10.30.223.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::f21f:afff:fed5:215a/64 scope link
       valid_lft forever preferred_lft forever
3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether f0:1f:af:d5:21:5b brd ff:ff:ff:ff:ff:ff

[owen@barium ~]$ tail -n 100 /var/log/elasticsearch/*.log
==> /var/log/elasticsearch/76900ee2-f769-413c-9948-850204a96b32_index_indexing_slowlog.log <==

==> /var/log/elasticsearch/76900ee2-f769-413c-9948-850204a96b32_index_search_slowlog.log <==

==> /var/log/elasticsearch/76900ee2-f769-413c-9948-850204a96b32.log <==
[2016-01-27 12:38:15,947][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:18,953][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:21,959][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:24,966][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:27,972][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:30,978][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:31,960][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:38:32,060][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:38:32,063][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:38:32,066][DEBUG][action.admin.cluster.state] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2016-01-27 12:38:32,168][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:38:33,985][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:36,990][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:39,996][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:43,001][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:46,007][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:49,015][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:51,408][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:38:51,418][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:38:51,459][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:38:51,459][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:38:52,021][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:52,416][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:38:52,424][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:38:52,469][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:38:52,469][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:38:55,027][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:38:58,034][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:01,040][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:01,175][DEBUG][action.admin.cluster.state] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:39:04,046][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:07,053][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:10,059][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:13,066][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:16,072][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:19,079][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:22,087][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:25,092][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:28,098][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:31,104][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:31,176][DEBUG][action.admin.cluster.state] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2016-01-27 12:39:31,284][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:39:32,064][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:39:32,168][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:39:32,170][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:39:34,110][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:37,116][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:40,122][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:43,128][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:46,133][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:49,140][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:52,147][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:52,416][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:39:52,426][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:39:52,469][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:39:52,470][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:39:53,425][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:39:53,432][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:39:53,480][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:39:53,481][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:39:55,155][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:39:58,161][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:01,167][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:01,290][DEBUG][action.admin.cluster.state] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:40:04,173][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:07,181][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:10,187][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:13,193][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:16,199][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:19,204][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:22,211][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:25,217][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:28,223][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:31,229][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:31,285][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:40:31,288][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:40:31,291][DEBUG][action.admin.cluster.state] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2016-01-27 12:40:31,412][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:40:32,171][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:40:34,235][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:37,241][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:40,246][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:43,254][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:46,260][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:49,266][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:52,272][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:53,425][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:40:53,433][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:40:53,481][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:40:53,481][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-01-27 12:40:54,434][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:40:54,439][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:40:54,491][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:40:54,492][DEBUG][action.admin.indices.create] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:40:55,280][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:40:58,286][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:41:01,291][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:41:01,405][DEBUG][action.admin.cluster.state] [c69070c3-90bb-49f3-9199-9037ff45d3bc] no known master node, scheduling a retry
[2016-01-27 12:41:04,297][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]
[2016-01-27 12:41:07,303][INFO ][discovery.zen            ] [c69070c3-90bb-49f3-9199-9037ff45d3bc] failed to send join request to master [[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][fBkO7iAlSR6Lj7mea49lIg][calcium.colo.seagate.com][inet[/10.30.216.20:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[707a55e2-e8cb-4073-a800-2fbc5bd4c6df][inet[/10.30.216.20:9300]][internal:discovery/zen/join]]; nested: ConnectTransportException[[c69070c3-90bb-49f3-9199-9037ff45d3bc][inet[/10.30.216.56:9300]] connect_timeout[30s]]; nested: NoRouteToHostException[No route to host]; ]

==> /var/log/elasticsearch/ed2451eb-d1c2-48d5-977f-714a9eb0c1ab_index_indexing_slowlog.log <==

==> /var/log/elasticsearch/ed2451eb-d1c2-48d5-977f-714a9eb0c1ab_index_search_slowlog.log <==

==> /var/log/elasticsearch/ed2451eb-d1c2-48d5-977f-714a9eb0c1ab.log <==

[owen@barium ~]$ cat /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml
##################### Elasticsearch Configuration Example #####################

# This file contains an overview of various configuration settings,
# targeted at operations staff. Application developers should
# consult the guide at <http://elasticsearch.org/guide>.
#
# The installation procedure is covered at
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.
#
# Elasticsearch comes with reasonable defaults for most settings,
# so you can try it out without bothering with configuration.
#
# Most of the time, these defaults are just fine for running a production
# cluster. If you're fine-tuning your cluster, or wondering about the
# effect of certain configuration option, please _do ask_ on the
# mailing list or IRC channel [http://elasticsearch.org/community].

# Any element in the configuration can be replaced with environment variables
# by placing them in ${...} notation. For example:
#
# node.rack: ${RACK_ENV_VAR}

# For information on supported formats and syntax for the config file, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>


################################### Cluster ###################################

# Cluster name identifies your cluster for auto-discovery. If you're running
# multiple clusters on the same network, make sure you're using unique names.
#
cluster.name: nagios_elasticsearch


#################################### Node #####################################

# Node names are generated dynamically on startup, so you're relieved
# from configuring them manually. You can tie this node to a specific name:
#
# node.name: "Franz Kafka"

# Every node can be configured to allow or deny being eligible as the master,
# and to allow or deny to store the data.
#
# Allow this node to be eligible as a master node (enabled by default):
#
# node.master: true
#
# Allow this node to store data (enabled by default):
#
# node.data: true

# You can exploit these settings to design advanced cluster topologies.
#
# 1. You want this node to never become a master node, only to hold data.
#    This will be the "workhorse" of your cluster.
#
# node.master: false
# node.data: true
#
# 2. You want this node to only serve as a master: to not store any data and
#    to have free resources. This will be the "coordinator" of your cluster.
#
# node.master: true
# node.data: false
#
# 3. You want this node to be neither master nor data node, but
#    to act as a "search load balancer" (fetching data from nodes,
#    aggregating results, etc.)
#
# node.master: false
# node.data: false

# Use the Cluster Health API [http://localhost:9200/_cluster/health], the
# Node Info API [http://localhost:9200/_nodes] or GUI tools
# such as <http://www.elasticsearch.org/overview/marvel/>,
# <http://github.com/karmi/elasticsearch-paramedic>,
# <http://github.com/lukas-vlcek/bigdesk> and
# <http://mobz.github.com/elasticsearch-head> to inspect the cluster state.

# A node can have generic attributes associated with it, which can later be used
# for customized shard allocation filtering, or allocation awareness. An attribute
# is a simple key value pair, similar to node.key: value, here is an example:
#
# node.rack: rack314

# By default, multiple nodes are allowed to start from the same installation location
# to disable it, set the following:
node.max_local_storage_nodes: 1


#################################### Index ####################################

# You can set a number of options (such as shard/replica options, mapping
# or analyzer definitions, translog settings, ...) for indices globally,
# in this file.
#
# Note, that it makes more sense to configure index settings specifically for
# a certain index, either when creating it or by using the index templates API.
#
# See <http://elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules.html> and
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/indices-create-index.html>
# for more information.

# Set the number of shards (splits) of an index (5 by default):
#
# index.number_of_shards: 5

# Set the number of replicas (additional copies) of an index (1 by default):
#
# index.number_of_replicas: 1

# Note, that for development on a local machine, with small indices, it usually
# makes sense to "disable" the distributed features:
#
# index.number_of_shards: 1
# index.number_of_replicas: 0

# These settings directly affect the performance of index and search operations
# in your cluster. Assuming you have enough machines to hold shards and
# replicas, the rule of thumb is:
#
# 1. Having more *shards* enhances the _indexing_ performance and allows to
#    _distribute_ a big index across machines.
# 2. Having more *replicas* enhances the _search_ performance and improves the
#    cluster _availability_.
#
# The "number_of_shards" is a one-time setting for an index.
#
# The "number_of_replicas" can be increased or decreased anytime,
# by using the Index Update Settings API.
#
# Elasticsearch takes care about load balancing, relocating, gathering the
# results from nodes, etc. Experiment with different settings to fine-tune
# your setup.

# Use the Index Status API (<http://localhost:9200/A/_status>) to inspect
# the index status.


#################################### Paths ####################################

# Path to directory containing configuration (this file and logging.yml):
#
# path.conf: /path/to/conf

# Path to directory where to store index data allocated for this node.
#
# path.data: /path/to/data
#
# Can optionally include more than one location, causing data to be striped across
# the locations (a la RAID 0) on a file level, favouring locations with most free
# space on creation. For example:
#
# path.data: /path/to/data1,/path/to/data2

# Path to temporary files:
#
# path.work: /path/to/work

# Path to log files:
#
# path.logs: /path/to/logs

# Path to where plugins are installed:
#
# path.plugins: /path/to/plugins


#################################### Plugin ###################################

# If a plugin listed here is not installed for current node, the node will not start.
#
# plugin.mandatory: mapper-attachments,lang-groovy


################################### Memory ####################################

# Elasticsearch performs poorly when JVM starts swapping: you should ensure that
# it _never_ swaps.
#
# Set this property to true to lock the memory:
#
bootstrap.mlockall: true

# Make sure that the ES_MIN_MEM and ES_MAX_MEM environment variables are set
# to the same value, and that the machine has enough memory to allocate
# for Elasticsearch, leaving enough memory for the operating system itself.
#
# You should also make sure that the Elasticsearch process is allowed to lock
# the memory, eg. by using `ulimit -l unlimited`.


############################## Network And HTTP ###############################

# Elasticsearch, by default, binds itself to the 0.0.0.0 address, and listens
# on port [9200-9300] for HTTP traffic and on port [9300-9400] for node-to-node
# communication. (the range means that if the port is busy, it will automatically
# try the next port).

# Set the bind address specifically (IPv4 or IPv6):
#
# network.bind_host: 192.168.0.1

# Set the address other nodes will use to communicate with this node. If not
# set, it is automatically derived. It must point to an actual IP address.
#
# network.publish_host: 192.168.0.1

# Set both 'bind_host' and 'publish_host':
#
# network.host: 192.168.0.1

# Set a custom port for the node to node communication (9300 by default):
#
# transport.tcp.port: 9300

# Enable compression for all communication between nodes (disabled by default):
#
transport.tcp.compress: true

# Set a custom port to listen for HTTP traffic:
#
# http.port: 9200

# Set a custom allowed content length:
#
# http.max_content_length: 100mb

# Disable HTTP completely:
#
# http.enabled: false

# Set the HTTP host to listen to
#
http.host: "localhost"

################################### Gateway ###################################

# The gateway allows for persisting the cluster state between full cluster
# restarts. Every change to the state (such as adding an index) will be stored
# in the gateway, and when the cluster starts up for the first time,
# it will read its state from the gateway.

# There are several types of gateway implementations. For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html>.

# The default gateway type is the "local" gateway (recommended):
#
# gateway.type: local

# Settings below control how and when to start the initial recovery process on
# a full cluster restart (to reuse as much local data as possible when using shared
# gateway).

# Allow recovery process after N nodes in a cluster are up:
#
# gateway.recover_after_nodes: 1

# Set the timeout to initiate the recovery process, once the N nodes
# from previous setting are up (accepts time value):
#
# gateway.recover_after_time: 5m

# Set how many nodes are expected in this cluster. Once these N nodes
# are up (and recover_after_nodes is met), begin recovery process immediately
# (without waiting for recover_after_time to expire):
#
# gateway.expected_nodes: 2


############################# Recovery Throttling #############################

# These settings allow to control the process of shards allocation between
# nodes during initial recovery, replica allocation, rebalancing,
# or when adding and removing nodes.

# Set the number of concurrent recoveries happening on a node:
#
# 1. During the initial recovery
#
# cluster.routing.allocation.node_initial_primaries_recoveries: 4
#
# 2. During adding/removing nodes, rebalancing, etc
#
# cluster.routing.allocation.node_concurrent_recoveries: 2

# Set to throttle throughput when recovering (eg. 100mb, by default 20mb):
#
# indices.recovery.max_bytes_per_sec: 20mb

# Set to limit the number of open concurrent streams when
# recovering a shard from a peer:
#
# indices.recovery.concurrent_streams: 5


################################## Discovery ##################################

# Discovery infrastructure ensures nodes can be found within a cluster
# and master node is elected. Multicast discovery is the default.

# Set to ensure a node sees N other master eligible nodes to be considered
# operational within the cluster. Its recommended to set it to a higher value
# than 1 when running more than 2 nodes in the cluster.
#
# discovery.zen.minimum_master_nodes: 1

# Set the time to wait for ping responses from other nodes when discovering.
# Set this option to a higher value on a slow or congested network
# to minimize discovery failures:
#
# discovery.zen.ping.timeout: 3s

# For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>

# Unicast discovery allows to explicitly control which nodes will be used
# to discover the cluster. It can be used when multicast is not present,
# or to restrict the cluster communication-wise.
#
# 1. Disable multicast discovery (enabled by default):
#
discovery.zen.ping.multicast.enabled: false
#
# 2. Configure an initial list of master nodes in the cluster
#    to perform discovery when new nodes (master or data) are started:
#
discovery.zen.ping.unicast.hosts: ["localhost"]

# EC2 discovery allows to use AWS EC2 API in order to perform discovery.
#
# You have to install my butt-aws plugin for enabling the EC2 discovery.
#
# For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>
#
# See <http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>
# for a step-by-step tutorial.

# GCE discovery allows to use Google Compute Engine API in order to perform discovery.
#
# You have to install my butt-gce plugin for enabling the GCE discovery.
#
# For more information, see <https://github.com/elasticsearch/elasticsearch-butt-gce>.

# Azure discovery allows to use Azure API in order to perform discovery.
#
# You have to install my butt-azure plugin for enabling the Azure discovery.
#
# For more information, see <https://github.com/elasticsearch/elasticsearch-butt-azure>.

################################## Slow Log ##################################

# Shard level query and fetch threshold logging.

#index.search.slowlog.threshold.query.warn: 10s
#index.search.slowlog.threshold.query.info: 5s
#index.search.slowlog.threshold.query.debug: 2s
#index.search.slowlog.threshold.query.trace: 500ms

#index.search.slowlog.threshold.fetch.warn: 1s
#index.search.slowlog.threshold.fetch.info: 800ms
#index.search.slowlog.threshold.fetch.debug: 500ms
#index.search.slowlog.threshold.fetch.trace: 200ms

#index.indexing.slowlog.threshold.index.warn: 10s
#index.indexing.slowlog.threshold.index.info: 5s
#index.indexing.slowlog.threshold.index.debug: 2s
#index.indexing.slowlog.threshold.index.trace: 500ms

################################## GC Logging ################################

#monitor.jvm.gc.young.warn: 1000ms
#monitor.jvm.gc.young.info: 700ms
#monitor.jvm.gc.young.debug: 400ms

#monitor.jvm.gc.old.warn: 10s
#monitor.jvm.gc.old.info: 5s
#monitor.jvm.gc.old.debug: 2s
Top
2evanowen
Posts: 35
Joined: Fri Jan 16, 2015 1:24 pm

Re: Slave became master and won't go back

Post by 2evanowen »

After another restart of the elastic search service on the slave, it is now just stuck on the screen:

"Waiting for Database Startup
It looks like your local elasticsearch service is starting.

Why am I getting this error?

Elasticsearch can take a little while to start up because of it's indexing. This may take a few seconds.

The page will refresh automatically after 5 seconds..."
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Slave became master and won't go back

Post by jolson »

I'd like to perform a remote session with you, as both nodes currently believe they are the master, which can lead to a split-brain scenario. There are precautions that we can take against this, but we should get a remote scheduled as soon as we can. I'm available at 1600CT tonight, could you send an email to [email protected] with a reference to this thread please?

In the meantime, make the following changes on both nodes:

Open up vi /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml and make the following change:
Change: # discovery.zen.minimum_master_nodes: 1
To: discovery.zen.minimum_master_nodes: 2

And restart elasticsearch:

Code: Select all

service elasticsearch restart
This may take your cluster offline temporarily, but when the problem is resolved the nodes should connect with one another. Be sure to perform the above on both nodes, as we want to experience as little database drift as possible.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
2evanowen
Posts: 35
Joined: Fri Jan 16, 2015 1:24 pm

Re: Slave became master and won't go back

Post by 2evanowen »

Yeah that will work for me, email sent and changes were made.
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Slave became master and won't go back

Post by jolson »

Great - ticket has been received, I'll lock this thread up.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
Locked

Return to “Nagios Log Server”