Waiting (forever) for Database Startup

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
vmesquita
Posts: 315
Joined: Fri Aug 10, 2012 12:52 pm

Re: Waiting (forever) for Database Startup

Post by vmesquita »

Thanks guys, here's the results:
Node 1:

Code: Select all

[root@xxxxxxxx ~]# /usr/local/nagioslogserver/logstash/bin/logstash -V
logstash 1.5.1
[root@xxxxxxxx ~]# /usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v
Version: 1.6.0, Build: cdd3ac4/2015-06-09T13:36:34Z, JVM: 1.7.0_85
[root@xxxxxxxx ~]# cat /var/www/html/nagioslogserver/lsversion
VERSION=142
[root@xxxxxxxx ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          7854       5339       2514          0        151        533
-/+ buffers/cache:       4654       3200
Swap:         4095          0       4095
[root@xxxxxxxx ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      912G  256G  610G  30% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/sda1             477M   64M  388M  15% /boot
//yyyyyyyy/logserver      50G   50G  4.0K 100% /var/backup
[root@xxxxxxxx ~]# df -i
Filesystem             Inodes  IUsed    IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
                     60727296 119118 60608178    1% /
tmpfs                 1005388      1  1005387    1% /dev/shm
/dev/sda1              128016     46   127970    1% /boot
//yyyyyyy/logserver           0      0        0     - /var/backup
[root@xxxxxxxx ~]# tail -n 50 /var/log/elasticsearch/*.log
==> /var/log/elasticsearch/155bf3a6-eb50-41aa-8f41-112a4db21fc6_index_indexing_slowlog.log <==

==> /var/log/elasticsearch/155bf3a6-eb50-41aa-8f41-112a4db21fc6_index_search_slowlog.log <==

==> /var/log/elasticsearch/155bf3a6-eb50-41aa-8f41-112a4db21fc6.log <==
[2016-09-01 13:41:15,261][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:18,475][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:21,690][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:24,904][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:28,119][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:31,334][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:31,340][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-09-01 13:41:31,444][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-09-01 13:41:31,448][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] no known master node, scheduling a retry
[2016-09-01 13:41:31,477][DEBUG][action.admin.cluster.state] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2016-09-01 13:41:31,558][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] no known master node, scheduling a retry
[2016-09-01 13:41:34,548][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:37,763][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:40,978][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:44,192][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:47,407][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:50,621][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:53,836][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:41:57,051][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:00,266][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:01,585][DEBUG][action.admin.cluster.state] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] no known master node, scheduling a retry
[2016-09-01 13:42:03,480][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:06,695][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:09,910][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:13,124][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:16,339][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:19,553][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:22,768][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:25,981][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:29,195][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:31,449][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-09-01 13:42:31,559][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-09-01 13:42:31,562][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] no known master node, scheduling a retry
[2016-09-01 13:42:31,586][DEBUG][action.admin.cluster.state] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2016-09-01 13:42:31,664][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] no known master node, scheduling a retry
[2016-09-01 13:42:32,410][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:35,625][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:38,839][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:42,054][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:45,269][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:48,483][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:51,698][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:54,913][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:42:58,127][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:43:01,342][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:43:01,700][DEBUG][action.admin.cluster.state] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] no known master node, scheduling a retry
[2016-09-01 13:43:04,573][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:43:07,789][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:43:11,004][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-09-01 13:43:14,219][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][xxxxxxxx][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times

==> /var/log/elasticsearch/f5b14e01-68c8-4e4a-95ef-86407bfcf472_index_indexing_slowlog.log <==

==> /var/log/elasticsearch/f5b14e01-68c8-4e4a-95ef-86407bfcf472_index_search_slowlog.log <==

==> /var/log/elasticsearch/f5b14e01-68c8-4e4a-95ef-86407bfcf472.log <==
[root@xxxxxxxx ~]# iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
Node 2:

Code: Select all

[root@zzzzzzzzzz ~]# /usr/local/nagioslogserver/logstash/bin/logstash -V
logstash 1.5.1
[root@zzzzzzzzzz ~]# /usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v
Version: 1.6.0, Build: cdd3ac4/2015-06-09T13:36:34Z, JVM: 1.7.0_85
[root@zzzzzzzzzz ~]# cat /var/www/html/nagioslogserver/lsversion
VERSION=142
[root@zzzzzzzzzz ~]# free -m
             total       used       free     shared    buffers     cached
Mem:         15936      10823       5113          0        194       1230
-/+ buffers/cache:       9397       6538
Swap:         4095          0       4095
[root@zzzzzzzzzz ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      912G  259G  608G  30% /
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/sda1             477M   65M  388M  15% /boot
//yyyyyyyyy/logserver      50G   50G  4.0K 100% /var/backup
[root@zzzzzzzzzz ~]# df -i
Filesystem             Inodes  IUsed    IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
                     60727296 119751 60607545    1% /
tmpfs                 2039879      1  2039878    1% /dev/shm
/dev/sda1              128016     46   127970    1% /boot
//yyyyyyyy/logserver           0      0        0     - /var/backup
[root@zzzzzzzzzz ~]# tail -n 50 /var/log/elasticsearch/*.log
==> /var/log/elasticsearch/155bf3a6-eb50-41aa-8f41-112a4db21fc6_index_indexing_slowlog.log <==

==> /var/log/elasticsearch/155bf3a6-eb50-41aa-8f41-112a4db21fc6_index_search_slowlog.log <==

==> /var/log/elasticsearch/155bf3a6-eb50-41aa-8f41-112a4db21fc6.log <==
[root@zzzzzzzzzz ~]# iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Waiting (forever) for Database Startup

Post by mcapra »

My best guess is that one of the nodes was disconnected from the cluster and elected itself master. This will make achieving consensus difficult if both nodes think they are the master. You will likely need to restart all the nodes on the cluster to make sure the correct master is elected.
Former Nagios employee
https://www.mcapra.com/
Top
vmesquita
Posts: 315
Joined: Fri Aug 10, 2012 12:52 pm

Re: Waiting (forever) for Database Startup

Post by vmesquita »

Excellent. Rebooted both servers at the same time, and it's back working.
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Waiting (forever) for Database Startup

Post by mcapra »

Awesome! Is it alright if we lock this thread and mark the issue as resolved?
Former Nagios employee
https://www.mcapra.com/
Top
vmesquita
Posts: 315
Joined: Fri Aug 10, 2012 12:52 pm

Re: Waiting (forever) for Database Startup

Post by vmesquita »

Yes, feel free to lock it.
Top
Locked

Return to “Nagios Log Server”