Waiting (forever) for Database Startup

[vmesquita]

I rebooted one of out instances, and now Log Server is trapped in this screen for a long time:
[quote]
Waiting for Database Startup
It looks like your local elasticsearch service is starting.

Why am I getting this error?

Elasticsearch can take a little while to start up because of it's indexing. This may take a few seconds.

The page will refresh automatically after 5 seconds...
[/code]
I checked the logstash.log and found this:

Code: Select all

[2016-08-31 16:23:18,420][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-08-31 16:23:21,640][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-08-31 16:23:24,858][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-08-31 16:23:28,077][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-08-31 16:23:31,296][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-08-31 16:23:31,347][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-08-31 16:23:31,478][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-08-31 16:23:31,482][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] no known master node, scheduling a retry
[2016-08-31 16:23:31,526][DEBUG][action.admin.cluster.state] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2016-08-31 16:23:31,606][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] no known master node, scheduling a retry
[2016-08-31 16:23:34,514][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-08-31 16:23:37,732][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times

Any ideas?

[mcapra]

Are these two machines able to communicate with each other? It looks as if the slave is not able to talk to the master.

Try running an nmap from these machines to each other and share the results. You can also PM them if you have security concerns.

[vmesquita]

I got this result with nmap:

Code: Select all

Starting Nmap 5.51 ( http://nmap.org ) at 2016-08-31 17:12 BRT
Nmap scan report for sb585.selic.bc (172.27.36.109)
Host is up (0.0027s latency).
Not shown: 997 filtered ports
PORT     STATE  SERVICE
80/tcp   open   http
443/tcp  closed https
5544/tcp open   unknown

Nmap done: 1 IP address (1 host up) scanned in 4.54 seconds

Is there any other port in the other node that should be open?

[rkennedy]

Can you also run the following? The error indicates it can't communicate on port 9300.

Code: Select all

nmap 192.168.47.2 -p 9300

[vmesquita]

Output below:

Code: Select all

# nmap 192.168.47.2 -p 9300

Starting Nmap 5.51 ( http://nmap.org ) at 2016-08-31 17:52 BRT
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.10 seconds

[rkennedy]

Output below:

Code: Select all

# nmap 192.168.47.2 -p 9300

Starting Nmap 5.51 ( http://nmap.org ) at 2016-08-31 17:52 BRT
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.10 seconds

Ack, please replace 192.168.47.2 with the IP of your machine. I was doing testing on my end before posting this.

Code: Select all

nmap 172.27.36.109 -p 9300

[vmesquita]

Ok:

Code: Select all

Starting Nmap 5.51 ( http://nmap.org ) at 2016-08-31 17:57 BRT
Nmap scan report for xxxxxxxxxxxx (172.27.36.109)
Host is up (0.0019s latency).
PORT     STATE SERVICE
9300/tcp open  vrace

Nmap done: 1 IP address (1 host up) scanned in 0.11 seconds

[Box293]

What is the output of these commands on both nodes:

Code: Select all

/usr/local/nagioslogserver/logstash/bin/logstash -V
/usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v
cat /var/www/html/nagioslogserver/lsversion
free -m
df -h
df -i
tail -n 50 /var/log/elasticsearch/*.log

[eloyd]

I would add to that list:

Code: Select all

iptables -L -v -n

[mcapra]

Thanks @eloyd!

@vmesquita let us know when you have the items requested.