Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Waiting (forever) for Database Startup

https://support.nagios.com/forum/viewtopic.php?t=40063

Page 1 of 2

Waiting (forever) for Database Startup

Posted: Wed Aug 31, 2016 2:24 pm
by vmesquita
I rebooted one of out instances, and now Log Server is trapped in this screen for a long time:
[quote]
Waiting for Database Startup
It looks like your local elasticsearch service is starting.

Why am I getting this error?

Elasticsearch can take a little while to start up because of it's indexing. This may take a few seconds.

The page will refresh automatically after 5 seconds...
[/code]
I checked the logstash.log and found this:

Code: Select all

[2016-08-31 16:23:18,420][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-08-31 16:23:21,640][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-08-31 16:23:24,858][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-08-31 16:23:28,077][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-08-31 16:23:31,296][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-08-31 16:23:31,347][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-08-31 16:23:31,478][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2016-08-31 16:23:31,482][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] no known master node, scheduling a retry
[2016-08-31 16:23:31,526][DEBUG][action.admin.cluster.state] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2016-08-31 16:23:31,606][DEBUG][action.admin.indices.create] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] no known master node, scheduling a retry
[2016-08-31 16:23:34,514][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
[2016-08-31 16:23:37,732][INFO ][discovery.zen            ] [6a7ce4ea-e1b9-47a1-af18-1c4d47243d20] failed to send join request to master [[636f559a-1fd5-4158-9dee-92e1a8403a1e][GkzwBa-QTHCOKiuoWgmR8Q][localhost][inet[/127.0.0.1:9300]]{max_local_storage_nodes=1}], reason [RemoteTransportException[[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][inet[/127.0.0.1:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}] not master for join request from [[6a7ce4ea-e1b9-47a1-af18-1c4d47243d20][0XjhXD2VRp6K3k8Eq4UDlQ][sa585][inet[/172.27.164.109:9300]]{max_local_storage_nodes=1}]]; ], tried [3] times
Any ideas?

Re: Waiting (forever) for Database Startup

Posted: Wed Aug 31, 2016 3:05 pm
by mcapra
Are these two machines able to communicate with each other? It looks as if the slave is not able to talk to the master.

Try running an nmap from these machines to each other and share the results. You can also PM them if you have security concerns.

Re: Waiting (forever) for Database Startup

Posted: Wed Aug 31, 2016 3:13 pm
by vmesquita
I got this result with nmap:

Code: Select all

Starting Nmap 5.51 ( http://nmap.org ) at 2016-08-31 17:12 BRT
Nmap scan report for sb585.selic.bc (172.27.36.109)
Host is up (0.0027s latency).
Not shown: 997 filtered ports
PORT     STATE  SERVICE
80/tcp   open   http
443/tcp  closed https
5544/tcp open   unknown

Nmap done: 1 IP address (1 host up) scanned in 4.54 seconds
Is there any other port in the other node that should be open?

Re: Waiting (forever) for Database Startup

Posted: Wed Aug 31, 2016 3:40 pm
by rkennedy
Can you also run the following? The error indicates it can't communicate on port 9300.

Code: Select all

nmap 192.168.47.2 -p 9300

Re: Waiting (forever) for Database Startup

Posted: Wed Aug 31, 2016 3:53 pm
by vmesquita
Output below:

Code: Select all

# nmap 192.168.47.2 -p 9300

Starting Nmap 5.51 ( http://nmap.org ) at 2016-08-31 17:52 BRT
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.10 seconds

Re: Waiting (forever) for Database Startup

Posted: Wed Aug 31, 2016 3:55 pm
by rkennedy
vmesquita wrote:Output below:

Code: Select all

# nmap 192.168.47.2 -p 9300

Starting Nmap 5.51 ( http://nmap.org ) at 2016-08-31 17:52 BRT
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.10 seconds
Ack, please replace 192.168.47.2 with the IP of your machine. I was doing testing on my end before posting this.

Code: Select all

nmap 172.27.36.109 -p 9300

Re: Waiting (forever) for Database Startup

Posted: Wed Aug 31, 2016 3:57 pm
by vmesquita
Ok:

Code: Select all

Starting Nmap 5.51 ( http://nmap.org ) at 2016-08-31 17:57 BRT
Nmap scan report for xxxxxxxxxxxx (172.27.36.109)
Host is up (0.0019s latency).
PORT     STATE SERVICE
9300/tcp open  vrace

Nmap done: 1 IP address (1 host up) scanned in 0.11 seconds

Re: Waiting (forever) for Database Startup

Posted: Wed Aug 31, 2016 6:43 pm
by Box293
What is the output of these commands on both nodes:

Code: Select all

/usr/local/nagioslogserver/logstash/bin/logstash -V
/usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v
cat /var/www/html/nagioslogserver/lsversion
free -m
df -h
df -i
tail -n 50 /var/log/elasticsearch/*.log

Re: Waiting (forever) for Database Startup

Posted: Thu Sep 01, 2016 8:24 am
by eloyd
I would add to that list:

Code: Select all

iptables -L -v -n

Re: Waiting (forever) for Database Startup

Posted: Thu Sep 01, 2016 11:02 am
by mcapra
Thanks @eloyd!

@vmesquita let us know when you have the items requested.
All times are UTC-05:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited