Fresh Install - Waiting for Database Startup

[chris83]

I did a fresh install on a new CentOS 7.5 DigitalOcean Droplet. Used the automatic install script and installation was finished successfull. I then went on to the webinterface and entered the details for the admin account. When I clicked on proceed the browser was stuck for more than an hour waiting for the server. I gave up and opened a new browser session. Log server is now just stuck on waiting for database startup.

It seems to me elasticsearch is starting as it is supposed to:

Code: Select all

max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2018-09-27 17:16:48,803][INFO ][http                     ] [d70db179-d12e-4112-a1ef-5bcb0c7fc44a] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[localhost/127.0.0.1:9200]}
[2018-09-27 17:16:48,803][INFO ][node                     ] [d70db179-d12e-4112-a1ef-5bcb0c7fc44a] started
[2018-09-27 17:16:48,808][INFO ][gateway                  ] [d70db179-d12e-4112-a1ef-5bcb0c7fc44a] recovered [0] indices into cluster_state
[2018-09-27 18:10:31,943][INFO ][node                     ] [d70db179-d12e-4112-a1ef-5bcb0c7fc44a] version[1.7.6], pid[28387], build[c730b59/2016-11-18T15:21:16Z]
[2018-09-27 18:10:31,944][INFO ][node                     ] [d70db179-d12e-4112-a1ef-5bcb0c7fc44a] initializing ...
[2018-09-27 18:10:32,194][INFO ][plugins                  ] [d70db179-d12e-4112-a1ef-5bcb0c7fc44a] loaded [knapsack-1.7.3.0-d0ea246], sites []
[2018-09-27 18:10:32,335][INFO ][env                      ] [d70db179-d12e-4112-a1ef-5bcb0c7fc44a] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [22.1gb], net total_space [24.9gb], types [rootfs]

I´ve tried restart the service and then rebooting the server without any success. Any suggestions?

[scottwilkerson]

One thing I can think of would be is localhost defined in /etc/hosts

Code: Select all

cat /etc/hosts

If not this could prevent the Web UI from being able to see the server as it uses localhost instead of 127.0.0.1 directly.

[chris83]

Thanks for your suggestion. This is how my /etc/host file is looking:

Code: Select all

cat /etc/hosts
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.redhat.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
# The following lines are desirable for IPv4 capable hosts
127.0.0.1 nagios-syslog-server-2gb nagios-syslog-server-2gb
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4

# The following lines are desirable for IPv6 capable hosts
::1 nagios-syslog-server-2gb nagios-syslog-server-2gb
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

I did try another fresh install. It´s a 2GB 50GB SSD DigitalOcean CentOS 7.5. Only thing I have done is to run yum update and then the automatic install script. This server was working for about an hour or so before the exact same issue occured. Restarting the apache and elasticsearch service it runs for a couple of minutes before the "waiting for database" problem happens again.

I did notice this in the Apache logs:

httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2018-09-27 22:15:54 CEST; 8h ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 3590 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
Main PID: 3657 (httpd)
Status: "Total requests: 4017; Current requests/sec: 0.2; Current traffic: 1.2KB/sec"
CGroup: /system.slice/httpd.service
├─ 3657 /usr/sbin/httpd -DFOREGROUND
├─16280 /usr/sbin/httpd -DFOREGROUND
├─16281 /usr/sbin/httpd -DFOREGROUND
├─16282 /usr/sbin/httpd -DFOREGROUND
├─27629 /usr/sbin/httpd -DFOREGROUND
├─28910 /usr/sbin/httpd -DFOREGROUND
├─29766 /usr/sbin/httpd -DFOREGROUND
├─29912 /usr/sbin/httpd -DFOREGROUND
├─30165 /usr/sbin/httpd -DFOREGROUND
├─30406 /usr/sbin/httpd -DFOREGROUND
└─30651 /usr/sbin/httpd -DFOREGROUND

Sep 28 06:23:38 nagios-syslog-server-2gb sudo[20466]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Sep 28 06:23:43 nagios-syslog-server-2gb sudo[20482]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Sep 28 06:23:49 nagios-syslog-server-2gb sudo[20525]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Sep 28 06:23:54 nagios-syslog-server-2gb sudo[20540]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Sep 28 06:23:59 nagios-syslog-server-2gb sudo[20557]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Sep 28 06:24:05 nagios-syslog-server-2gb sudo[20611]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Sep 28 06:24:10 nagios-syslog-server-2gb sudo[20628]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Sep 28 06:24:16 nagios-syslog-server-2gb sudo[20646]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Sep 28 06:24:21 nagios-syslog-server-2gb sudo[20691]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Sep 28 06:24:26 nagios-syslog-server-2gb sudo[20707]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status

[cdienger]

Are there any errors or warning logged in the /var/log/elasticsearch/<uuid>.log when the database goes unavailable?

Feel free to PM us a profile as well which can be generated under Admin > System > System Status or generated on the UI by running:

/usr/local/nagioslogserver/scripts/profile.sh

The profile can be found at /tmp/system-profile.tar.gz.

[chris83]

Thanks for helping out. I only see INFO and DEBUG information in /var/log/elasticsearch/*.log. Nothing flagged as ERROR or WARNING.

I'm sending you a PM with the system profile.

I see the PM I've sent is stuck in the outbox and not being moved to the sent folder. I'm new to this forum and would have expected the message to end up in the sent folder. Please report if you have not received the message.

[cdienger]

We received it. The elasticsearch process is crashing and it looks like it was down at the time the profile was taken so some details are missing, but I would suspect there may be a problem with the size of the indices and the available memory. Elasticsearch can only use half of the total system memory and in this case that would only be about 1gig. I would try increasing the memory to 4 or 8gig if possible.

[chris83]

Thanks for the suggestion. I have resized the VM to 4GB RAM and 2 CPU´s. Hopefully this resolves our issues.

I went with the 2GB droplet since it was within the minimum requirements and we currently only have 1 host sending syslog to this server for evaluation purposes.

[cdienger]

Yeah, it normally doesn't take long for an install to quickly outgrow the 2gig minimum but it is a bit odd with just the single host. Keep an eye on it though and feel free to send us a fresh profile if needed.

[chris83]

The server has been running stable for 5 days now since upgrading the hardware so I think it is safe to say we have found the solution to the issue.

[cdienger]

Glad to hear!