Page 1 of 1
Web stuck 'Waiting for Database Startup'
Posted: Thu Dec 10, 2020 1:38 am
by cdcsysadmin
Dear support,
The web has been stuck at the same page for several days.
Please advise how to troubleshoot the issue.
Best regards
Re: Web stuck 'Waiting for Database Startup'
Posted: Thu Dec 10, 2020 12:29 pm
by cdienger
Please provide a profile from the system. It can be gathered under Admin > System > System Status > Download System Profile or from the command line with:
Code: Select all
/usr/local/nagioslogserver/scripts/profile.sh
This will create /tmp/system-profile.tar.gz. Please send it to me via a private message.
Note that this file can be very large and may not be able to be uploaded through the system. You can split the file into smaller files with the split command on the NLS(or other Linux machine) command line:
Code: Select all
split -b 5000000 /tmp/system-profile.tar.gz system-profile- -d
The above command will split the system-profile.tar.gz into 5MB segments and save them to files with the naming convention system-profile-NN.
Re: Web stuck 'Waiting for Database Startup'
Posted: Fri Dec 11, 2020 6:03 pm
by cdienger
Elasticsearch is down but it isn't clear exactly why. The last item logged was from the 8th where it appears to have been overloaded. Is this a node from a cluster of NLS machines? The profile is inconclusive. If you're able to, try increasing the memory on the system to 64GB and then restart elasticsearch:
and check the status:
Please provide a copy of the /var/log/elasticsearch/631c92d7-4b9d-4d8b-a603-87bbe9ccf75d.log after running the above commands.
Re: Web stuck 'Waiting for Database Startup'
Posted: Mon Dec 14, 2020 3:04 am
by cdcsysadmin
is there any option besides increasing memory?
Re: Web stuck 'Waiting for Database Startup'
Posted: Mon Dec 14, 2020 4:24 pm
by cdienger
The logs are showing a lot of:
Code: Select all
now throttling indexing: numMergesInFlight=5, maxNumMerges=4
This usually indicates the disks are not fast enough. If this is a VM machine, make sure that VMTools is installed. This may also indicate another VM on the same server is using a lot of the drive's bandwidth. You can also try disabling logstash which will stop some of the writing to the localdisk(make sure to redirct any clients to another NLS machine) and hopefully allow elasticseach to catch up:
and check the status of the nagioslogserver index:
Code: Select all
curl 'localhost:9200/_cat/indices?pretty' | grep nagioslogserver
Re: Web stuck 'Waiting for Database Startup'
Posted: Tue Dec 15, 2020 1:00 am
by cdcsysadmin
'service logstash stop' did not solve the issue.
Is it possible for you to do a remote checking?
Re: Web stuck 'Waiting for Database Startup'
Posted: Tue Dec 15, 2020 4:24 pm
by cdienger
Please open a ticket at support.nagios.com/tickets/ and provide a fresh profile along with the last 6 characters of the license or the customer account ID number.
Re: Web stuck 'Waiting for Database Startup'
Posted: Thu Dec 17, 2020 8:18 pm
by cdcsysadmin
ticket created, #632817
Re: Web stuck 'Waiting for Database Startup'
Posted: Fri Dec 18, 2020 8:46 am
by scottwilkerson
cdcsysadmin wrote:ticket created, #632817
Locking thread moving to ticket