Web stuck 'Waiting for Database Startup'

[cdcsysadmin]

Dear support,

The web has been stuck at the same page for several days.
Please advise how to troubleshoot the issue.

Best regards

[cdienger]

Please provide a profile from the system. It can be gathered under Admin > System > System Status > Download System Profile or from the command line with:

Code: Select all

/usr/local/nagioslogserver/scripts/profile.sh

This will create /tmp/system-profile.tar.gz. Please send it to me via a private message.

Note that this file can be very large and may not be able to be uploaded through the system. You can split the file into smaller files with the split command on the NLS(or other Linux machine) command line:

Code: Select all

split -b 5000000 /tmp/system-profile.tar.gz system-profile- -d

The above command will split the system-profile.tar.gz into 5MB segments and save them to files with the naming convention system-profile​-NN.

[cdienger]

Elasticsearch is down but it isn't clear exactly why. The last item logged was from the 8th where it appears to have been overloaded. Is this a node from a cluster of NLS machines? The profile is inconclusive. If you're able to, try increasing the memory on the system to 64GB and then restart elasticsearch:

Code: Select all

systemctl restart elasticsearch

and check the status:

Code: Select all

systemctl status elasticseach -l

Please provide a copy of the /var/log/elasticsearch/631c92d7-4b9d-4d8b-a603-87bbe9ccf75d.log after running the above commands.

[cdcsysadmin]

is there any option besides increasing memory?

[cdienger]

The logs are showing a lot of:

Code: Select all

now throttling indexing: numMergesInFlight=5, maxNumMerges=4

This usually indicates the disks are not fast enough. If this is a VM machine, make sure that VMTools is installed. This may also indicate another VM on the same server is using a lot of the drive's bandwidth. You can also try disabling logstash which will stop some of the writing to the localdisk(make sure to redirct any clients to another NLS machine) and hopefully allow elasticseach to catch up:

Code: Select all

service logstash stop

and check the status of the nagioslogserver index:

Code: Select all

curl 'localhost:9200/_cat/indices?pretty' | grep nagioslogserver

[cdcsysadmin]

'service logstash stop' did not solve the issue.
Is it possible for you to do a remote checking?

[cdienger]

Please open a ticket at support.nagios.com/tickets/ and provide a fresh profile along with the last 6 characters of the license or the customer account ID number.

[cdcsysadmin]

ticket created, #632817

[scottwilkerson]

ticket created, #632817

Locking thread moving to ticket