Waiting for Database Startup

Description

Nagios Log Server is a useful way to search and store large amounts of data however the start up process is resource intensive and may take some time to complete. If your instance of Log Server presents the message "Waiting for Database Startup" for more than a few minutes you may want to investigate the problem further.

First Verify The Process Is Running

First you will want to confirm the elasticsearch (2024R1) or opensearch (2024R2+) process is running as expected. Login to your Nagios Log Server instance via the console or SSH and run the following command as root:

Log Server 2024R1

systemctl status elasticsearch

You should see output similar to the block below:

● elasticsearch.service - LSB: This service manages the elasticsearch daemon
     Loaded: loaded (/etc/rc.d/init.d/elasticsearch; generated)
     Active: active (running) since Tue 2025-01-28 14:28:34 CST; 20h ago
       Docs: man:systemd-sysv-generator(8)
    Process: 967 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
      Tasks: 59 (limit: 10883)
     Memory: 1.1G
        CPU: 26min 7.966s
     CGroup: /system.slice/elasticsearch.service
             └─1180 /bin/java -Xms870m -Xmx870m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -

Log Server 2024R2+

systemctl status opensearch

You should see output similar to the block below:

● opensearch.service - OpenSearch
Loaded: loaded (/usr/lib/systemd/system/opensearch.service; enabled; preset: disabled)
Active: active (running) since Fri 2025-08-22 17:12:48 CDT; 4 days ago
Docs: https://www.elastic.co
Process: 1981 ExecStartPost=php /var/www/html/nagioslogserver/www/index.php cli CLI_SynchronizeOpensearchConfiguration run>
Main PID: 1023 (java)
Tasks: 126 (limit: 48676)
Memory: 4.2G (peak: 5.3G)
CPU: 3h 46min 30.692s
CGroup: /system.slice/opensearch.service
└─1023 /usr/local/nagioslogserver/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 ->

The "Active:" line in the output should read "active (running)" as in the example above. If it does not read active try restarting the process by running the following command as root on the command line of your server:

Log Server 2024R1

systemctl restart elasticsearch

Log Server 2024R2+

systemctl restart opensearch

Then after a few minutes check again to see if the process is in a running state. If it is not you will want to contact the Nagios Support team to help investigate the problem.

Nagios Support Resources

If The Process Is Running:

If the process is in a running state and the "Waiting for Database Startup" is still present it may indicate that you have a system that is under resourced. You will want to use the tools and documentation for your operating system to make a determination as to where the restriction maybe however you may want to try the following if possible:

1) Increase the amount of RAM available to the system up to 64Gb
2) Increase the number of processors available to the system ( Minimum 4 )
3) Ensure that your your iowait times are low ( below 5% ).

If you are using a virtual system you will also want to make sure your hypervisor isn't over provisioned and has enough resources for all virtual systems at peak times.

Finally if you are at your resource limits for your current host or cluster you might want to consider adding another node to your Log Server cluster.

Adding Additional Instances To A Nagios Log Server Cluster - 2024R1

Adding Additional Instances To A Nagios Log Server Cluster - 2024R2+

Final Thoughts

Additional considerations and/or tuning recommendations may be available on our community forums. We encourage and value community contributions/feedback that help us to make our products better.

https://support.nagios.com/forum/