z

Commercial Support Clients: Clients with support contracts can get escalated support assistance by visiting Nagios Answer Hub. These forums are for community support services. Although we at Nagios try our best to help out on the forums here, we always give priority support to our support clients.

Waiting for database startup

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.

Waiting for database startup

Postby NMFSTeam » Wed Mar 09, 2022 10:43 pm

We are running Nagios Log Server 2.1.10 on Ubuntu 20.04 LTS. For the past few days, we have received the error "Waiting for database startup." Per the support article https://support.nagios.com/kb/article.php?id=550, we rebooted the VM, and that seemed to resolve the issue. However, it has happened twice now in three days. Anything we can do to prevent this?

Thank you.
NMFSTeam
 
Posts: 75
Joined: Thu Nov 12, 2015 9:01 am

Re: Waiting for database startup

Postby pbroste » Wed Mar 16, 2022 1:43 pm

Hello @NMFSTeam

Thanks for reaching out and want to get the System Profile on this so we can see what is going on.

Code: Select all
/usr/local/nagioslogserver/scripts/profile.sh


The System Profile is found in '/tmp/system-profile.tar.gz'

To size down log server profile by removing extra logstash:

Code: Select all
cd /tmp/
gzip -d system-profile.tar.gz -c | tar --delete --wildcards system-profile/logstashlogs/logstash.log-*.gz | gzip - > /tmp/tmp.$$.tar.gz && mv /tmp/tmp.$$.tar.gz system-profile.tar.gz


Thanks,
Perry

p.s.
We're moving to a new support system!

The Nagios Answer Hub is a place where you can get help with technical questions from our experts. There, you can quickly open tickets and join discussion boards.

Request Nagios Answer Hub access here: https://info.nagios.com/answer-hub-access-new-users

After completing the access form, you will be given access to a portal where new tickets can be created. We will keep the old customer forum sections and ticket system available for current cases to be resolved.
User avatar
pbroste
 
Posts: 1287
Joined: Tue Jun 01, 2021 1:27 pm

Re: Waiting for database startup

Postby NMFSTeam » Fri Mar 18, 2022 4:22 pm

I am having trouble running this command, because there is only 1GiB of space on /tmp, and it is running out of space when I run it. Any way to output it to somewhere else? I have tons of space on the root file system / .

Thanks for your help.
NMFSTeam
 
Posts: 75
Joined: Thu Nov 12, 2015 9:01 am

Re: Waiting for database startup

Postby pbroste » Tue Mar 22, 2022 5:19 pm

Hello @NMFSTeam

Thanks for following up, option to mount to a disk with space.

Code: Select all
df -h
fdisk -l
mkdir /mnt/mountpoint
mount /dev/sdx# /mnt/mountpoint


Update the 'basedir="/tmp/"' found in the '/usr/local/nagioslogserver/scripts/profile.sh' script to the '/mnt/mountpoint/'

I did not test this, meaning I am not sure if this script is writing directly to the 'basedir' or if it caching it somewhere before it does. Please let us know.

Thanks,
Perry
User avatar
pbroste
 
Posts: 1287
Joined: Tue Jun 01, 2021 1:27 pm

Re: Waiting for database startup

Postby NMFSTeam » Tue Mar 29, 2022 10:35 am

I got a system profile generated, it is just under 500MB now. What shall I do with it?

Thanks!
NMFSTeam
 
Posts: 75
Joined: Thu Nov 12, 2015 9:01 am

Re: Waiting for database startup

Postby pbroste » Tue Mar 29, 2022 12:32 pm

Hello @NMFSTeam

Yeah that is pretty large size, did you try to size it down?

Code: Select all
cd /tmp/
gzip -d system-profile.tar.gz -c | tar --delete --wildcards system-profile/logstashlogs/logstash.log-*.gz | gzip - > /tmp/tmp.$$.tar.gz && mv /tmp/tmp.$$.tar.gz system-profile.tar.gz


Even the split command will not get it down to size.

Have you signed up for the new Answer Hub? This would be a good time to do so, this will allow you to upload the System Profile.

The Nagios Answer Hub is a place where you can get help with technical questions from our experts. There, you can quickly open tickets and join discussion boards.

Request Nagios Answer Hub access here: https://info.nagios.com/answer-hub-access-new-users

After completing the access form, you will be given access to a portal where new tickets can be created. We will keep the old customer forum sections and ticket system available for current cases to be resolved.


Thanks,
Perry
User avatar
pbroste
 
Posts: 1287
Joined: Tue Jun 01, 2021 1:27 pm

Re: Waiting for database startup

Postby NMFSTeam » Tue Mar 29, 2022 6:39 pm

Yes, I have already signed up for the Nagios Answer Hub. Should I go to the discussion forums there? Perhaps under NLS Troubleshooting?

Code: Select all
root@hqnaglogi1:/home/john# /usr/local/nagioslogserver/scripts/profile.sh
system-profile/
system-profile/logstashlogs/
system-profile/logstashlogs/logstash.log.5.gz
system-profile/logstashlogs/logstash.log.2.gz
system-profile/logstashlogs/logstash.log.7.gz
system-profile/logstashlogs/logstash.log.3.gz
system-profile/logstashlogs/logstash.log
system-profile/logstashlogs/logstash.log.4.gz
system-profile/logstashlogs/logstash.log.1.gz
system-profile/logstashlogs/logstash.log.6.gz
system-profile/elasticsearchlog/
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff_index_indexing_slowlog.log
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.4.gz
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.6.gz
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.5.gz
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff_index_search_slowlog.log
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.3.gz
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.1.gz
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.7.gz
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.2.gz
system-profile/lsversion.txt
system-profile/network.txt
system-profile/pending-tasks.txt
system-profile/shard-health.txt
system-profile/psaux.txt
system-profile/masters.txt
system-profile/logstash-confd/
system-profile/logstash-confd/999_outputs.conf
system-profile/logstash-confd/000_inputs.conf
system-profile/logstash-confd/998_live_outputs.conf
system-profile/logstash-confd/500_filters.conf
system-profile/logstash-confd/501_live_filters.conf
system-profile/iptables.txt
system-profile/recovery.txt
system-profile/memory.txt
system-profile/indices-status.txt
system-profile/ufw.txt
system-profile/netstat.txt
system-profile/logstash-plugins.txt
system-profile/systemlog.txt
system-profile/jvm-status.txt
system-profile/disk.txt
system-profile/timestamp.txt
system-profile/plugins.txt
system-profile/shard-status.txt
system-profile/cluster.txt
system-profile/apachelogs/
system-profile/apachelogs/access.log
system-profile/apachelogs/other_vhosts_access.log
system-profile/apachelogs/error.log
system-profile/store.txt
system-profile/nodes.txt
root@hqnaglogi1:/home/john# gzip -d system-profile.tar.gz -c | tar --delete --wildcards system-profile/logstashlogs/logstash.log-*.gz | gzip - > /home/john/tmp.$$.tar.gz && mv /home/john/tmp.$$.tar.gz system-profile.tar.gz
tar: system-profile/logstashlogs/logstash.log-*.gz: Not found in archive
tar: Exiting with failure status due to previous errors
NMFSTeam
 
Posts: 75
Joined: Thu Nov 12, 2015 9:01 am

Re: Waiting for database startup

Postby pbroste » Wed Mar 30, 2022 2:40 pm

Hello @NMFSTeam

Thanks for following up, and signing up for our new Answer Hub. The Answer Hub is our new support ticketing system. Please login into your Answer Hub account and open a new support ticket and link back to this one.

Without the system profile, I did some digging on the issue that you are experiencing, and sounds like the issue is around syncing elasticsearch data between nodes master, and slave. Depending on the number of nodes you have appears that you can have up to two master nodes. The config found in:

/usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml:

node.master: true


Again without the System Profile, it appears that we can go ahead and clear the elasticsearch data on the slave(s) so it rebuilds.

In my case, I moved the folders just in case something went wonky and restarted the services and see that the data folders are getting replaced.

Code: Select all
mv /usr/local/nagioslogserver/elasticsearch/data /tmp/
systemctl restart logstash elasticsearch httpd


Let me know how things are looking.

Thanks,
Perry
User avatar
pbroste
 
Posts: 1287
Joined: Tue Jun 01, 2021 1:27 pm

Re: Waiting for database startup

Postby NMFSTeam » Thu Mar 31, 2022 2:43 pm

We only have one node in total, so master and slave are all on one box.

Our /data directory has 643GB in it, nowhere I can really put that, unless I mount up another 1TB disk. Should I do that?

I created a case on the Nagios Answer Hub, and uploaded the system profile.

Thank you very much for your assistance, it is most appreciated.
NMFSTeam
 
Posts: 75
Joined: Thu Nov 12, 2015 9:01 am

Re: Waiting for database startup

Postby pbroste » Thu Mar 31, 2022 4:25 pm

Excellent, @NMFSTeam will take a look at the System Profile in Answer Hub and follow up with you shortly.

Will go ahead and lock this and continue to communicate via Answer Hub.

Thanks,
Perry
User avatar
pbroste
 
Posts: 1287
Joined: Tue Jun 01, 2021 1:27 pm


Return to Nagios Log Server

Who is online

Users browsing this forum: No registered users and 3 guests