Waiting for database startup

[NMFSTeam]

We are running Nagios Log Server 2.1.10 on Ubuntu 20.04 LTS. For the past few days, we have received the error "Waiting for database startup." Per the support article https://support.nagios.com/kb/article.php?id=550, we rebooted the VM, and that seemed to resolve the issue. However, it has happened twice now in three days. Anything we can do to prevent this?

Thank you.

[pbroste]

Hello @NMFSTeam

Thanks for reaching out and want to get the System Profile on this so we can see what is going on.

Code: Select all

/usr/local/nagioslogserver/scripts/profile.sh

The System Profile is found in '/tmp/system-profile.tar.gz'

To size down log server profile by removing extra logstash:

Code: Select all

cd /tmp/
gzip -d system-profile.tar.gz -c | tar --delete --wildcards system-profile/logstashlogs/logstash.log-*.gz | gzip - > /tmp/tmp.$$.tar.gz && mv /tmp/tmp.$$.tar.gz system-profile.tar.gz

Thanks,
Perry

p.s.

We're moving to a new support system!

The Nagios Answer Hub is a place where you can get help with technical questions from our experts. There, you can quickly open tickets and join discussion boards.

Request Nagios Answer Hub access here: https://info.nagios.com/answer-hub-access-new-users

After completing the access form, you will be given access to a portal where new tickets can be created. We will keep the old customer forum sections and ticket system available for current cases to be resolved.

[NMFSTeam]

I am having trouble running this command, because there is only 1GiB of space on /tmp, and it is running out of space when I run it. Any way to output it to somewhere else? I have tons of space on the root file system / .

Thanks for your help.

[pbroste]

Hello @NMFSTeam

Thanks for following up, option to mount to a disk with space.

Code: Select all

df -h
fdisk -l
mkdir /mnt/mountpoint
mount /dev/sdx# /mnt/mountpoint

Update the 'basedir="/tmp/"' found in the '/usr/local/nagioslogserver/scripts/profile.sh' script to the '/mnt/mountpoint/'

I did not test this, meaning I am not sure if this script is writing directly to the 'basedir' or if it caching it somewhere before it does. Please let us know.

Thanks,
Perry

[NMFSTeam]

I got a system profile generated, it is just under 500MB now. What shall I do with it?

Thanks!

[pbroste]

Hello @NMFSTeam

Yeah that is pretty large size, did you try to size it down?

Code: Select all

cd /tmp/
gzip -d system-profile.tar.gz -c | tar --delete --wildcards system-profile/logstashlogs/logstash.log-*.gz | gzip - > /tmp/tmp.$$.tar.gz && mv /tmp/tmp.$$.tar.gz system-profile.tar.gz

Even the split command will not get it down to size.

Have you signed up for the new Answer Hub? This would be a good time to do so, this will allow you to upload the System Profile.

The Nagios Answer Hub is a place where you can get help with technical questions from our experts. There, you can quickly open tickets and join discussion boards.

Request Nagios Answer Hub access here: https://info.nagios.com/answer-hub-access-new-users

After completing the access form, you will be given access to a portal where new tickets can be created. We will keep the old customer forum sections and ticket system available for current cases to be resolved.

Thanks,
Perry

[NMFSTeam]

Yes, I have already signed up for the Nagios Answer Hub. Should I go to the discussion forums there? Perhaps under NLS Troubleshooting?

Code: Select all

root@hqnaglogi1:/home/john# /usr/local/nagioslogserver/scripts/profile.sh
system-profile/
system-profile/logstashlogs/
system-profile/logstashlogs/logstash.log.5.gz
system-profile/logstashlogs/logstash.log.2.gz
system-profile/logstashlogs/logstash.log.7.gz
system-profile/logstashlogs/logstash.log.3.gz
system-profile/logstashlogs/logstash.log
system-profile/logstashlogs/logstash.log.4.gz
system-profile/logstashlogs/logstash.log.1.gz
system-profile/logstashlogs/logstash.log.6.gz
system-profile/elasticsearchlog/
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff_index_indexing_slowlog.log
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.4.gz
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.6.gz
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.5.gz
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff_index_search_slowlog.log
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.3.gz
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.1.gz
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.7.gz
system-profile/elasticsearchlog/c2f33fb7-081b-4317-abd1-643e2a8a2aff.log.2.gz
system-profile/lsversion.txt
system-profile/network.txt
system-profile/pending-tasks.txt
system-profile/shard-health.txt
system-profile/psaux.txt
system-profile/masters.txt
system-profile/logstash-confd/
system-profile/logstash-confd/999_outputs.conf
system-profile/logstash-confd/000_inputs.conf
system-profile/logstash-confd/998_live_outputs.conf
system-profile/logstash-confd/500_filters.conf
system-profile/logstash-confd/501_live_filters.conf
system-profile/iptables.txt
system-profile/recovery.txt
system-profile/memory.txt
system-profile/indices-status.txt
system-profile/ufw.txt
system-profile/netstat.txt
system-profile/logstash-plugins.txt
system-profile/systemlog.txt
system-profile/jvm-status.txt
system-profile/disk.txt
system-profile/timestamp.txt
system-profile/plugins.txt
system-profile/shard-status.txt
system-profile/cluster.txt
system-profile/apachelogs/
system-profile/apachelogs/access.log
system-profile/apachelogs/other_vhosts_access.log
system-profile/apachelogs/error.log
system-profile/store.txt
system-profile/nodes.txt
root@hqnaglogi1:/home/john# gzip -d system-profile.tar.gz -c | tar --delete --wildcards system-profile/logstashlogs/logstash.log-*.gz | gzip - > /home/john/tmp.$$.tar.gz && mv /home/john/tmp.$$.tar.gz system-profile.tar.gz
tar: system-profile/logstashlogs/logstash.log-*.gz: Not found in archive
tar: Exiting with failure status due to previous errors

[pbroste]

Hello @NMFSTeam

Thanks for following up, and signing up for our new Answer Hub. The Answer Hub is our new support ticketing system. Please login into your Answer Hub account and open a new support ticket and link back to this one.

Without the system profile, I did some digging on the issue that you are experiencing, and sounds like the issue is around syncing elasticsearch data between nodes master, and slave. Depending on the number of nodes you have appears that you can have up to two master nodes. The config found in:

/usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml:

node.master: true

Again without the System Profile, it appears that we can go ahead and clear the elasticsearch data on the slave(s) so it rebuilds.

In my case, I moved the folders just in case something went wonky and restarted the services and see that the data folders are getting replaced.

Code: Select all

mv /usr/local/nagioslogserver/elasticsearch/data /tmp/
systemctl restart logstash elasticsearch httpd

Let me know how things are looking.

Thanks,
Perry

[NMFSTeam]

We only have one node in total, so master and slave are all on one box.

Our /data directory has 643GB in it, nowhere I can really put that, unless I mount up another 1TB disk. Should I do that?

I created a case on the Nagios Answer Hub, and uploaded the system profile.

Thank you very much for your assistance, it is most appreciated.

[pbroste]

Excellent, @NMFSTeam will take a look at the System Profile in Answer Hub and follow up with you shortly.

Will go ahead and lock this and continue to communicate via Answer Hub.

Thanks,
Perry