Scheduled SSH and Local Back Ups Failing

[NVESMM]

We have a high availability set up for XI with a prod, backup, and test instance. I can verify that on all three instances, there is enough free disk space.

We seem to be getting crashed database notifications randomly on all three and then scheduled local backups failing because of crashed database tables per /usr/local/nagiosxi/var/components/scheduledbackups.log with various errors such as:
Info:Database nagios = {Table nagios_contactnotifications has a problem: Table ./nagios/nagios_contactnotifications is marked as crashed and last (automatic?) repair failed}
Info:Database nagios = {Table nagios_commenthistory has a problem: Table ./nagios/nagios_commenthistory is marked as crashed and last (automatic?) repair failed}

A couple days this week we have received various XI alerts (for check MySQL Table Status - nagios DB service checks we have set up) on all three instances. I have run a repair and force repair as listed here (https://answerhub.nagios.com/support/s/ ... s-8f5b7a37), and it will be fine for a couple of days but then we start to see backup failures again.

Since we seemingly are getting errors so often, do you have any suggestions?

[jsimon]

Hi @NVESMM,

Can you advise what Linux distro and release you're using? I would recommend taking a look at log files within

Code: Select all

/var/log/mysql

If the issue appears to not be disk space, I'm wondering how often these servers are shut down and what method is used to do that. If the issue is affecting three separate machines, it seems possible that it could be due to process rather than hardware.