NagiosXi databases on restart require repair

[paul.jobb]

Hi;

This weekend we are conducting DR tests between Data centres, I have one of my nagios instances defined to be SRM protected in VMware. After the SRM failover I had to run a database repair on the server, what is the best way to make the Nagiosxi databases more tolerant to failovers, I would like for the server to failover without requiring intervention.

Thanks,
Paul

[rkennedy]

Could you describe the process that is currently occurring for your replication / failover? The main point that I cannot stress enough is to make sure your services are shut down prior to moving the data over, and then starting up cleanly once again.

[avandemore]

Graceful shutdowns should solve the large majority of that type of issue. You can run periodic MySQL integrity checks:

Repairing The Database

Or you could do things like off-load the DB into a fault-tolerant MySQL cluster, but that is well beyond anything we support other than the offloading part.

[paul.jobb]

I guess thats the thing I have noticed when the server is shutdown in an orderly manner (i.e using the shutdown command) quite often, but not always, I will have to run the repair_database.sh command on reboot.

Typically if I know I have to reboot the server I will stop nagios and mysql services ahead of issuing a shutdown. In this instance vmware automation submitted the shutdown command before moving the VM from the Edmonton data centre to the Calgary DC, on fail back the database survived so its not always this happens. I guess since its monitoring it would be nice if it were more resilient to a reboot.

So I am wondering if there is anything I can do on server server startup to check for DB health and repair if needed?

or should I put something in the srm recovery plan that will stop nagios and mysql services ahead of a shutdown?

[avandemore]

The repair scripts utilize myisamchk

This can be altered to hook into mysql/mariadb startup, however that is out of scope. This is our document on the topic:

Repairing The Database

A standard shutdown via reboot(1) or shutdown(1) on RHEL or CentOS should gracefully close the db outside of some strangeness.

A srm recovery plan is also not in scope but generalities do apply. MySQL should always be shutdown gracefully for the best chance of normal startup. This isn't going to help edge cases like FS corruption, but if you're having that type of issue there are bigger problems.

[paul.jobb]

Ok thanks I will look into myisamchk and auto repair. Failing that does it hurt anything , for me if I choose, to schedule a repair_database.sh on system startup whether it needs it or not? I just like the idea of the nagios server being able to survive an unexpected shutdown without having to take corrective action.

Thanks for your help

[avandemore]

No it should not hurt anything. However make sure you have recent backups as always.

[paul.jobb]

great thankyou

[rkennedy]

Are we good to mark this thread as resolved, or do you have further questions?