Slowness after offloading the DB

[reincarne]

Hi,
We migrated to nagios DB to a separated DB according to this documentation:
https://assets.nagios.com/downloads/nag ... Server.pdf

However, after the migration instead of fixing some performance issues, we experience issue when applying new configuration - it takes a lot of time, you canno run any command (like reschedule check from a service command) for at least 10 minutes.
The load on the server increased.
I'm talking about 1550 servers and about 27000 service checks.

What could be the reason for such bad performances after offloading the DB?

[tgriep]

There are a lot of reason for the slowdown to happen when the MYSQL database is offloaded to another server.
Some of the things that could affect it are network bandwidth, hard drive speed of the MYSQL server, configuration settings for the MYSQL daemon, etc...

Check the mysql log files on the remote server to see if there are any errors.
You may want to run a repair and optimization on the mysql server to make sure the data is not corrupted, etc..
Also, if you can post the mysql server's configuration files, there could be some settings that could affect the speed.

On the Nagios XI system, you can go through this document to optimize the settings that the XI server uses to connect to the mysql server.
https://assets.nagios.com/downloads/nag ... zation.pdf
On the XI server, if you look at the Apache log file in the /var/log/httpd folder, you might see some errors about connection to the mysql server, post them here.

You may also want to go through these KB articled for troubleshooting this issue.
https://support.nagios.com/kb/article/n ... eeded.html

Having 27000 service checks is very high for the Nagios server and it take a lot of setting / tweaks to run that many checks so one option is to look in to splitting the server in to 2 separate systems.