XI: high CPU load while users are adding nodes to environmen

[inversecow]

Ahoy SUPP folks,

FYI, my users are presently loading our new Nagios XI PROD environment with their servers for monitoring.
While this activity is going on, we see a tremendous hit to our XI server during the business day (CPU: spikes to load average in the 14.xx - 20.xx range).

After a point XI seems to "stagger" under this load, where it starts to report DB Errors to the user's browser sessions (dB node itself looks fine).
Eventually the dB errors go away, once the CPU load reduces (assume this is about when ndo2db finishes).
We see a large quantity of `http` processes in top also, but expect this is related to the API calls being made (EG: via curl or the equivalent for our Windows users)

Also of note, after hours (when the users are not adding nodes), the CPU load average is more respectable (~1.x or so).

Is there any tuning we might do with our XI node to mitigate the impact of XI adding nodes for monitoring?
Is it just a matter of throwing more vCPUs at the node to "clear the hump" while we add nodes?
Is there a better way that is less impactful to the XI node?

I understand the (simplified) process is as such:

API call to XI to add a node for monitoring (EG: https://nagios.fqdn.tld/nagiosxi/api/v1 ... ?ARGS_HERE )
XI:CCM creates record, writes record to related MariadB (off-box in our scenario)
XI:CCM "dumps" dB to local FS "object" files, which is what Nagios CORE understands
XI:CCM re-generates permissions mapping, along with other associations within these object files

Our ENV:

# XI (XI 5.6.9)
virtual machine
RHEL 7.x
4 vCPU
32 GB MEM
lots of disk

# dB (mariadb 5.5.64-1, off-box)
virtual machine
RHEL 7.x
4 vCPU
32 GB MEM
lots of disk

Thank you,
-Rowan

[Box293]

How many host and service objects currently exist? Admin > System Config > System Profile > View System Info has a section that displays total hosts and services.

If you are going through adding a lot of objects continually you can hold off doing an apply config each time. Then every so often do an apply config.

Do you have the server configured for a ram disk?

https://support.nagios.com/kb/article.php?id=288