Offloading Nagios XI DB

[jbennett]

I am wanting to start pushing to make this happen for our install. Since our install is very large, I'm wanting to make sure I'm not going to run into anything unforseen by my eyes.

We have close to 1000 hosts and nearly 19000 service checks. I am utilizing NRDS on some of the hosts that are passing the majority of the service checks.

Will the network traffic between the Nagios box and the SQL server box be too much with the majority of the NRDS checks being run on a 3 minute interval? I understand that this is dependant on the network here. I'm wondering if this raises any flags otherwise.

[abrist]

I understand that this is dependant on the network here. I'm wondering if this raises any flags otherwise.

That would be the key question. As you are already using nrds, the inbound web requests would not change, just the outbound queries to the db server. I presume this would probably be a wash, though you will still see the load benefits of offloading.

As always, I suggest this is implemented in a test environment first, before deploying to production.

[tonyyarusso]

I haven't worked with things as big as your setup, but I've yet to see anything come remotely close to saturating a gigabit network connection before the SQL server choked somewhere else, so I'm inclined to say you'll be fine. If that turns out to not be the case, you can always trunk a couple of ethernet connections together into one logical pipe, or have multiple SQL servers in a replication setup and balance the traffic among them.

[abrist]

I can't imagine that XI SQL queries would require a cluster, but I have yet to lay my eyes on a 50k+ check install. Bonding is a great option when you hit the throughput wall, though I would agree with Tony that it would be quite impressive if your offloaded server needed bonded interfaces.

[jbennett]

I am trying this on a test VM environment currently but I'm stuck.

Per the documentation:

d) Rollover Datases
Now its time to move the data in the local Nagios MySQL database to the remote MySQL database. This task will probably require
downtime as it is inadvisable to copy an active database. Once you feel it is safe to copy the databases over, use the following
command. Keep in mind, this command is being run the Nagios server, so the -u and -p on the mysqldump are associated with the
mysql database on the Nagios server, while the mysql side credentials are associated with the remote MySQL server. The mysql dump
can take quite a while if the Nagios XI server has been running for some time.

Code: Select all

service nagios stop
service ndo2db stop
mysqldump -u root -p'nagiosxi' nagios | mysql -u nagios -p'nagios' -h 192.168.5.88 nagios
mysqldump -u root -p'nagiosxi' nagiosql | mysql -u nagiosql -p'nagiosql' -h 192.168.5.88 nagiosql

But I'm not certain I understand which IP this should be using? loopback? The IP of the Nagios machine?

[sreinhardt]

The ip's at the end of the command should be the remote mysql machine that you now have running, however the commands are run from the nagiosxi machine with the current full database. Hope that clears it up some!

[jbennett]

Thanks.

For my own knowledge, I'm assuming this is then telling nagios to place the dump on the remote MySQL server in the correct location?

[sreinhardt]

You are correct, it just dumps the current db from the nagios server, logs into the remote mysql instance, and stores all the dumped information there.

[jbennett]

Thanks. That worked great.

Now that I have this older snapshot of our install offloaded to a remote database, I would like to import a backup from our currently active Nagios XI instance.

However, being that I have a ton of NRDS service checks, I'm not sure if it would be even remotely accurate. Since those checks are all pointed to our current install, how would this work? For reference, we have about 17000 passive NRDS checks currently.

I'm really hoping to test the load, but I'm wondering how that would be possible in the given situation.

[sreinhardt]

An interesting dilemma, I believe that you could forward the checks from the current XI instance to the remote instance. Your test system will still have to actively process mysql queries as normal and performance data. Importing your current database into the remote setup should allow all of those checks to be imported and work similar to how they are now.