in our company we have a Nagios instance serving c.a. 20k hosts and 80k services. Thanks to mod-gearman distributed load and many other large-scale tweaks (ramdisk, db optimizations etc.) it runs quite smoothly (on a machine with 32 CPU cores and 64G of RAM).
However, one thing is creepy... Whenever we start/restart Nagios instance (e.g. when applying the configuration), it takes about 4-5 minutes for Nagios to load our views. Watching database logs meanwhile, I have observed thousands of SQL update queries in that time. Seems that Nagios rebuilds the entire database with "INSERT INTO nagios_servicestatus..." queries for every single service, and that takes so much time. I know that it is rather I/O consuming process, but maybe there is any possibility to get it a bit faster, especially that in fact we have already quite fast disks under laying our virtual machines. Or maybe there is some "tweak" to skip all this database rewriting each time Nagios process is restarted? Seems that this downtime during applying configuration is the main downside of Nagios at all.
Hope for any ideas
