host / service limit for 32-bit Nagios XI

[pnewlon]

Our current Nagios XI environment consists of 103 locations, 2216 host checks and 5725 service checks. We are running 'in the yellow' continuously for server load and I/O wait is often 'in the red' even though we have offloaded MySQL to a separate server and put status.dat and objects.cache in a ramdisk. Adding new hosts has become a painfully slow experience.

I expect to have a 50% increase in the number of locations by the end of the year, and double that (300-350) by the end of next year. Is this normal / expected behavior for XI given the number of hosts/services? Shamas would love to sell me a new server for every 100 locations but that will become cost-prohibitive pretty soon.




103 individual locations

Nagios XI Data
CPU Load 15: 12.42
Total Hosts: 2216
Total Services: 5725

System:
lpnagv01.global.wiie.com 2.6.18-164.9.1.el5 i686
CentOS release 5.4 (Final)
Gnome is not installed
Apache Information
PHP Version: 5.1.6
Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.96 Safari/537.4

info {
created=1351540450
version=3.4.1
last_update_check=1344419434
update_available=1
last_version=3.2.3
new_version=3.4.1
}

[slansing]

What does the XI server's hardware setup look like?

[pnewlon]

32-bit VM, 4GB RAM, dual core. Sounds anemic but we are not swapping memory.

[root@lpnagv01 check_nagiostats]# free -k
total used free shared buffers cached
Mem: 3115808 2588832 526976 0 243260 1206120
-/+ buffers/cache: 1139452 1976356
Swap: 1048568 68 1048500

top - 16:32:50 up 76 days, 2:23, 1 user, load average: 9.67, 10.50, 11.67
Tasks: 511 total, 8 running, 501 sleeping, 0 stopped, 2 zombie
Cpu(s): 36.5%us, 17.7%sy, 0.0%ni, 39.3%id, 2.6%wa, 1.0%hi, 2.9%si, 0.0%st
Mem: 3115808k total, 2573768k used, 542040k free, 241928k buffers
Swap: 1048568k total, 68k used, 1048500k free, 1205544k cached

[root@lpnagv01 check_nagiostats]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 4
cpu MHz : 2659.999
cache size : 8192 KB

[scottwilkerson]

According to the standard Hardware recommendations
http://assets.nagios.com/downloads/nagi ... ements.pdf

To monitor > 500 hosts & > 2500 services (>3000 total) the recommendation is > 4 CPU cores & >8GB RAM

You are monitoring almost 3 times this many with 1/2 as many CPU cores and 1/2 as much RAM...

Additionally, if you have a high IO wait as you describe, you likely also need faster disks to support the checks you have...

[pnewlon]

upgrade to 64-bit in the works

[mguthrie]

Utilizing a RAM disk will also go a long way to help the disk IO (if you weren't using it already).
http://assets.nagios.com/downloads/nagi ... p#boosting

[pnewlon]

migrated to 64 bit VM image with 8GB RAM and a 750MB ramdisk. Load runs pretty much continuously at 10% now. This can be closed.

[slansing]

Holy smokes this was from a while ago, sounds good, closing this as resolved. Just a heads up that 2014 is coming, and should help with performance as well.

[pnewlon]

Yeah, built server last October and just got time in my sched to migrate. It wasn't the 'day or two' that I estimated, unfortunately, but it is just about done now. Thought you'd want to close outstanding issues