Nagios Support Forum

Our current Nagios XI environment consists of 103 locations, 2216 host checks and 5725 service checks. We are running 'in the yellow' continuously for server load and I/O wait is often 'in the red' even though we have offloaded MySQL to a separate server and put status.dat and objects.cache in a ramdisk. Adding new hosts has become a painfully slow experience.

I expect to have a 50% increase in the number of locations by the end of the year, and double that (300-350) by the end of next year. Is this normal / expected behavior for XI given the number of hosts/services? Shamas would love to sell me a new server for every 100 locations but that will become cost-prohibitive pretty soon.




103 individual locations

Nagios XI Data
CPU Load 15: 12.42
Total Hosts: 2216
Total Services: 5725

System:
lpnagv01.global.wiie.com 2.6.18-164.9.1.el5 i686
CentOS release 5.4 (Final)
Gnome is not installed
Apache Information
PHP Version: 5.1.6
Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.96 Safari/537.4

info {
created=1351540450
version=3.4.1
last_update_check=1344419434
update_available=1
last_version=3.2.3
new_version=3.4.1
}

What does the XI server's hardware setup look like?

32-bit VM, 4GB RAM, dual core. Sounds anemic but we are not swapping memory.

[root@lpnagv01 check_nagiostats]# free -k
total used free shared buffers cached
Mem: 3115808 2588832 526976 0 243260 1206120
-/+ buffers/cache: 1139452 1976356
Swap: 1048568 68 1048500

top - 16:32:50 up 76 days, 2:23, 1 user, load average: 9.67, 10.50, 11.67
Tasks: 511 total, 8 running, 501 sleeping, 0 stopped, 2 zombie
Cpu(s): 36.5%us, 17.7%sy, 0.0%ni, 39.3%id, 2.6%wa, 1.0%hi, 2.9%si, 0.0%st
Mem: 3115808k total, 2573768k used, 542040k free, 241928k buffers
Swap: 1048568k total, 68k used, 1048500k free, 1205544k cached

[root@lpnagv01 check_nagiostats]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 4
cpu MHz : 2659.999
cache size : 8192 KB

According to the standard Hardware recommendations
http://assets.nagios.com/downloads/nagi ... ements.pdf

To monitor > 500 hosts & > 2500 services (>3000 total) the recommendation is > 4 CPU cores & >8GB RAM

You are monitoring almost 3 times this many with 1/2 as many CPU cores and 1/2 as much RAM...

Additionally, if you have a high IO wait as you describe, you likely also need faster disks to support the checks you have...

upgrade to 64-bit in the works

Utilizing a RAM disk will also go a long way to help the disk IO (if you weren't using it already).
http://assets.nagios.com/downloads/nagi ... p#boosting

migrated to 64 bit VM image with 8GB RAM and a 750MB ramdisk. Load runs pretty much continuously at 10% now. This can be closed.

Holy smokes this was from a while ago, sounds good, closing this as resolved. Just a heads up that 2014 is coming, and should help with performance as well.

Yeah, built server last October and just got time in my sched to migrate. It wasn't the 'day or two' that I estimated, unfortunately, but it is just about done now. Thought you'd want to close outstanding issues