We had a consultant in last week to work on system performance, listed is apply times we were seeing. The mysql database has been offloaded to its own vm, about half of our service checks run at 5 min intervals with the rest running every 15 - 60 minutes. Attached the system and monitoring status screen shots. My coworker here Jason may have spoken to you about this last week as well.
system_status.PNG
monitoring_engine_perf.PNG
applyconfig timings:
failure - 7m 41s
baseline - 7m 28 s
after db repair - 7m 33s
lowered php.ini 512mb ->256Mb - 7m 34s
after vmware tools install - 7m 9s
with nagios shutdown - 5m 43
after ramdisk changes - 6m 48s (both status.dat and perfdata spool writing to ramdisks)
after sysctl tuning and upgrade - 8m 47s (upgrade to version 1.3, increased values for sysctl)
after 4 vCPU -> 2 vCPU - 7m 29s - moved server back to 2 vcpu's, VM cpu scheduling in our environment has a hugely negative impact on nagios i.e. high latency
Outside of the four duplicate definitions the rest of the warnings are for no check time period.
Warning: Duplicate definition found for service 'CPU Usage' on host 'VNN1170' (config file '/usr/local/nagios/etc/services/windows-servers-ds.cfg', starting on line 65)
Warning: Duplicate definition found for service 'Memory Usage' on host 'VNN1170' (config file '/usr/local/nagios/etc/services/windows-servers-ds.cfg', starting on line 114)
Warning: Duplicate definition found for service 'Page File Usage' on host 'VNN1170' (config file '/usr/local/nagios/etc/services/windows-servers-ds.cfg', starting on line 128)
Warning: Duplicate definition found for service 'System Uptime' on host 'VNN1170' (config file '/usr/local/nagios/etc/services/windows-servers-ds.cfg', starting on line 144)
......
Warning: Service 'Memory Usage' on host 'vnn995' has no check time period defined!
Warning: Service 'Page File Usage' on host 'vnn995' has no check time period defined!
Warning: Service 'System Uptime' on host 'vnn995' has no check time period defined!
Checked 13021 services.
Checking hosts...
Checked 1694 hosts.
Checking host groups...
Checked 36 host groups.
Checking service groups...
Checked 2 service groups.
Checking contacts...
Checked 54 contacts.
Checking contact groups...
Checked 7 contact groups.
Checking service escalations...
Checked 0 service escalations.
Checking service dependencies...
Checked 0 service dependencies.
Checking host escalations...
Checked 0 host escalations.
Checking host dependencies...
Checked 0 host dependencies.
Checking commands...
Checked 128 commands.
Checking time periods...
Checked 44 time periods.
Checking for circular paths between hosts...
Checking for circular host and service dependencies...
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...
Total Warnings: 7385
Total Errors: 0