Nagios XI 2014R2.6 - extremely high load on cpu

[[email protected]]

Version: Nagios XI 2014R2.6
Virtual Machine: 2 CPU/6 Gig
Centos 6
64-bit

I just migrated from Nagios2012 to Nagios XI 2014R2.6. I used the standard virtual machine image, and imported all of my historical information/configuration in this information.

The load average on this server is incredibly high - the load is constantly high and it does not seem to spike (as referenced by one of your FAQs). I have tried all of the normal rebooting, restarting services, etc

top - 10:34:25 up 15:14, 1 user, load average: 50.23, 46.68, 45.88
Tasks: 245 total, 28 running, 217 sleeping, 0 stopped, 0 zombie
Cpu(s): 80.9%us, 19.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 5991496k total, 4817572k used, 1173924k free, 237176k buffers
Swap: 2064380k total, 30548k used, 2033832k free, 3097208k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1355 mysql 20 0 1166m 46m 3812 S 3.4 0.8 55:01.41 mysqld
3740 nagios 20 0 42068 18m 1488 R 2.5 0.3 2:53.52 nagios
6154 nagios 20 0 139m 10m 2144 R 2.2 0.2 0:00.07 check_ifopersta
29699 apache 20 0 440m 29m 4280 R 2.2 0.5 0:00.97 httpd
6095 nagios 20 0 139m 10m 2144 R 1.9 0.2 0:00.06 check_ifopersta
6141 nagios 20 0 139m 10m 2144 R 1.9 0.2 0:00.06 check_ifopersta
6142 nagios 20 0 139m 10m 2144 R 1.9 0.2 0:00.06 check_ifopersta
6607 nagios 20 0 139m 10m 2144 R 1.9 0.2 0:00.06 check_ifopersta
6647 nagios 20 0 138m 9.9m 2140 R 1.9 0.2 0:00.06 check_ifopersta
6651 nagios 20 0 139m 10m 2144 R 1.9 0.2 0:00.06 check_ifopersta
6652 nagios 20 0 138m 9.9m 2144 R 1.9 0.2 0:00.06 check_ifopersta
6653 nagios 20 0 138m 10m 2144 R 1.9 0.2 0:00.06 check_ifopersta
25115 apache 20 0 440m 30m 4376 R 1.9 0.5 0:02.30 httpd
6612 nagios 20 0 133m 8596 2092 R 1.6 0.1 0:00.05 check_ifopersta
6629 nagios 20 0 133m 8652 2092 R 1.6 0.1 0:00.05 check_ifopersta
3750 nagios 20 0 53716 5564 1000 S 1.3 0.1 1:10.87 ndo2db
6657 nagios 20 0 133m 8572 2092 R 1.3 0.1 0:00.04 check_ifopersta

Any help in troubleshooting this issue would be most appreciated.

[abrist]

How many checks are configured (per 5min.)?
You may need to increase system resources.

[[email protected]]

I adjust the system resources to 3 CPU/8 Gig and installed a RAMDisk.


-bash: usr/local/nagios/bin/nagiostats: No such file or directory
[root@spam ~]# /usr/local/nagios/bin/nagiostats -c /usr/local/nagios/etc/nagios.cfg

Nagios Stats 4.0.8
Copyright (c) 2003-2008 Ethan Galstad (www.nagios.org)
Last Modified: 08-12-2014
License: GPL

CURRENT STATUS DATA
------------------------------------------------------
Status File: /var/nagiosramdisk/status.dat
Status File Age: 0d 0h 0m 2s
Status File Version: 4.0.8

Program Running Time: 0d 0h 9m 3s
Nagios PID: 1606

Total Services: 8822
Services Checked: 8822
Services Scheduled: 8822
Services Actively Checked: 8822
Services Passively Checked: 0
Total Service State Change: 0.000 / 21.580 / 0.177 %
Active Service Latency: 0.000 / 0.322 / 0.002 sec
Active Service Execution Time: 0.001 / 30.077 / 1.035 sec
Active Service State Change: 0.000 / 21.580 / 0.177 %
Active Services Last 1/5/15/60 min: 1811 / 8784 / 8820 / 8822
Passive Service Latency: 0.000 / 0.000 / 0.000 sec
Passive Service State Change: 0.000 / 0.000 / 0.000 %
Passive Services Last 1/5/15/60 min: 0 / 0 / 0 / 0
Services Ok/Warn/Unk/Crit: 6717 / 123 / 71 / 1911
Services Flapping: 10
Services In Downtime: 0

Total Hosts: 506
Hosts Checked: 506
Hosts Scheduled: 506
Hosts Actively Checked: 506
Host Passively Checked: 0
Total Host State Change: 0.000 / 49.470 / 0.117 %
Active Host Latency: 0.000 / 0.234 / 0.001 sec
Active Host Execution Time: 0.002 / 10.028 / 0.363 sec
Active Host State Change: 0.000 / 49.470 / 0.117 %
Active Hosts Last 1/5/15/60 min: 129 / 503 / 506 / 506
Passive Host Latency: 0.000 / 0.000 / 0.000 sec
Passive Host State Change: 0.000 / 0.000 / 0.000 %
Passive Hosts Last 1/5/15/60 min: 0 / 0 / 0 / 0
Hosts Up/Down/Unreach: 484 / 22 / 0
Hosts Flapping: 1
Hosts In Downtime: 0

Active Host Checks Last 1/5/15 min: 304 / 1936 / 3371
Scheduled: 214 / 1278 / 2295
On-demand: 90 / 658 / 1076
Parallel: 214 / 1278 / 2295
Serial: 0 / 0 / 0
Cached: 90 / 658 / 1076
Passive Host Checks Last 1/5/15 min: 0 / 0 / 0
Active Service Checks Last 1/5/15 min: 1869 / 8862 / 15776
Scheduled: 1869 / 8862 / 15776
On-demand: 0 / 0 / 0
Cached: 0 / 0 / 0
Passive Service Checks Last 1/5/15 min: 0 / 0 / 0

External Commands Last 1/5/15 min: 0 / 0 / 2


[root@spam ~]#

[jdalrymple]

This is the only thing that's particularly alarming...

Code: Select all

Active Service Execution Time: 0.001 / 30.077 / 1.035 sec

What type of checks are you running that are taking so long? Either way - has adjusting the resources helped the load situation, or did you make those adjustments prior to your earlier post?