CPU Load spike every 7 hours

[tmcdonald]

Well it's good that it has stabilized at least. We did reproduce this so we can probably take it from here, but if you have any other details of course we'd be interested to hear from you again.

[SDK]

We have the same problem in our environment since the upgrade to 2014 R1.1, only the interval is different

I have tried to track the problem down, but was unsuccessful in doing so.

The only thing i saw was an increased sys usage of the OS and increased overall procs.

To fix this problem is to restart the nagios core and the load
goes back to normal (~ 4 to 5)

The Vmware CPU Stats in VCenter show no difference in CPU usage when the load is that high so my guess is, the load isn't caused because of CPU usage

Here are some screenshots:

Load:

Load.png

CPU Stats:

CPU Stats.png

Total processes:

Procs.png

Any ideas?

[slansing]

SDK, what kind of hardware do you have dedicated to this VM? CPU's, core's/cpu, memory, what type of disks/drives. Also, is this 32-bit or 64? And RHEL or CENTOS? What version of the distro?

[SDK]

SDK, what kind of hardware do you have dedicated to this VM? CPU's, core's/cpu, memory, what type of disks/drives. Also, is this 32-bit or 64? And RHEL or CENTOS? What version of the distro?

Hi slansing,

it is a 4 vCPU 4 GB RAM CentOS 6.4 32Bit virtual machine. The storage array behind it is really powerful so that can't be the issue (as seen in the screenshots the I/O waits are very low).

I must have overlooked that it could be reproduced so i think it is a general problem not related to our environment/hardware sizing etc. Besides the spikes the sever runs just fine.

Kind regards

SDK

[lmiltchev]

Can you run the following commands during a load spike and show us the output in code wraps?

Code: Select all

top
ps -aef
ps aux --sort -rss

[BanditBBS]

Have we gotten anywhere with this? I just spun up a new 2014r1.2 server and check this out:

Nag04LOAD.PNG

All this server is doing is monitoring itself and one other Nagios server.

[slansing]

Bandit, on this fresh server what are it's specs:

Code: Select all

Distro/Arch/Version:
Memory:
CPU(cores):
Average I/O(disk type):

[BanditBBS]

Distro/Arch/Version:Linux iss-chi-nag04.net.itciss.com 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May 10 15:42:40 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.1 (Santiago)
Memory:32GB
CPU(cores):8
Average I/O(disk type):0.98 avg IO Wait and External fiber connected netapp san.

I know its not at 100% used since it has 8 cores, but for 2 hours now it has been high and doing nothing.

[mikew]

Just to add support to what has been said before, my test server, CentOS 6.4 64_bit with 32 Core and 8 GB of RAM is doing exactly the same thing, every 7 hours.

[lmiltchev]

Thanks, Mike! I will keep this thread open so that other users can hopefully provide more info on the issue, that can help us troubleshoot it.