Page 2 of 5

Re: CPU Load spike every 7 hours

Posted: Wed Jun 18, 2014 11:31 am
by tmcdonald
Well it's good that it has stabilized at least. We did reproduce this so we can probably take it from here, but if you have any other details of course we'd be interested to hear from you again.

Re: CPU Load spike every 7 hours

Posted: Thu Jun 19, 2014 10:24 am
by SDK
We have the same problem in our environment since the upgrade to 2014 R1.1, only the interval is different

I have tried to track the problem down, but was unsuccessful in doing so.

The only thing i saw was an increased sys usage of the OS and increased overall procs.

To fix this problem is to restart the nagios core and the load
goes back to normal (~ 4 to 5)

The Vmware CPU Stats in VCenter show no difference in CPU usage when the load is that high so my guess is, the load isn't caused because of CPU usage

Here are some screenshots:

Load:
Load.png
CPU Stats:
CPU Stats.png
Total processes:
Procs.png
Any ideas?

Re: CPU Load spike every 7 hours

Posted: Thu Jun 19, 2014 10:48 am
by slansing
SDK, what kind of hardware do you have dedicated to this VM? CPU's, core's/cpu, memory, what type of disks/drives. Also, is this 32-bit or 64? And RHEL or CENTOS? What version of the distro?

Re: CPU Load spike every 7 hours

Posted: Thu Jun 19, 2014 11:45 am
by SDK
slansing wrote:SDK, what kind of hardware do you have dedicated to this VM? CPU's, core's/cpu, memory, what type of disks/drives. Also, is this 32-bit or 64? And RHEL or CENTOS? What version of the distro?
Hi slansing,

it is a 4 vCPU 4 GB RAM CentOS 6.4 32Bit virtual machine. The storage array behind it is really powerful so that can't be the issue (as seen in the screenshots the I/O waits are very low).

I must have overlooked that it could be reproduced so i think it is a general problem not related to our environment/hardware sizing etc. Besides the spikes the sever runs just fine.

Kind regards

SDK

Re: CPU Load spike every 7 hours

Posted: Fri Jun 20, 2014 10:03 am
by lmiltchev
Can you run the following commands during a load spike and show us the output in code wraps?

Code: Select all

top
ps -aef
ps aux --sort -rss

Re: CPU Load spike every 7 hours

Posted: Fri Jun 27, 2014 1:41 pm
by BanditBBS
Have we gotten anywhere with this? I just spun up a new 2014r1.2 server and check this out:
Nag04LOAD.PNG
All this server is doing is monitoring itself and one other Nagios server.

Re: CPU Load spike every 7 hours

Posted: Fri Jun 27, 2014 2:09 pm
by slansing
Bandit, on this fresh server what are it's specs:

Code: Select all

Distro/Arch/Version:
Memory:
CPU(cores):
Average I/O(disk type):

Re: CPU Load spike every 7 hours

Posted: Fri Jun 27, 2014 2:15 pm
by BanditBBS
Distro/Arch/Version:Linux iss-chi-nag04.net.itciss.com 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May 10 15:42:40 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.1 (Santiago)
Memory:32GB
CPU(cores):8
Average I/O(disk type):0.98 avg IO Wait and External fiber connected netapp san.

I know its not at 100% used since it has 8 cores, but for 2 hours now it has been high and doing nothing.

Re: CPU Load spike every 7 hours

Posted: Sun Jun 29, 2014 1:20 pm
by mikew
Just to add support to what has been said before, my test server, CentOS 6.4 64_bit with 32 Core and 8 GB of RAM is doing exactly the same thing, every 7 hours.

Re: CPU Load spike every 7 hours

Posted: Mon Jun 30, 2014 1:16 pm
by lmiltchev
Thanks, Mike! I will keep this thread open so that other users can hopefully provide more info on the issue, that can help us troubleshoot it.