modgerman worker with high load

[ctrlshawkkey]

I see the load to high on one of the mod german worker server.

top - 17:37:03 up 28 days, 7:02, 1 user, load average: 39.70, 39.40, 39.26
Tasks: 397 total, 41 running, 356 sleeping, 0 stopped, 0 zombie
Cpu(s): 78.4%us, 3.2%sy, 0.0%ni, 12.9%id, 0.0%wa, 0.0%hi, 5.5%si, 0.0%st
Mem: 8058164k total, 2549156k used, 5509008k free, 179716k buffers
Swap: 2064376k total, 0k used, 2064376k free, 431196k cached

VMSTAT

[root@NagiosWorker1 ~]# vmstat 4 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
40 0 0 5450276 179716 431196 0 0 0 0 1 1 78 9 13 0 0
39 0 0 5516784 179716 431196 0 0 0 0 12132 1623 89 11 0 0 0
41 0 0 5498968 179716 431196 0 0 0 3 12815 1669 90 10 0 0 0
39 0 0 5600400 179716 431196 0 0 0 0 13117 1750 91 9 0 0 0
38 0 0 5627012 179716 431196 0 0 0 0 12420 1530 90 10 0 0 0
[root@NagiosWorker1 ~]#

They are lot of jobs in the run queue .

Total processor/core on the system is 16.

top - 17:39:00 up 28 days, 7:04, 1 user, load average: 39.14, 39.40, 39.28
Tasks: 395 total, 38 running, 357 sleeping, 0 stopped, 0 zombie
Cpu0 : 81.6%us, 3.5%sy, 0.0%ni, 14.5%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu1 : 78.6%us, 3.2%sy, 0.0%ni, 12.7%id, 0.0%wa, 0.0%hi, 5.5%si, 0.0%st
Cpu2 : 77.7%us, 3.2%sy, 0.0%ni, 13.2%id, 0.0%wa, 0.0%hi, 5.8%si, 0.0%st
Cpu3 : 79.5%us, 3.0%sy, 0.0%ni, 11.5%id, 0.0%wa, 0.0%hi, 6.1%si, 0.0%st
Cpu4 : 77.5%us, 3.2%sy, 0.0%ni, 13.4%id, 0.0%wa, 0.0%hi, 5.9%si, 0.0%st
Cpu5 : 79.3%us, 3.0%sy, 0.0%ni, 11.7%id, 0.0%wa, 0.0%hi, 6.1%si, 0.0%st


Cpu6 : 77.4%us, 3.3%sy, 0.0%ni, 13.5%id, 0.0%wa, 0.0%hi, 5.8%si, 0.0%st
Cpu7 : 79.0%us, 3.0%sy, 0.0%ni, 11.9%id, 0.0%wa, 0.0%hi, 6.1%si, 0.0%st
Cpu8 : 77.4%us, 3.3%sy, 0.0%ni, 13.6%id, 0.0%wa, 0.0%hi, 5.7%si, 0.0%st
Cpu9 : 79.1%us, 3.0%sy, 0.0%ni, 11.9%id, 0.0%wa, 0.0%hi, 6.1%si, 0.0%st
Cpu10 : 77.1%us, 3.3%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 5.7%si, 0.0%st
Cpu11 : 78.9%us, 3.0%sy, 0.0%ni, 12.0%id, 0.0%wa, 0.0%hi, 6.0%si, 0.0%st
Cpu12 : 77.2%us, 3.3%sy, 0.0%ni, 13.8%id, 0.0%wa, 0.0%hi, 5.6%si, 0.0%st
Cpu13 : 78.8%us, 3.0%sy, 0.0%ni, 12.2%id, 0.0%wa, 0.0%hi, 5.9%si, 0.0%st
Cpu14 : 77.1%us, 3.5%sy, 0.0%ni, 13.8%id, 0.0%wa, 0.0%hi, 5.5%si, 0.0%st
Cpu15 : 78.5%us, 3.1%sy, 0.0%ni, 12.7%id, 0.0%wa, 0.0%hi, 5.8%si, 0.0%st
Mem: 8058164k total, 2672720k used, 5385444k free, 179716k buffers
Swap: 2064376k total, 0k used, 2064376k free, 431196k cached

With 16 cpus and 30 jobs in the run queue , so much high load. Any inputs on this to get the system to minimize the load

Regards,

Siva

[slansing]

When the load hits this high spike can you provide the full output of top?:

Code: Select all

top

I'd like to see what processes are potentially the culprits. Can we get some background on what else is sharing the same hardware as this VM?