ndo2db Hogging ALL the CPU

[mikew]

Just to add some more info.

The system I am seeing this on currently has 20 vCPU and 16 GB of RAM.

top reports
ndo2db (now three instances each running from 80-100%
nagios process running at 85%
postmaster 85-100%

No event queue is showing....just blank Gearman is running at about 50% of capacity....meaning that there are 200-300 more workers available over serveral servers. Load is 3-4 which means that there are 16 vCPU doing nothing!

So the issue has something to do with scheduling in my thinking.

[mrochelle]

Mike, I believe at this point your system is in a problem state that will not recover on its own. I would suggest that you stop and restart nagios, wait a minute then ndodb2. Your nagios.cfg option use_retained_scheduling_info=0 to force scheduling to be restarted.
my opinion,
Marcus

[mikew]

Thanks for the input....giving it a whirl and will report back.

[mrochelle]

It might take a few attempts before things stable back out. Marcus

[lmiltchev]

Thanks for the input....giving it a whirl and will report back.

Mike, we will be waiting for your feedback.

[mikew]

No change has ocurred. top shows the server running at 4 CPU but ndo2db is 100% and service checks have a latency of over 2 sec while 16 CPU are doing nothing.

[abrist]

Mike,
This is very odd. Could you open a ticket and attach your profile.zip to the email? I wonder if there are queries failing and hanging ndo. If all is well, you may have to offload mysql and potentially ndo2db to reduce the load.

[mikew]

Update: The ndo2db and MySQL have been moved to another server. Nothing has changed. The 100% load for ndo2db is now on the other server and on the Nagios server 16 CPU(out of 20) are doing nothing and the thing is a dog.,..I have checked every config file and they are all default.

[lmiltchev]

Mike, can you email us the "profile.zip" at xisupport@nagios.com?

[mikew]

Your tech guys have this profile and are working on it. I am just trying to keep up with status to help other who may have this issue to help resolve their issues also. I will keep updating as long as the issue exists.