Just to add some more info.
The system I am seeing this on currently has 20 vCPU and 16 GB of RAM.
top reports
ndo2db (now three instances each running from 80-100%
nagios process running at 85%
postmaster 85-100%
No event queue is showing....just blank Gearman is running at about 50% of capacity....meaning that there are 200-300 more workers available over serveral servers. Load is 3-4 which means that there are 16 vCPU doing nothing!
So the issue has something to do with scheduling in my thinking.
ndo2db Hogging ALL the CPU
Re: ndo2db Hogging ALL the CPU
Mike Weber
Nagios Training/Consulting
Nagios Training/Consulting
Re: ndo2db Hogging ALL the CPU
Mike, I believe at this point your system is in a problem state that will not recover on its own. I would suggest that you stop and restart nagios, wait a minute then ndodb2. Your nagios.cfg option use_retained_scheduling_info=0 to force scheduling to be restarted.
my opinion,
Marcus
my opinion,
Marcus
Re: ndo2db Hogging ALL the CPU
Thanks for the input....giving it a whirl and will report back.
Mike Weber
Nagios Training/Consulting
Nagios Training/Consulting
Re: ndo2db Hogging ALL the CPU
It might take a few attempts before things stable back out. Marcus
Re: ndo2db Hogging ALL the CPU
Mike, we will be waiting for your feedback.Thanks for the input....giving it a whirl and will report back.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: ndo2db Hogging ALL the CPU
No change has ocurred. top shows the server running at 4 CPU but ndo2db is 100% and service checks have a latency of over 2 sec while 16 CPU are doing nothing.
Mike Weber
Nagios Training/Consulting
Nagios Training/Consulting
Re: ndo2db Hogging ALL the CPU
Mike,
This is very odd. Could you open a ticket and attach your profile.zip to the email? I wonder if there are queries failing and hanging ndo. If all is well, you may have to offload mysql and potentially ndo2db to reduce the load.
This is very odd. Could you open a ticket and attach your profile.zip to the email? I wonder if there are queries failing and hanging ndo. If all is well, you may have to offload mysql and potentially ndo2db to reduce the load.
Former Nagios employee
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
Re: ndo2db Hogging ALL the CPU
Update: The ndo2db and MySQL have been moved to another server. Nothing has changed. The 100% load for ndo2db is now on the other server and on the Nagios server 16 CPU(out of 20) are doing nothing and the thing is a dog.,..I have checked every config file and they are all default.
Mike Weber
Nagios Training/Consulting
Nagios Training/Consulting
Re: ndo2db Hogging ALL the CPU
Mike, can you email us the "profile.zip" at xisupport@nagios.com?
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: ndo2db Hogging ALL the CPU
Your tech guys have this profile and are working on it. I am just trying to keep up with status to help other who may have this issue to help resolve their issues also. I will keep updating as long as the issue exists.
Mike Weber
Nagios Training/Consulting
Nagios Training/Consulting