Page 1 of 2

Extremely high CPU usage after upgrade

Posted: Thu Oct 01, 2015 11:34 am
by mbowden
I upgraded to the latest Nagiosxi yesterday and noticed a pretty big delay when applying configuration. This morning the machine continued to be taxed with top readings of 100+ I was then presented with the following screen;
Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh

SQL: SQL Error [nagiosxi] : Database connection failed SQL: SQL Error [nagiosxi] : Database connection failed SQL: SQL Error [nagiosxi] : Database connection failedMessage: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh

I ran the command and the box is so CPU bound at this point its useless. The machine functioned appropriately prior to the upgrade, no additional monitors have been added.

RHEL 5.7

Re: Extremely high CPU usage after upgrade

Posted: Thu Oct 01, 2015 11:39 am
by tmcdonald
What version were you running prior to the upgrade?

If you're still able to use the interface, go to Configure -> Core Config Manager, then on the left under Tools click Write Config Files. Then click Delete, then Write, then Verify and post a screenshot.

Re: Extremely high CPU usage after upgrade

Posted: Thu Oct 01, 2015 11:57 am
by mbowden
I will get your other screen shot momentarily, I thought I would share this one first

Re: Extremely high CPU usage after upgrade

Posted: Thu Oct 01, 2015 11:58 am
by mbowden
And the find processes still keep coming

Re: Extremely high CPU usage after upgrade

Posted: Thu Oct 01, 2015 12:02 pm
by mbowden
requested screen shot, 16 find processes running now

Re: Extremely high CPU usage after upgrade

Posted: Thu Oct 01, 2015 12:06 pm
by mbowden
ok last find screenshot

Re: Extremely high CPU usage after upgrade

Posted: Thu Oct 01, 2015 12:39 pm
by tgriep
Can you run the following in a shell and post the output so we can see what the find command is running?

Code: Select all

ps aux --cols=300 |grep find

Re: Extremely high CPU usage after upgrade

Posted: Thu Oct 01, 2015 3:30 pm
by mbowden
I apologize about the delay in responding, The box basically continued spinning up find processes until it crashed. I rebooted and a minute or 2 later there was a single find process, then another..

Re: Extremely high CPU usage after upgrade

Posted: Thu Oct 01, 2015 3:31 pm
by mbowden
wrong screen shot

Re: Extremely high CPU usage after upgrade

Posted: Thu Oct 01, 2015 3:47 pm
by mbowden
The perfdata directory its looking at is 32G not sure if that is relevant or not

We don't necessarily care about the historical perfdata if that is the issue, we could dump it and start collecting data moving forward.