ndo2db Hogging ALL the CPU

[BanditBBS]

Code: Select all

0 9 * * *       /usr/bin/php /usr/local/nagiosxi/html/includes/components/scheduledreporting/sendreport.php --report=2lroug --username=andy > /dev/null 2>&1
0 8 * * *       /usr/bin/php /usr/local/nagiosxi/html/includes/components/scheduledreporting/sendreport.php --report=cgca5j --username=scott > /dev/null 2>&1
0 9 * * *       /usr/bin/php /usr/local/nagiosxi/html/includes/components/scheduledreporting/sendreport.php --report=c78f7d --username=trevor > /dev/null 2>&1
0 18 * * *      /usr/bin/php /usr/local/nagiosxi/html/includes/components/scheduledreporting/sendreport.php --report=r2qm5t --username=ethan > /dev/null 2>&1
32 16 * * *     /usr/bin/php /usr/local/nagiosxi/html/includes/components/scheduledreporting/sendreport.php --report=riu6p0 --username=ethan > /dev/null 2>&1

Code: Select all

https://blah/nagiosxi/includes/components/xicore/status.php?show=services&sortby=&sortorder=asc&host=&hostgroup=&servicegroup=&hoststatustypes=2&servicestatustypes=28&hostattr=10&serviceattr=10&search=&page=1&records=250&pagemove=Go
Top Alert Producers Report
Notifications Report

The last two were just added yesterday and are just pdf attachments of 2 different dashbaords.

It's 9:06am today and so far no spike since yesterday. The last one ended around 2:30pm. I love diagnosing a sporadic issue!

EDIT: 9:10AM, load just spiked to 40 and bouncing all over! I've been watching a top for the last hour, nothing seems different at all. I'm still leaning towards a usage issue in http.

[abrist]

I'm still leaning towards a usage issue in http.

Could you expand on this? Could it be everyone logging in at the same time, etc?

EDIT: Has anyone tried the current master branch version of ndo in a test environment?
https://github.com/NagiosEnterprises/nd ... its/master

[BanditBBS]

I'm still leaning towards a usage issue in http.

Could you expand on this? Could it be everyone logging in at the same time, etc?

EDIT: Has anyone tried the current master branch version of ndo in a test environment?
https://github.com/NagiosEnterprises/nd ... its/master

Yeah Andy, I'm imaging perhaps its a bunch of my users logging in and bringing up dashboards and running reports and stuff like that. Some of the dashboards are pretty darn busy too, attaching one for example that might be up on multiple computers at one customer.

Capture.PNG

As far as trying the ndo, I have not. I don't have this issue on my dev server so I wouldn't really be able to test anything except "it functions".

[abrist]

I wish there was a good way to separate apache from the core box. This would allow load balancing for the frontend, and combined with offloaded dbs, minimal impact on the nagios core server resources.

[scottwilkerson]

Yeah Andy, I'm imaging perhaps its a bunch of my users logging in and bringing up dashboards and running reports and stuff like that. Some of the dashboards are pretty darn busy too, attaching one for example that might be up on multiple computers at one customer.

Shot in the dark, but if you have a ton of users loading stuff at the same time, mysql (even on your offloaded server) could be hitting max_connections

Try the following - open the "my.cnf" (on mysql server) file in a text editor:

Code: Select all

vi /etc/my.cnf

and add the following line under "user=mysql":

Code: Select all

set-variable=max_connections=500

Restart mysql:

Code: Select all

service mysqld restart

Default is normally 50.

What I have seen in the past is if many connection are attempted, all of the other connections will have to wait for one to be available. This will hold up httpd processes or ndo processes etc...

[BanditBBS]

Yeah Andy, I wish!

Scott, done and no change. CPU on mysql/ndo server is still 0 and nagios server is still in the middle of a spike and fluctuating between 20 and 200.

[mrochelle]

Abrist, I don't mind trying the current master branch version of ndo if I could get some instructions unique for installation on nagiosxi 2014R20.

[scottwilkerson]

Abrist, I don't mind trying the current master branch version of ndo if I could get some instructions unique for installation on nagiosxi 2014R20.

Easiest way is to do the following:

Code: Select all

cd /tmp
wget https://github.com/NagiosEnterprises/ndoutils/archive/master.zip
cd ndoutils-master
./configure
make
make install-init
cp -f src/ndomod-4x.o /usr/local/nagios/bin/ndomod.o
cp -f src/ndo2db-4x /usr/local/nagios/bin/ndo2db
cp -f src/file2sock /usr/local/nagios/bin/
cp -f src/log2ndo /usr/local/nagios/bin/
cp -f src/sockdebug /usr/local/nagios/bin/
echo "table_trim_interval=60" >> /usr/local/nagios/etc/ndo2db.cfg
service ndo2db stop
service nagios stop
service ndo2db start
service nagios start

[mrochelle]

Thanks Scott. I'm scheduled to install in test Monday and if things go well there, I going to place it on our problem child discussed prior in this topic.

[slansing]

Awesome, keep us all updated! Thanks!