Page 1 of 1

High Load issues

Posted: Mon Dec 19, 2016 5:22 pm
by Fred Kroeger
We have recently deployed another couple of Nagios instance and set them up to feed events via Outbound NRDP to the main server. Even though these are all passive monitors, performance is really suffering as the load average sometimes hits +25. All the performance tuning / RamDisk tips have been deployed.
I thought I had fixed it at one stage as the Load avg dropped right down to about 5. Then realised that npcd wasn't wasn't running. I started that service and the Load avg shot up again.
I looked at perfdataproc.log file during a period of high CPU utilisation and saw multiple entries for the same error.

Code: Select all

PHP Warning:  mysql_pconnect(): Too many connections in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 476
Any ideas what this may be ?
I'm running v5.3.3
regards... Fred

Re: High Load issues

Posted: Mon Dec 19, 2016 5:42 pm
by avandemore
It means it wasn't able to connect to mysql because of MySQL's configuration.

Code: Select all

# mysql -u root -pnagiosxi -e 'SELECT @@GLOBAL.max_connections;'
+--------------------------+
| @@GLOBAL.max_connections |
+--------------------------+
|                      151 |
+--------------------------+
You can set this in your DB configuration file /etc/my.cnf under section [mysqld] and restart it:

Code: Select all

max_connections=250

Re: High Load issues

Posted: Tue Jan 10, 2017 9:27 pm
by Fred Kroeger
Brilliant - thanks for the MySQL Tip. Looking in the my.cnf file I can see that it was set to 100 as part of some performance tuning recommendations.
I've increased it to 200 .
regards... Fred

Re: High Load issues

Posted: Wed Jan 11, 2017 9:59 am
by rkennedy
Awesome - are we good to mark this one resolved?

Re: High Load issues

Posted: Wed Jan 11, 2017 8:15 pm
by Fred Kroeger
Yes mark as resolved but I'm still having high load issues. It's got as bad as Nagios not stopping after I apply a configuration. It appears that it is taking too long for the Nagios process to stop, gives up and then starts a new one. So I have 2 Nagios processes running which creates havoc with modgearmand.
I'm having a closer look at the ESX environment now as my WaitIO is all over the place.

thanks... Fred

Re: High Load issues

Posted: Thu Jan 12, 2017 10:21 am
by rkennedy
It may be due to the IO / sheer number of objects at this point - have you seen this article to increase the time in your init script? https://support.nagios.com/kb/article.php?id=172