High Load issues

[Fred Kroeger]

We have recently deployed another couple of Nagios instance and set them up to feed events via Outbound NRDP to the main server. Even though these are all passive monitors, performance is really suffering as the load average sometimes hits +25. All the performance tuning / RamDisk tips have been deployed.
I thought I had fixed it at one stage as the Load avg dropped right down to about 5. Then realised that npcd wasn't wasn't running. I started that service and the Load avg shot up again.
I looked at perfdataproc.log file during a period of high CPU utilisation and saw multiple entries for the same error.

Code: Select all

PHP Warning:  mysql_pconnect(): Too many connections in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 476

Any ideas what this may be ?
I'm running v5.3.3
regards... Fred

[avandemore]

It means it wasn't able to connect to mysql because of MySQL's configuration.

Code: Select all

# mysql -u root -pnagiosxi -e 'SELECT @@GLOBAL.max_connections;'
+--------------------------+
| @@GLOBAL.max_connections |
+--------------------------+
|                      151 |
+--------------------------+

You can set this in your DB configuration file /etc/my.cnf under section [mysqld] and restart it:

Code: Select all

max_connections=250

[Fred Kroeger]

Brilliant - thanks for the MySQL Tip. Looking in the my.cnf file I can see that it was set to 100 as part of some performance tuning recommendations.
I've increased it to 200 .
regards... Fred

[rkennedy]

Awesome - are we good to mark this one resolved?

[Fred Kroeger]

Yes mark as resolved but I'm still having high load issues. It's got as bad as Nagios not stopping after I apply a configuration. It appears that it is taking too long for the Nagios process to stop, gives up and then starts a new one. So I have 2 Nagios processes running which creates havoc with modgearmand.
I'm having a closer look at the ESX environment now as my WaitIO is all over the place.

thanks... Fred

[rkennedy]

It may be due to the IO / sheer number of objects at this point - have you seen this article to increase the time in your init script? https://support.nagios.com/kb/article.php?id=172