Nagios Load very high

[jdunitz]

According to your output, you have the default number of connections (100) and you're using them all plus one (101). Increasing your connection limit to 1000 as previously suggested should help with your problem.

See: Nagios XI - MySQL/MariaDB - Max Connections

Let us know if this does it for you.

--Jeffrey

[lgaddam]

Hi Jeff,

To increase max connections , please provide me steps. Im not able to get actual steps from the provided link.
and also let me know whether i have to do repairdatabases.sh script after doing modification or reboot is required or any services need to be started to get affect.

[jdunitz]

Was your Max_used_connections output from before or after you changed the connection limit and rebooted?
If it was after, it doesn't seem to have taken effect for some reason, and we'd want to figure out what went wrong there.

If it was before, please apply the changes and reboot and see how it's doing.

[lmiltchev]

To increase max connections , please provide me steps. Im not able to get actual steps from the provided link.

Have you modified the my.cnf file by adding the following lines in the section [mysqld]?

Code: Select all

max_connections=1000
open_files_limit = 4096

Can you show us what you have in that file?

and also let me know whether i have to do repairdatabases.sh script after doing modification or reboot is required or any services need to be started to get affect.

Running the repair on the database won't hurt anything, but you don't have to do it, unless you have crashed tables/errors in the mysqld.log (or mariadb.log).

[lgaddam]

Thanks I have done the changes in /etc/my.cnf & restarted mysqld as per procedure provided by below link.

https://support.nagios.com/kb/article.php?id=513

Now its showing the max connections as 1000.

[root@nagiosp01 httpd]# mysql -uroot -pnagiosxi -e "show variables like 'max_connections';"
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 1000 |
+-----------------+-------+
You have new mail in /var/spool/mail/root
[root@glnagiosp01 httpd]#

[lmiltchev]

Great! You can reboot now, and see if your server will perform better now.

If the high load is back, PM us your latest profile, so that we can review your logs. Thanks!

[lgaddam]

Hi,

Sorry this didn't helped, we identified the issue three times after doing this change, high load & high process count observed.

We observed this issue today when we do apply config as well as when we didn't do apply config & also observed monitoring engine flapping sometimes.

But main issue is that, when load is high, bulk of node down alerts are generating, not sure why these false alerts are generating, its creating panic to IT teams. We have to fix this ASAP.

After running dbrepair.sh script the load came down within 20 minutes. Don't know is it the right way to do to decrease load but I am doing this everytime we find high load, not sure everytime if we run this script whether any other complications will be araised ?

And also I observed that when we run this script, Nagios process more taking time to stop.

But we haven't rebooted the server, this is physical server , so Linux team not recommending as of now. Is really required a reboot ?
If require reboot then we need to provide proper justification because same hosts(3646) & services(14780) are also present in secondary and did not find any load issues on that.

Please find the screenshot of the load graph, process count graph and also profile.zip attached.

Moderator's Note: The profile has been shared with the support team but has been removed from the public forum.

[lmiltchev]

A remote session may be needed in order to further troubleshoot this issue. Please open a new support ticket via our support center here:

https://support.nagios.com/tickets/