A database connection error has been detected

[RIDS_I2MP]

Hello Team,

We are facing an issue from last few weeks on Nagios XI servers. The Nagios XI GUI becomes inaccessible and the server goes into hung state.
After rebooting the server, the GUI becomes accessible with error below error message:

"Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh

Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh


We have to run the DB repair script as below:
service mysqld stop
/usr/local/nagiosxi/scripts/repairmysql.sh nagios
service mysqld start

After we run DB repair script, GUI becomes accessible.
The problem is, every time we perform a reboot, we have to run the DB repair script. The issue has occurred a number of time in last few weeks.

We are not sure about Nagios database, how and from where we can access it.
Please let me know what needs to be done to avoid this recurring issue.

Thanks in advance!!!

[tgriep]

How you are describing the issue sounds like the Max Connections to the MYSQL database is getting exceeded and corrupting the database.
You can increase the connections to the database by following this KB article.
https://support.nagios.com/kb/article/n ... s-513.html

Let us know if this solves the MYSQL corruption on your server.

[RIDS_I2MP]

Hello,

Thanks for your reply!!

As per document, we ran below two queries and it seems that there is no issue related to max connections value, so we have not executed the other steps, that is, we have not done any changes in max_connections value. I am attaching the output here.

Can you please tell me what exactly Max_used_connections value defines?

Please let us know what needs to be done next.

[RIDS_I2MP]

Hello,

I just wanted to give an update. We checked the value of Max_used_connections again on our server and we found that it was increased to 91.
[nagios@eu2napu003 ~]$ mysql -uroot -pnagiosxi -e "show global status like 'Max_ used_connections';"
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| Max_used_connections | 91 |
+----------------------+-------+

I mean, the value of Max_used_connections is changing. So, it seems that whenever we face the DB related issues as I described, at that time the value of Max_used_connections reaches upto 151 or closer to it.

Please help us on it.

[tgriep]

The max_connections setting is the number of maximum connections that the MYSQL database will allow. Any number over that, it will not allow.
The Max_used_connections is the number of max connections that the server had at one time. It will change depending on what is connecting to the server.

The issue is if the Max_used_connections gets up to the max_connections setting, the MYSQL server will not allow the connection and that causes the corruption.
Something else may be causing the corruption so if you can post your System Profile here, we can take a look at it for any issues.
To get your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and upload it to the post.

[RIDS_I2MP]

Hello,

We are facing the same issue on two Nagios XI servers.
I am attaching profile.zip file for both the servers here.

[tgriep]

I took a look at both of the Profiles and didn't see anything outstanding on what is causing the issue with the server hanging.
If you are powering off the server or forcing a reboot, that is causing the corruption, which happens when that is done.
Without seeing what is happening to the server when it becomes inaccessible, it is hard to say what is causing the issue.

[RIDS_I2MP]

Hello,

Thank you again for your support.
Please let me know what all data we can provide to you now or we can wait for the next occurrence of the issue and provide the required data to you. I am attaching some log data here, I found some error related to tables in DB.
Also, can we change the Max_connections value just to see if it resolves issue, please suggest.

[tgriep]

Those log entries are from the year 2016 so they are not relevant to the issues you are having now.

But yes, you can increase the max_connections for MYSQL to see if it helps the issue.

[RIDS_I2MP]

Hello,

We would be waiting for one or two weeks to see if it occurs again and will do Max_connection value changes after that.
We will also provide you update on the same, please keep this topic open