Recurring Database Connection Error

[venturegardengroup]

Thanks npolovenko,

Please find attached as requested

[dwasswa]

Hi @venturegardengroup,

Can you please delete these files again:

Code: Select all

rm /var/lib/mysql/ib_logfile0
rm /var/lib/mysql/ib_logfile1

Then restart mysqld:

Code: Select all

service mysqld restart

[tacolover101]

My immediate reaction is that almost 50,000 failed logins in as root in three days is a serious issue that needs to be addressed. Use a non-standard port, set up firewalls, activate fail2ban, and audit accounting output to make sure someone hasn't managed to guess your root password. That's pretty insane; you're averaging one failed login every six seconds, continuously.

My second reaction is to issue this command first:

Code: Select all

cd /usr/local/nagiosxi/scripts

Before executing:

Code: Select all

./repair_databases.sh

Some of the XI scripts are funny and need to be run from the directory, not just called by name.

+1

or maybe they've got a security team who is actually doing their job =). or a service account using the wrong credentials. though this seems a bit excessive, and concerning.

@venturegardengroup - how large is your installation? what type of disks are you running for your DB server? is it offloaded?

[npolovenko]

@venturegardengroup, In addition to the previous answers, I'd ask you to upload the MariaDB log from:

Code: Select all

tail - f /var/log/mariadb/

and please run the following two commands and show us the output here as well:

Code: Select all

ls -l /var/lib/mysql/mysql

Code: Select all

ls -l /var/lib/mysql/nagiosxi

Also, you've done this before but it's important that all 4 files are deleted from

Code: Select all

/var/lib/mysql

at the same time. ib_logfile0, ib_logfile1 , ibdata1, ibdata1.bckup
So fisrt make sure that mysqld is stopped with

Code: Select all

service mysqld stop

and then please go to that folder again and delete all 4 files. After all 4 files are gone please restart mysql.

Code: Select all

 service mysqld restart

[venturegardengroup]

@tacolover: Thanks for your recommendations. Except for the security aspect which our team here is still working on, I had run the other suggestions but nothing happened and it returned no outputs as well.

@Npolovenko: Thanks... I have deleted 4 of the files you suggested, please also see outputs of other commands attached.

Regards,

[dwasswa]

Hi @venturegardengroup,

Please run the following commands and post the output:

Code: Select all

tail -50 /var/log/mysqld.log
tail -50 /var/log/mariadb/mariadb.log

[venturegardengroup]

Hello,

Please find attached.

Regards,

[dwasswa]

Hi @venturegardengroup,

Please run the following commands as root and post the /tmp/info.txt file

Code: Select all

ls -lR /var/lib/mysql >/tmp/info.txt
tail -100 /var/log/mariadb/mariadb.log >>/tmp/info.txt

Secondly, I strongly suggest that you open a ticket so we can get access to the live system for a closer look.

Below is the link to open a ticket.
https://support.nagios.com/tickets/

Note: You can log in with your Nagios support forum username and password.

[venturegardengroup]

Hello Dwasswa,

Thank for your feedback. Please find attached as requested. I am also going to open a ticket for this as suggested.

Regards,

[dwasswa]

Hi @venturegardengroup,

Please attach your my.cnf file by following the steps below:

Step 1:Locate my.cnf file located at:

On CentOS and Fedora:/etc/my.cnf

Step 2:
Attach or post the my.cnf file.

Secondly, did you open a ticket as I previously suggested, If you didn't, please do?