Page 3 of 4

Re: Recurring Database Connection Error

Posted: Thu Nov 02, 2017 5:52 am
by venturegardengroup
Thanks npolovenko,

Please find attached as requested

Re: Recurring Database Connection Error

Posted: Thu Nov 02, 2017 3:05 pm
by dwasswa
Hi @venturegardengroup,

Can you please delete these files again:

Code: Select all

rm /var/lib/mysql/ib_logfile0
rm /var/lib/mysql/ib_logfile1
Then restart mysqld:

Code: Select all

service mysqld restart

Re: Recurring Database Connection Error

Posted: Fri Nov 03, 2017 6:51 am
by tacolover101
eloyd wrote:My immediate reaction is that almost 50,000 failed logins in as root in three days is a serious issue that needs to be addressed. Use a non-standard port, set up firewalls, activate fail2ban, and audit accounting output to make sure someone hasn't managed to guess your root password. That's pretty insane; you're averaging one failed login every six seconds, continuously.

My second reaction is to issue this command first:

Code: Select all

cd /usr/local/nagiosxi/scripts
Before executing:

Code: Select all

./repair_databases.sh
Some of the XI scripts are funny and need to be run from the directory, not just called by name.
+1

or maybe they've got a security team who is actually doing their job =). or a service account using the wrong credentials. though this seems a bit excessive, and concerning.

@venturegardengroup - how large is your installation? what type of disks are you running for your DB server? is it offloaded?

Re: Recurring Database Connection Error

Posted: Fri Nov 03, 2017 10:47 am
by npolovenko
@venturegardengroup, In addition to the previous answers, I'd ask you to upload the MariaDB log from:

Code: Select all

tail - f /var/log/mariadb/
and please run the following two commands and show us the output here as well:

Code: Select all

ls -l /var/lib/mysql/mysql

Code: Select all

ls -l /var/lib/mysql/nagiosxi

Also, you've done this before but it's important that all 4 files are deleted from

Code: Select all

/var/lib/mysql
at the same time. ib_logfile0, ib_logfile1 , ibdata1, ibdata1.bckup
So fisrt make sure that mysqld is stopped with

Code: Select all

service mysqld stop
and then please go to that folder again and delete all 4 files. After all 4 files are gone please restart mysql.

Code: Select all

 service mysqld restart

Re: Recurring Database Connection Error

Posted: Mon Nov 06, 2017 3:50 am
by venturegardengroup
@tacolover: Thanks for your recommendations. Except for the security aspect which our team here is still working on, I had run the other suggestions but nothing happened and it returned no outputs as well.

@Npolovenko: Thanks... I have deleted 4 of the files you suggested, please also see outputs of other commands attached.

Regards,

Re: Recurring Database Connection Error

Posted: Mon Nov 06, 2017 2:21 pm
by dwasswa
Hi @venturegardengroup,

Please run the following commands and post the output:

Code: Select all

tail -50 /var/log/mysqld.log
tail -50 /var/log/mariadb/mariadb.log

Re: Recurring Database Connection Error

Posted: Wed Nov 08, 2017 3:40 am
by venturegardengroup
Hello,

Please find attached.

Regards,

Re: Recurring Database Connection Error

Posted: Wed Nov 08, 2017 1:35 pm
by dwasswa
Hi @venturegardengroup,

Please run the following commands as root and post the /tmp/info.txt file

Code: Select all

ls -lR /var/lib/mysql >/tmp/info.txt
tail -100 /var/log/mariadb/mariadb.log >>/tmp/info.txt
Secondly, I strongly suggest that you open a ticket so we can get access to the live system for a closer look.

Below is the link to open a ticket.
https://support.nagios.com/tickets/

Note: You can log in with your Nagios support forum username and password.

Re: Recurring Database Connection Error

Posted: Mon Nov 13, 2017 4:47 am
by venturegardengroup
Hello Dwasswa,

Thank for your feedback. Please find attached as requested. I am also going to open a ticket for this as suggested.

Regards,

Re: Recurring Database Connection Error

Posted: Mon Nov 13, 2017 3:56 pm
by dwasswa
Hi @venturegardengroup,

Please attach your my.cnf file by following the steps below:

Step 1:Locate my.cnf file located at:

On CentOS and Fedora:/etc/my.cnf

Step 2:
Attach or post the my.cnf file.

Secondly, did you open a ticket as I previously suggested, If you didn't, please do?