run repair and mysql still won't start

[harveyscott]

I tried to log into the web interface and it says Database isn't running. So I ran the /usr/local/nagiosxi/scripts/repair_databases.sh and when I do that it runs the repair fine, but then it says "Starting mysqld [Failed]"

I tried to restarted and run the repair multiple times, but I get the same error.

- recovering (with sort) MyISAM-table 'tbl_variabledefinition.MYI'
Data records: 219
- Fixing index 1
MySQL Daemon failed to start.
Starting mysqld: [FAILED]
/var/lib/mysql/nagios

===============
REPAIR COMPLETE
===============
ndo2db was not running... could not stop
Starting ndo2db: done.
Running configuration check...

=======================
nagios database repair succeeded
nagiosql database repair succeeded

[tgriep]

Is the server's hard drive full?
That could cause the MYSQL daemon to not start up.
To check it, run the following commands as root.

Code: Select all

df -h
df -i

Try running command to repair all of the databases on the server and then see if it starts up.
Run this as root

Code: Select all

mysqlcheck -f -r -u root -pnagiosxi --all-databases --use_frm

Then see id the MYSQL database starts up.

If it still doesn't start, you will have to look at the MYSQL or MariaDB log files to see why it is not starting and post the errors here.

Code: Select all

/var/log/mysqld.log
/var/log/mariadb/mariadb.log

[harveyscott]

I used your template VM, that came pre-configured with a specific size. I haven't really made any changes in a while. In any case it says I have 7.3 Size and 7.0 GB available. So here is all the output.

[root@localhost nagios]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
7.3G 7.0G 0 100% /
tmpfs 498M 0 498M 0% /dev/shm
/dev/sda1 477M 66M 386M 15% /boot
[root@localhost nagios]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
494832 98574 396258 20% /
tmpfs 127411 1 127410 1% /dev/shm
/dev/sda1 128016 50 127966 1% /boot
[root@localhost nagios]# mysqlcheck -f -r -u root -pnagiosxi --all-databases --use_frm
mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) when trying to connect
[root@localhost nagios]#

[harveyscott]

Also see this in the log:


181019 18:49:03 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
181019 18:49:03 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
181019 18:49:03 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
181019 18:49:04 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
181019 18:49:04 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
181019 18:49:06 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
181019 18:49:06 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
181019 18:49:06 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
181019 18:49:06 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
181019 18:49:06 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
181019 18:49:06 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
181019 18:49:06 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
181019 18:49:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
181019 18:49:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
181019 18:49:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
181019 18:49:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
181019 18:49:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
181019 18:49:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
181019 18:49:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
181019 18:49:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_systemcommands' is marked as crashed and should be repaired
181019 18:49:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_systemcommands' is marked as crashed and should be repaired
181019 18:49:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_systemcommands' is marked as crashed and should be repaired
181019 18:49:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_systemcommands' is marked as crashed and should be repaired

[harveyscott]

I got it working. What happened was because the DB was corrupt, it filled up the log with errors every few minutes until it ran out of space. Now it's running. Any idea how I add space to the VM? For now it's running, but I'd like to add extra space

[harveyscott]

Sorry I spoke too soon. Ok, the repair runs and shows this now:

Data records: 219
- Fixing index 1
Starting mysqld: [ OK ]
/usr

===============
REPAIR COMPLETE
===============
Stopping ndo2db: done.
Starting ndo2db: done.
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.

=======================
nagios database repair succeeded
nagiosql database repair succeeded


But when I try to login to web interface it keeps on saying I need to repair the DB

[tgriep]

Did the server run out of space again?

Did you run the mysqlcheck command again?

I would increase the space of the hard drive, run the repair and see if that helps out.
The following instructions are how to increase the size of the hard drive it it is runnins as a virtual machine.
https://support.nagios.com/kb/article.php?id=266

[harveyscott]

Not i didn't run out of space again, and I did run the MySQL check again. But what did work, was rebooting the box after the repair. Now it is working.

[harveyscott]

Did the server run out of space again?

Did you run the mysqlcheck command again?

I would increase the space of the hard drive, run the repair and see if that helps out.
The following instructions are how to increase the size of the hard drive it it is runnins as a virtual machine.
https://support.nagios.com/kb/article.php?id=266

Thanks for the help, I will add space to prevent this in the future following the instructions above.

[tgriep]

Your welcome. Glad the server is working now. Let us know if you have any further questions or that it is OK to lock the post.