Error on database restore

[jkelly1959]

We ran out of space on /usr/local/nagios lv which corrupted some files. We were getting config errors so we decided to do a restore from the back we took about an hour earlier after added space.

Here is the output.

[

Code: Select all

nagios@lnxvnaghq001 nagiosxi]$ sudo /usr/local/nagiosxi/scripts/restore_xi.sh /store/backups/nagiosxi/nagiosxi.1521830402.tar.gz
TS=1521838821
Extracting backup to /store/backups/nagiosxi/1521838821-restore...
In /store/backups/nagiosxi/1521838821-restore/nagiosxi.1521830402...
Backup files look okay.  Preparing to restore...
Shutting down services...
Restoring directories to /...
Restoring Nagios Core...
rm: cannot remove ‘/usr/local/nagios’: Device or resource busy
Restoring Nagios XI...
Restoring NagiosQL...
Restoring NagiosQL backups...
Restoring NRDP backups...
Restoring MRTG...
Restoring Nagvis backups...
Restoring MySQL databases...
ERROR 1010 (HY000) at line 22: Error dropping database (can't rmdir './nagios', errno: 16)
Error restoring MySQL database 'nagios' - check the password in this script!

This left the database in a lurch....our system cant display anything.
Please see attachment:

Help to get a clean restore would be appreciated

Some additional information.
One issue with the restore could be that to move nagios off the 'system' filesystems we gave them there own logical volume and mount point

F

Code: Select all

ilesystem                             Size  Used Avail Use% Mounted on
/dev/mapper/rhel_lnxvnaghq001-root     7.5G  3.4G  4.1G  46% /
devtmpfs                               5.8G     0  5.8G   0% /dev
tmpfs                                  5.8G     0  5.8G   0% /dev/shm
tmpfs                                  5.8G  594M  5.3G  11% /run
tmpfs                                  5.8G     0  5.8G   0% /sys/fs/cgroup
/dev/mapper/rhel_lnxvnaghq001-home     3.8G  300M  3.5G   8% /home
/dev/mapper/vg_nagios-lv_local_nagios   15G  4.0G   12G  27% /usr/local/nagios
/dev/mapper/rhel_lnxvnaghq001-patrol   1.4G   81M  1.3G   6% /patrol
/dev/sda1                              473M  226M  248M  48% /boot
/dev/mapper/vg_nagios-lv_store          25G  7.1G   18G  29% /store
/dev/mapper/rhel_lnxvnaghq001-var      3.8G  3.1G  687M  82% /var
/dev/mapper/vg_nagios-lv_mysql_nagios  5.0G   33M  5.0G   1% /var/lib/mysql/nagios
tmpfs                                  1.2G     0  1.2G   0% /run/user/10009
tmpfs                                  1.2G     0  1.2G   0% /run/user/10016
tmpfs                                  1.2G     0  1.2G   0% /run/user/50966
tmpfs                                  1.2G     0  1.2G   0% /run/user/10017

This will inhibit the some of rm's in the restore script.

[jkelly1959]

Code: Select all

Test mysql root password by line command with:
[root@lnxvnaghq001 scripts]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 1158609
Server version: 5.5.56-MariaDB MariaDB Server

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> 

We got in ok:

[‎3/‎23/‎2018 6:11 PM] Pennington, Brian:
N

Code: Select all

o Title 
MariaDB [(none)]> status
--------------
mysql  Ver 15.1 Distrib 5.5.56-MariaDB, for Linux (x86_64) using readline 5.1

Connection id:          1158609
Current database:
Current user:           root@localhost
SSL:                    Not in use
Current pager:          stdout
Using outfile:          ''
Using delimiter:        ;
Server:                 MariaDB
Server version:         5.5.56-MariaDB MariaDB Server
Protocol version:       10
Connection:             Localhost via UNIX socket
Server characterset:    latin1
Db     characterset:    latin1
Client characterset:    utf8
Conn.  characterset:    utf8
UNIX socket:            /var/lib/mysql/mysql.sock
Uptime:                 19 days 18 hours 6 min 5 sec

Threads: 49  Questions: 195118948  Slow queries: 0  Opens: 430647  Flush tables: 2  Open tables: 13  Queries per second avg: 114.320
-------------- 
 

[scottwilkerson]

Code: Select all

/dev/mapper/vg_nagios-lv_mysql_nagios  5.0G   33M  5.0G   1% /var/lib/mysql/nagios

This is because of how you moved data for the database to a mount point that cannot be deleted by the DROP DATABASE in the restore script/

/var/lib/mysql/nagios needs to be able to be deleted before it is re-created by the restore script

[jkelly1959]

Scott,
Thats the conclusion we came to as well. the DB was easy. we changed the mount point from /var/lib/mysql/nagios to /var/lib/mysql so it would treat nagios as a directory.

the /usr/local/nagios was a bit tougher but we noticed an If Clause in the script for what it would try to remove depending on architecture and version. We noticed the else left the mount point alone and removed at a deeper level with were just directories. Restored successfully.

We copied or modified script off before we ran it since the restore would overwrite it.

I do have one other question. This restore was to the same server. If we restore to another server say at DR in an emergency does the license get restored or do we need to reload. Also how does the restore handle certs? Our certs are tied to the server name.

Thanks for you input.
John

[scottwilkerson]

I do have one other question. This restore was to the same server. If we restore to another server say at DR in an emergency does the license get restored or do we need to reload. Also how does the restore handle certs? Our certs are tied to the server name.

the license would get restored, however the certs would not as they are a custom configuration, not part of XI directly