Nagios Support Forum

Greetings,

I'm trying to restore nagios using the /usr/local/nagiosxi/scripts/restore_xi.sh script from a back up that i have created using backup_xi.sh
i'm encountering the following error:

[root@NagiosServer scripts]# ./restore_xi.sh /www/backups/nagiosxi/system/1536684731.tar.gz
WARNING: you are trying to restore a OS 7 backup on a OS 6 system
Compiled plugins and other binaries as well as httpd configurations
will NOT be restored.

Are you sure you want to continue? [y/N] y
TS=1536761815
Extracting backup to /www/backups/nagiosxi/1536761815-restore...

In /www/backups/nagiosxi/1536761815-restore/1536684731...
Backup files look okay. Preparing to restore...
Shutting down services...
Stopping nagios (via systemctl): [ OK ]
Stopping ndo2db (via systemctl): [ OK ]
NPCD Stopped.
Restoring directories to /...
Restoring Nagios Core...
Restoring Nagios XI...
Restoring NagiosQL...
Restoring NagiosQL backups...
Restoring NRDP backups...
Restoring MRTG...
Restoring Nagvis backups...
Restoring MySQL databases...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Error restoring MySQL database 'nagios' - check the password in this script!
You have new mail in /var/mail/root

Everything seems to be going good until when it starts to restore the MYSQL database.
I've checked and the database is running fine
[root@NagiosServer scripts]# systemctl status mariadb.service
● mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2018-09-07 14:11:23 EDT; 4 days ago
Main PID: 27570 (mysqld_safe)
CGroup: /system.slice/mariadb.service
├─27570 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
└─28021 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid ...

Sep 07 14:11:21 ma209dlvnag001 systemd[1]: Starting MariaDB database server...
Sep 07 14:11:21 ma209dlvnag001 mariadb-prepare-db-dir[27539]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Sep 07 14:11:21 ma209dlvnag001 mariadb-prepare-db-dir[27539]: If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Sep 07 14:11:21 ma209dlvnag001 mysqld_safe[27570]: 180907 14:11:21 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Sep 07 14:11:21 ma209dlvnag001 mysqld_safe[27570]: 180907 14:11:21 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Sep 07 14:11:23 ma209dlvnag001 systemd[1]: Started MariaDB database server.
You have new mail in /var/mail/root

Can you please help here.

Thanks.

When i tried logging on to the nagios gui, i see the following message
Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh

i tried running the script as suggested
output:
[root@NagiosServer scripts]# ./repair_databases.sh
Redirecting to /bin/systemctl status mysqld.service
Unit mysqld.service could not be found.
Redirecting to /bin/systemctl start mysqld.service
Failed to start mysqld.service: Unit not found.
Redirecting to /bin/systemctl status mysqld.service
Unit mysqld.service could not be found.
DATABASE: nagios
TABLE:
/var/lib/mysql/nagios /usr/local/nagiosxi/scripts
Redirecting to /bin/systemctl stop mysqld.service
Failed to stop mysqld.service: Unit mysqld.service not loaded.
- recovering (with sort) MyISAM-table 'nagios_acknowledgements.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_commands.MYI'
Data records: 225
- Fixing index 1
- Fixing index 2

---------
.
.
.
.
.
- recovering (with sort) MyISAM-table 'tbl_user.MYI'
Data records: 2
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_variabledefinition.MYI'
Data records: 0
- Fixing index 1
Redirecting to /bin/systemctl start mysqld.service
Failed to start mysqld.service: Unit not found.
/usr/local/nagiosxi/scripts

===============
REPAIR COMPLETE
===============
Restarting ndo2db (via systemctl): [ OK ]
Restarting nagios (via systemctl): [ OK ]

Nagios gui is still not loading up.

The issue might actually be with mysql database. Any thoughts?

After doing a systemctl restart mariadb.service i was able to bring up the gui, but the database backend (ndo2db) is not coming up.
I see the following message when trying to restart ndo2db

[root@NagiosServer nagios]# /etc/init.d/nagios status
nagios (pid 17918) is running...
[root@NagiosServer nagios]# /etc/init.d/npcd status
NPCD running (pid 29017).
[root@NagiosServer nagios]# /etc/init.d/ndo2db status
ndo2db is not running but subsystem locked
[root@NagiosServer nagios]# /etc/init.d/ndo2db restart
Restarting ndo2db (via systemctl): [ OK ]
[root@NagiosServer nagios]# /etc/init.d/ndo2db status
ndo2db is not running but subsystem locked

lets run

interestingly i'm missing those files:
here's the output:
[root@NagiosServer nagios]# ls -l /usr/local/nagios/var/ndo2db.lock
ls: cannot access /usr/local/nagios/var/ndo2db.lock: No such file or directory
[root@NagiosServer nagios]# ls -l /usr/local/nagios/var/ndo.sock
ls: cannot access /usr/local/nagios/var/ndo.sock: No such file or directory
[root@NagiosServer nagios]# ls -l /usr/local/nagios/
total 56624
drwx------ 3 nagios nagios 22 Nov 8 2016 audit
drwxr-xr-x 3 nagios nagios 57 Aug 24 17:53 backups
drwxr-xr-x 2 nagios nagios 192 Apr 13 13:48 bin
drwsrwsr-x 11 apache nagios 4096 Sep 12 12:00 etc
drwxr-xr-x 2 root root 6 Nov 4 2015 include
drwxrwxr-x 8 nagios nagios 12288 Sep 12 10:36 libexec
drwx------ 2 root root 4096 Nov 8 2015 libexec_old
-rw-rw-rw- 1 nagios nagios 55701338 Apr 13 13:40 libexec_server.tar.gz
drwxr-xr-x 2 nagios nagios 4096 Aug 23 16:27 sbin
-rw------- 1 nagios nagios 2238260 Apr 3 13:10 sbin.tar.gz
drwxr-xr-x 18 nagios nagios 4096 Nov 4 2015 share
drwxrwxr-x 6 nagios nagios 4096 Sep 12 13:10 var
[root@NagiosServer nagios]# ls -l /usr/local/nagios/var/
total 161672
drwxr-xr-x 2 nagios nagios 45056 Sep 11 00:00 archives
-rw-r--r-- 1 nagios nagios 312 Sep 12 13:10 host-perfdata
-rw-r--r-- 1 nagios nagios 140350 Sep 12 12:54 nagios.configtest
-rw-r--r-- 1 nagios nagios 176267 Sep 12 13:10 nagios.debug
-rw-r--r-- 1 nagios nagios 1000031 Sep 12 13:10 nagios.debug.old
-rw-r--r-- 1 nagios nagios 6 Sep 12 11:50 nagios.lock
-rwxr-xr-x 1 nagios nagios 9964927 Sep 12 13:10 nagios.log
-rw-rw-r-- 1 nagios nagios 9681120 Oct 13 2016 nagios.tmp3ikp4W
-rw------- 1 nagios nagios 0 Oct 12 2016 nagios.tmp4fdt5H
-rw-rw-r-- 1 nagios nagios 3039519 Feb 11 2016 nagios.tmpHwE6ZM
-rw------- 1 nagios nagios 1060864 Oct 17 2016 nagios.tmpKvm9QR
-rw-rw-r-- 1 nagios nagios 363642 Nov 8 2015 nagios.tmplQmMiY
-rw------- 1 nagios nagios 9530853 Oct 12 2016 nagios.tmpmIaHd6
-rw-rw-r-- 1 nagios nagios 3610197 Apr 13 2016 nagios.tmpP8vwdG
-rw-rw-r-- 1 nagios nagios 23708060 Aug 31 2017 nagios.tmpPNFlJc
-rw-rw-r-- 1 nagios nagios 9494375 Oct 12 2016 nagios.tmpRO1M1B
-rw-rw-r-- 1 nagios nagios 9493107 Oct 12 2016 nagios.tmpTb8pxK
-rw-rw-r-- 1 nagios nagios 9462483 Oct 12 2016 nagios.tmpXurj5O
-rw-r--r-- 1 nagios nagios 3934877 Sep 12 12:02 npcd.log
-rw-r--r-- 1 nagios nagios 10485784 Oct 6 2016 npcd.log.old
-rwxr-xr-x 1 nagios nagios 11257171 Sep 12 11:50 objects.cache
-rwxr-xr-x 1 nagios nagios 11257171 Sep 12 12:54 objects.precache
-rw-rw-r-- 1 nagios nagios 7963733 Sep 12 11:26 perfdata.log
-rw------- 1 nagios nagios 14388190 Sep 12 12:50 retention.dat
drwxrwsr-x 2 nagios nagcmd 41 Sep 12 11:50 rw
-rw-r--r-- 1 nagios nagios 52110 Sep 12 13:10 service-perfdata
drwxr-xr-x 5 nagios nagios 55 Nov 4 2015 spool
drwxr-xr-x 2 nagios nagios 48 Sep 12 13:10 stats
-rw-rw-r-- 1 nagios nagios 14303088 Sep 12 13:10 status.dat
You have new mail in /var/mail/root
[root@NagiosServer nagios]#

Ok can we start ndo2db and if that starts restart nagios

[root@NagiosServer nagios]# systemctl restart ndo2db
[root@NagiosServer nagios]# systemctl status ndo2db
● ndo2db.service - Nagios Data Out Daemon
Loaded: loaded (/usr/lib/systemd/system/ndo2db.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2018-09-12 14:29:53 EDT; 10s ago
Docs: http://www.nagios.org/documentation
Process: 3661 ExecStopPost=/bin/rm -f /usr/local/nagios/var/ndo2db.pid (code=exited, status=0/SUCCESS)
Process: 3659 ExecStart=/usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f (code=exited, status=127)
Main PID: 3659 (code=exited, status=127)

Sep 12 14:29:53 NagiosServer systemd[1]: Started Nagios Data Out Daemon.
Sep 12 14:29:53 NagiosServer systemd[1]: Starting Nagios Data Out Daemon...
Sep 12 14:29:53 NagiosServer ndo2db[3659]: /usr/local/nagios/bin/ndo2db: error while loading shared libraries: libmysqlclient.so.16: cannot open shared object file: No such file or directory
Sep 12 14:29:53 NagiosServer systemd[1]: ndo2db.service: main process exited, code=exited, status=127/n/a
Sep 12 14:29:53 NagiosServer systemd[1]: Unit ndo2db.service entered failed state.
Sep 12 14:29:53 NagiosServer systemd[1]: ndo2db.service failed.
[root@NagiosServer nagios]# systemctl restart nagios
[root@NagiosServer nagios]# systemctl status nagios
● nagios.service - LSB: Starts and stops the Nagios monitoring server
Loaded: loaded (/etc/rc.d/init.d/nagios; bad; vendor preset: disabled)
Active: active (running) since Wed 2018-09-12 14:30:22 EDT; 13s ago
Docs: man:systemd-sysv-generator(8)
Process: 5817 ExecStop=/etc/rc.d/init.d/nagios stop (code=exited, status=0/SUCCESS)
Process: 5873 ExecStart=/etc/rc.d/init.d/nagios start (code=exited, status=0/SUCCESS)
Main PID: 5896 (nagios)
CGroup: /system.slice/nagios.service
├─5896 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
├─5898 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─5899 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─5900 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─5901 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─5903 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─5904 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─5918 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
├─6589 /usr/local/nagios/libexec/check_nrpe -H ma209dlvtib802 -t 30 -c check_cpu -a -w 85 -c 90
├─6638 /usr/local/nagios/libexec/check_nrpe -H ma209dlpdb2801 -t 30 -c check_cpu -a -w 85 -c 90
├─6681 /usr/local/nagios/libexec/check_nrpe -H ma299mlvnav111 -t 30 -c check_cpu
└─6682 /usr/local/nagios/libexec/check_nrpe -H ma299mlvnav115 -t 30 -c check_cpu

Sep 12 14:30:31 NagiosServer nagios[5896]: wproc: stderr line 02: BEGIN failed--compilation aborted at /usr/local/bin/eep-nagios.pl line 13.
Sep 12 14:30:31 NagiosServer nagios[5896]: wproc: NOTIFY job 41 from worker Core Worker 5898 is a non-check helper but exited with return code 7
Sep 12 14:30:31 NagiosServer nagios[5896]: wproc: host=ma209mlvdip111; service=http://ma209mlvdip111:8180; contact=ccr
Sep 12 14:30:31 NagiosServer nagios[5896]: wproc: early_timeout=0; exited_ok=1; wait_status=1792; error_code=0;
Sep 12 14:30:32 NagiosServer check_nrpe[6415]: Remote 10.121.66.9 does not support Version 3 Packets
Sep 12 14:30:32 NagiosServer check_nrpe[6416]: Remote 10.120.87.226 does not support Version 3 Packets
Sep 12 14:30:33 NagiosServer check_nrpe[6528]: Remote 10.121.28.152 does not support Version 3 Packets
Sep 12 14:30:33 NagiosServer check_nrpe[6527]: Remote 10.142.204.146 does not support Version 3 Packets
Sep 12 14:30:34 NagiosServer check_nrpe[6590]: Remote 10.142.204.178 does not support Version 3 Packets
Sep 12 14:30:34 NagiosServer check_nrpe[6589]: Remote 10.142.150.229 does not support Version 3 Packets
Sep 12 14:30:36 NagiosServer check_nrpe[6708]: Remote 10.142.150.229 does not support Version 3 Packets
You have new mail in /var/mail/root

Any chance this might be because of the error during the restore?
Restoring MySQL databases...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Error restoring MySQL database 'nagios' - check the password in this script!
From the first post i made.

Oh, you were restoring a OS 7 backup on a OS 6 system?
Can you share these files Also did you run through the steps on page 12 here
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

you likely also need to run

After executing the following commands i was able to bring up the ndo2db database.
cd /tmp/nagiosxi/subcomponents/ndoutils
./upgrade

Now Nagios is running fine. But i'm not sure if the nagios restore completed succesfully.
Sharing the files you asked for here.