Nagiosxi restore issue

[elinagios]

I have 3 nagiosxi instances, test, devel and live.
Test i migrated successfuly from centos 7 to redhat 9. No issues, all went smooth and after the migration was able to upgrade nagiosxi to the latest version too. Test nagiosxi has default mysql root password.

Now the devel version im not able to get migrated properly, here are the steps i follow.
1) On new instance, Redhat 9, i do nagios fullinstall. NagiosXI version is the same as the running devel version, 5.11.2
2) On the running version, centos 7 and nagiosxi 5.11.2 i create a backup from the webui and transfer it to the Redhat 9 server.
3) Fullinstall finishes without any errors.
4) /usr/local/nagiosxi/scripts/restore_xi.sh i change the „themysqlpass“ value to the mysql root password. On the running version i check it with „mysql -u root -p“ command, i got the correct mysql root password and its not the default. Mysql server version: 5.5.68-MariaDB
5) I run the „/usr/local/nagiosxi/scripts/restore_xi.sh /backupfile/location/file.gz“ to restore the backup that was created earlier.
6) Now here things get broken, the restore starts:
Restoring directories to /...
Restoring Nagios Core...
Restoring Nagios XI...
Restoring NRDP backups...
Restoring MRTG...
Restoring SNMP configuration files...
Restoring SNMP MIBs...
Restoring Nagvis backups...
Restoring nagios home dir...
Restoring MySQL databases...
ERROR: Could not connect to localhost:3306 with root password supplied.
Please enter the MySQL root password:
7) It will not accept the mysql root password from the running instance and what was changed to in themysqlpass value, instead it now accepts the default password, nagiosxi. For some reason the restore script changes the mysql root back to default
8) Restore completes without giving any errors, RESTORE COMPLETE.
9) i run restore_repair.sh skript:
Shared module ndo.so installed
/usr/local/nagios/etc/nagios.cfg configured to use NDO
UPGRADE: POST-UPGRADE: NDO post upgrade started...
Performing install...
> Creating database with user
> Enter MySQL root password:

It waits for the default nagiosxi password and it ends with:

mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1007 (HY000) at line 1: Can't create database 'nagios'; database exists
Something went wrong creating database 'nagios'
Finished restore repair OK

In nagios.log i have diferent errors:
[1714730335] Error: Could not load module '/usr/local/nagios/bin/ndomod.o' -> /usr/local/nagios/bin/ndomod.o: cannot open shared object file: No such file or directory
[1714730335] Error: Failed to load module '/usr/local/nagios/bin/ndomod.o'.
[1714730335] Error: Module loading failed. Aborting.
[1714730803] NDO-3: NDO 3.1.0 (c) Copyright 2009-2023 Nagios - Nagios Core Development Team
[1714730803] NDO-3: Unable to connect to mysql. Configuration may be incorrect or database may have temporarily disconnected.
[1714730803] NDO-3: NDO was not able to initialize the database (main context) and will not start.
[1714730803] Event broker module '/usr/local/nagios/bin/ndo.so' initialized successfully.

10) In mysql i run: GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION; and run restore_repair.sh again. Nothing changes, logs still have db connection errors and no running nagios.

Also not sure if its related or not, the new install now has no ndo2db file available also.
So probably the main issue is the restore of the dump and why the mysql root password is reset back to default.

[jsimon]

Hi @elinagios,

It sounds like you're running into issues with the old version you're trying to restore looking for the old password, and not the new password you set. I would recommend setting the password on your Redhat 9 server to the same value as the password from your CentOS7 server, run the restore, and then re-change the password, if you wish.

I've linked our recommended backup and restore process below:
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

As well as our recommended steps for changing your mysql password:
https://answerhub.nagios.com/support/s/ ... 4-9564b55d

Hopefully this helps you resolve your issue. Please advise if you run into any other problems with this process!

[danderson]

It sounds like you are doing everything properly.

I would make sure that the mysqlpass parameter in /usr/local/nagiosxi/etc/xi-sys.cfg and /usr/local/nagiosxi/var/xi-sys.cfg is the correct root password .

If that doesn't work, could you try upgrading to the latest version and then try restoring? A lot of recent changes were made to the backup and restore scripts that centered around ensuring the accuracy of the root passwords.

[elinagios]

First off, im following all the nagiosxi backup/restore reccomendations.
Now, luckily 2 posts have helped me a bit further, but the end is still the same.
First off, i went to check /usr/local/nagiosxi/etc/xi-sys.cfg and /usr/local/nagiosxi/var/xi-sys.cfg files and what do i see, in my current running centos 7 installation, the mysqlpass parameter is not the mysql root but the default password. Here is my first question, how can nagiosxi even work if those values dont match, mysql root password and the value defined in those 2 config files.

In centos 7 i changes the values to match the mysql root password, restart services, everything seems to be working. So, i create a new backup file, move the file to Redhat 9 installation where i run now the following steps.
1) Fullinstall is done without issues.
2) I start the restore process with the dump that was made in centos 7, the restore finishes this time without asking for mysql root password and waiting for the default password and restore completes without any errors or prompts. At this point it seemed to me victory is close.
3) Now to running restore_repair.sh, this finishes:
Shared module ndo.so installed
/usr/local/nagios/etc/nagios.cfg configured to use NDO
UPGRADE: POST-UPGRADE: NDO post upgrade started...
Performing install...
> Creating database with user
> Enter MySQL root password:
> Using command line supplied credentials for account/db creation...
> Username: ndoutils
> Password: *****
> Database: nagios
> Press <ENTER> to continue...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1007 (HY000) at line 1: Can't create database 'nagios'; database exists
Something went wrong creating database 'nagios'
Finished restore repair OK

I also try to run in mysql and rerun restore_repair.sh:
GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION;

But the problem remains, there is no running nagiosxi yet.

At this point i check the nagios.log file, i see:
714980345] Error: Could not load module '/usr/local/nagios/bin/ndomod.o' -> /usr/local/nagios/bin/ndomod.o: cannot open shared object file: No such file or directory
[1714980345] Error: Failed to load module '/usr/local/nagios/bin/ndomod.o'.
[1714980345] Error: Module loading failed. Aborting.

I restart nagios proccess, the error above seems to be gone, but now mysql connections issues exist:
[1714982520] NDO-3: NDO 3.1.0 (c) Copyright 2009-2023 Nagios - Nagios Core Development Team
[1714982520] NDO-3: Unable to connect to mysql. Configuration may be incorrect or database may have temporarily disconnected.
[1714982520] NDO-3: NDO was not able to initialize the database (main context) and will not start.

I recheck the xi-sys.cfg files, repair have reset mysqlpass again, i change it again to match the root, restart nagios service but the error remains:
[1714987781] Nagios 4.4.13 starting... (PID=67231)
[1714987781] Local time is Mon May 06 12:29:41 EEST 2024
[1714987781] LOG VERSION: 2.0
[1714987781] qh: Socket '/usr/local/nagios/var/rw/nagios.qh' successfully initialized
[1714987781] qh: core query handler registered
[1714987781] qh: echo service query handler registered
[1714987781] qh: help for the query handler registered
[1714987781] wproc: Successfully registered manager as @wproc with query handler
[1714987781] wproc: Registry request: name=Core Worker 67234;pid=67234
[1714987781] wproc: Registry request: name=Core Worker 67235;pid=67235
[1714987781] wproc: Registry request: name=Core Worker 67233;pid=67233
[1714987781] wproc: Registry request: name=Core Worker 67232;pid=67232
[1714987781] NDO-3: NDO 3.1.0 (c) Copyright 2009-2023 Nagios - Nagios Core Development Team
[1714987781] NDO-3: Unable to connect to mysql. Configuration may be incorrect or database may have temporarily disconnected.
[1714987781] NDO-3: NDO was not able to initialize the database (main context) and will not start.
[1714987781] Event broker module '/usr/local/nagios/bin/ndo.so' initialized successfully.
[1714987781] Successfully launched command file worker with pid 67236

I follow the "How to Change Default Passwords in Nagios XI" documentation, all the rest of user passwords are correct in in current centos 7 env and in new redhat9, they are not default but the do match in all config files.
Mysql is running and i can connec to to it via root:
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| nagios |
| nagiosql |
| nagiosxi |
| performance_schema |
| sys |
+--------------------+
7 rows in set (0.01 sec)

What else am i missing?

"If that doesn't work, could you try upgrading to the latest version and then try restoring? A lot of recent changes were made to the backup and restore scripts that centered around ensuring the accuracy of the root passwords."
Sadly no, i want to migrate first test (done), devel and live before upgrading to v2024.

[danderson]

I think what is happening is the ndo password is also wrong.

The mysqlpass parameter in xi-sys.cfg is the root password and is only used during things that require root access like special scripts, installs, and upgrades. There are four other passwords for four other users. Each user has it's own database. The ndoutils user accesses the nagios database. The nagiosxi user accesses the nagiosxi database. The dbmaint_nagiosxi user is for editing specifically the auditlog table in the nagiosxi database. The nagiosql user accesses the nagiosql database. You can see the passwords for these in /usr/local/nagiosxi/html/config.inc.php

What is happening is that you are restoring from a backup on a different machine, and the nagios user password stored in the ndo.cfg file is specific to that machine. On the new machine, the password stored in ndo.cfg becomes incorrect on restore because it's from the old machine. The restore script tries to install the nagios database, and fails.

If you don't want to do an upgrade before the restore, you can still use the restore script.

Code: Select all

cd /usr/local/nagiosxi/scripts && mv restore_xi.sh restore_xi.sh.backup
wget -O- https://assets.nagios.com/downloads/nagiosxi/2024/xi-2024R1.1.2.tar.gz | tar Pxzf - nagiosxi/nagiosxi/basedir/scripts/restore_xi.sh --strip-components 4

Otherwise, I would make it so that all the passwords for each user are the same between the machines, and I would update the following files on both machines accordingly

Code: Select all

/usr/local/nagiosxi/etc/xi-sys.cfg
/usr/local/nagiosxi/var/xi-sys.cfg
/usr/local/nagiosxi/html/config.inc.php
/usr/local/nagios/etc/ndo.cfg
If you have nagvis
/usr/local/nagvis/etc/nagvis.ini.php

[elinagios]

Please keep in mind my redhat 9 is totally clean server with empty default nagiosxi install on it
So, did few more tests:
1) If i take the new restore script, restore proccess ends with following:
Backup files look okay. Preparing to restore...
Shutting down services...
Restoring directories to /...
Restoring Nagios Core...
Restoring Nagios XI...
Restoring NRDP backups...
Restoring MRTG...
Restoring SNMP configuration files...
Restoring SNMP MIBs...
Restoring Nagvis backups...
Restoring nagios home dir...
Syncronizing MySQL root password...
MySQL root passwords unsyncronized...
Current MySQL root password worked
Restoring MySQL databases...
Syncronizing database passwords for MySQL user ndoutils...
Database passwords for MySQL user ndoutils unsyncronized...
Backup and current passwords both incorrect. Resetting MySQL password to current password...
mysql: [Warning] Using a password on the command line interface can be insecure.
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
Syncronizing database passwords for MySQL user nagiosql...
Database passwords for MySQL user nagiosql unsyncronized...
Backup and current passwords both incorrect. Resetting MySQL password to current password...
mysql: [Warning] Using a password on the command line interface can be insecure.
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
Restoring Nagios XI MySQL database...
Syncronizing database passwords for MySQL user nagiosxi...
Database passwords for MySQL user nagiosxi unsyncronized...
Backup and current passwords both incorrect. Resetting MySQL password to current password...
mysql: [Warning] Using a password on the command line interface can be insecure.
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
Syncronizing database passwords for MySQL user ...
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
Restarting database servers...
Restoring Apache cronjobs...
Restoring logrotate config files...
Skipping Apache config files restoration

===============
RESTORE COMPLETE
===============

/usr/local/nagios/etc/ndo.cfg
password is blank

xi-sys.cfg
mysqlpass passwords are reset to nagiosxi

/usr/local/nagiosxi/html/config.inc.php
passwords are reset to blank

So here on, i drop this try, not worth it.

2) I change the passwords in the config files of the default install to match the running centos 7 ones, all the users that are mentioned here, in the configs (by the way, those mysql users are working on centos 7 instance, i can connect via those to the db, even the ndoutils pass (not default).) Then i also change the default passwords in default install mysql to match with the ones in running setup.
Restore output:
Backup files look okay. Preparing to restore...
Shutting down services...
Restoring directories to /...
Restoring Nagios Core...
Restoring Nagios XI...
Restoring NRDP backups...
Restoring MRTG...
Restoring SNMP configuration files...
Restoring SNMP MIBs...
Restoring Nagvis backups...
Restoring nagios home dir...
Syncronizing MySQL root password...
Restoring MySQL databases...
Syncronizing database passwords for MySQL user ndoutils...
Database passwords for MySQL user ndoutils unsyncronized...
Backup password correct
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
Syncronizing database passwords for MySQL user nagiosql...
Database passwords for MySQL user nagiosql unsyncronized...
Backup password correct
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
Restoring Nagios XI MySQL database...
Syncronizing database passwords for MySQL user nagiosxi...
Database passwords for MySQL user nagiosxi unsyncronized...
Backup password correct
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
Syncronizing database passwords for MySQL user ...
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
Restarting database servers...
Restoring Apache cronjobs...
Restoring logrotate config files...
Skipping Apache config files restoration

===============
RESTORE COMPLETE
===============

At this point seems i can ignore the sed warnings, in config files the passwords are not changed to empty or reset to default.

Then i run restore_repair script, now it finishes totally different than in previous attempts:

make[1]: Leaving directory '/tmp/nagiosxi/subcomponents/ndo/ndoutils-2.1.3/src'
/bin/install -c -m 644 startup/default-service /usr/lib/systemd/system/ndo2db.service
./restore_repair.sh: line 69: [: -v: binary operator expected
Redirecting to /bin/systemctl restart ndo2db.service
Redirecting to /bin/systemctl restart nagios.service
Finished installing NDOutils...
Finished restore repair OK

At this point i seem to have a working nagiosxi installation, thank you @danderson. Now is it just me, or restoring nagiosxi without default passwords guide needs to be updated to match this info?

[danderson]

Most likely! I'll try and get that updated. Glad I could help. Let me know if you run into any other issues.

[elinagios]

Ok, seems it have stopped working at some point, in nagios.log i get:
[1715243846] ndomod: Still unable to connect to data sink. 2204 items lost, 5000 queued items to flush.
[1715244757] ndomod: Still unable to connect to data sink. 3019 items lost, 5000 queued items to flush.
[1715245668] ndomod: Still unable to connect to data sink. 3833 items lost, 5000 queued items to flush.
When i restart nagios proccess:
[1715247956] wproc: Successfully registered manager as @wproc with query handler
[1715247956] wproc: Registry request: name=Core Worker 35682;pid=35682
[1715247956] wproc: Registry request: name=Core Worker 35681;pid=35681
[1715247956] wproc: Registry request: name=Core Worker 35680;pid=35680
[1715247956] wproc: Registry request: name=Core Worker 35679;pid=35679
[1715247956] ndomod: NDOMOD 2.1.3 (2017-04-13) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
[1715247956] ndomod: Could not open data sink! I'll keep trying, but some output may get lost...

In syslog, there are messages:
Use of uninitialized value $NAGIOS{"MULTI_PARENT"} in printf at /usr/local/nagios/libexec/process_perfdata.pl line 466, <PDFILE> line 1.

In nagios web interface everything seems OK, but the checks arent doing anything (service and host checks are enabled)
Status on services is:
Last Check: 2024-05-08 13:52:02
Next Check: 2024-05-08 13:55:02
and even if i force the check, the Next check time will not change. Any ideas?
All i could think is that i did downgrade on bison package, are you depending on that with certain version?

[elinagios]

Ok, seems i partially found a solution to the data sink issue. After restore, ndo2db.service for some reason is in disabled at startup, so after restore once you reboot your server, the sink issue comes. Enable ndo2db.service and you are good to go.

Now for the lines in syslog (messages file):
Use of uninitialized value $NAGIOS{"MULTI_PARENT"} in printf at /usr/local/nagios/libexec/process_perfdata.pl line 466, <PDFILE> line 1.
Someone else had the same issue going from redhat 7 to 9, but his issue was not resolved sadly. Process data is gathered and charts updated, so not sure what that error is about.