Cannot upgrade from Nagios version 5.4.13 to 5.6.7

[safuanmansor]

Hi Support,

We are having difficulties running the upgrading the nagios from 5.4.13 to 5.6.7 and the upgrade process stop here.

nagios.jpeg

Upon checking , the remote database settings are correct as perplace as the guide below
https://assets.nagios.com/downloads/nag ... Server.pdf

The difference is the only nagiosql and nagiosxi db is in remote server while the nagios db is localhost.

Thanks,
Safuan

[benjaminsmith]

Hi Safuan,

Looking over the log, ndo2bd compiled ok but it's not connecting to the remote database as expected. There must be some incorrect settings in the configurations.

On page 12 of the offloading the database guide there are some test commands, do those return valid results?

Code: Select all

mysql -u nagiosql -p'nagiosqlP@ssw0rd' -h <IP_ADDRESS_OF_MYSQL_OR_MARIADB_SERVER> -e STATUS;

Please send us the system profile, so we can review the configuration as well.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.

[joash0809]

Hi Benjamin,

Attached is the system profile as requested

Moderator's Note: The profile has been shared with the support team but has been removed from the public forum.

[benjaminsmith]

Hi,

So I noticed a few things int the profile. THe database settings look fine to me, but there were some crashed database tables, and that may have caused some issues with the upgrade.

Code: Select all

200820  8:00:00 [ERROR] mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failed

Log in as root and run the following command to repair and check the tables.Let me know if you get any errors here.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

The system is having trouble connecting to our license server. If you have any network issues here that will affect the upgrade since it needs to download and update system packages. Are you able to successfully connect to your yum repositories? Do you have a proxy server configured?

Code: Select all

PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known

Lastly, you might want to increase the max open file limit on the server.

Code: Select all

200820  8:05:59 [Warning] Could not increase number of max_open_files to more than 1024 (request: 70011)

Take a look at the following pages for instructions.
https://unix.stackexchange.com/question ... -than-1024

Try running the upgrade again after running the db repair script and verifying the network, if it fails please post the full output of the upgrade log to the ticket.

[joash0809]

Hi Benjamin,

1. We auto schedule the db repair script everyday in the morning and it run successfully prior to the upgrade.

nagiosdb repair.PNG

2. We are using proxy where the proxy will resolve the target FQDN.

3. May we also know for ERROR 2005 (provided in the initial screenshot), from which configuration file are the credentials read from? This is so that we can also verify that the provided credentials are correct.

4. Under the circumstances that the remote DB is MySQL Enterprise (Version 14.14 Distribution 5.7.28) and the local DB is MySQL (Version 15.1 Distribution 5.5.64-MariaDB), are there any special/additional steps required when performing the Nagios version upgrade?

5. Will there be any issues if we were to perform the database inload prior to performing the upgrade? I found the steps in the URL below:
https://support.nagios.com/forum/viewto ... 16&t=55506

Last but not least, we will check on the open files settings that needs to be changed from 1024 to 2048 and revert as to whether or not the changes can be made.

Thanks.
Best Regards,
Joash

[benjaminsmith]

Hi Joash,

I'm not seeing a repair output fro the nagios database, which is the one with the crashed database tables. Can you verify that this was successfully repaired?

Regarding the proxy server, that is probably why I'm seeing those messages in the log. Take a look at the following kb article for configuring the proxy component to allow update checks to work.

Nagios XI - Problems Using Nagios XI With Proxies

Regarding the database error 2005, the upgrade script is not connecting to the nagiosql database.

I tracked down the portion of the code where it's failing.

Code: Select all

		# Remove old references to failure_prediction_enabled from nagiosql DB when we upgrade to Core 4
		if [ ! -z $cfg__db_info__nagiosql__db ]; then
			echo "Removing depricated failure_prediction_enabled from NagiosQL"
			echo "delete from tbl_variabledefinition where name='failure_prediction_enabled';"|mysql -h "$cfg__db_info__nagiosql__dbserver" -u "$cfg__db_info__nagiosql__user" -p"$cfg__db_info__nagiosql__pwd" "$cfg__db_info__nagiosql__db"
		fi

The connections settings can be found in usr/local/nagiosxi/html/config.inc.php. Can your try testing the connection from XI with a simple telnet test.

Code: Select all

 telnet <ipaddress> <port>

Also, since you're running a later version of MySQL, take a look at the following kb article to make sure STRICT_TRANS_TABLES is not enabled.

Nagios XI - STRICT_TRANS_TABLES

Lastly are you planning to bring the remote database back or change the local db to MySQL?

[joash0809]

I'm not seeing a repair output fro the nagios database, which is the one with the crashed database tables. Can you verify that this was successfully repaired?

log_entries.PNG

please find the output for the database that error out

Regarding the proxy server, that is probably why I'm seeing those messages in the log. Take a look at the following kb article for configuring the proxy component to allow update checks to work.

The connections settings can be found in usr/local/nagiosxi/html/config.inc.php. Can your try testing the connection from XI with a simple telnet test.

i tested the telnet to the remote database from nagios and it shows connected

telnet.PNG

Code: Select all

 # Remove old references to failure_prediction_enabled from nagiosql DB when we upgrade to Core 4
      if [ ! -z $cfg__db_info__nagiosql__db ]; then
         echo "Removing depricated failure_prediction_enabled from NagiosQL"
         echo "delete from tbl_variabledefinition where name='failure_prediction_enabled';"|mysql -h "$cfg__db_info__nagiosql__dbserver" -u "$cfg__db_info__nagiosql__user" -p"$cfg__db_info__nagiosql__pwd" "$cfg__db_info__nagiosql__db"
      fi

for the above code, may i know from which script was this invoked? because we performed a select statement against the table tbl_variabledefinition, there was no entry with the name 'failure_prediction_enabled'


Lastly are you planning to bring the remote database back or change the local db to MySQL?

we want to bring the remote DB back in and convert the local DB from MariaDB to Mysql.

In addition to the above mentioned, the logs submitted via the system profile before this was dated august 20th, may i know how was it derived that an issue with the nagios_logentries tables that contributed to the failure of the upgrade. Reason as to why i'm asking, is because the upgrade was done on the 19th while the error occurred on the 20th.

[tgriep]

The output provided, it looks like the logentries table was repaired.

The script tries to remove the following from the nagios database is the nagios database version is less than 2.1.2.
Removing depricated failure_prediction_enabled from NagiosQL

Can you run the following and post the output so see can see if is was updated?

Code: Select all

echo 'select * from nagios_dbversion;' |mysql  -t -u root -pnagiosxi nagios

This is the script that it ran.

Code: Select all

/tmp/nagiosxi/subcomponents/ndo/post-upgrade

The MYSQL table errors for the logentries table did not cause the upgrade failure, the script could not connect to the off-loaded nagiosql database was the cause of the upgrade failure.

To test the connection to the remote database from the nagios server, run the following command which should show the tables in the nagiosql database.

Code: Select all

echo 'show tables;' |mysql  -t -u nagiosql -pnagios --database nagios -h 10.102.10.254 -P 3301

In the command, I used the default password, which is nagios. If you changed it, update the command and after the -p option, change the password.

[jliewhm]

Attached is the screenshot of successful connection to the remote DB:

connection_to_db.JPG

To add to what was mentioned in the threads prior to this, the current setup for nagios is one in which the Nagios DB residing in the Nagios server is of type MariaDB v5.5 while the remote DB which consists of the Nagiosxi and Nagiosql DBs are of type MySQL 5.7. In the event that we would like to inload the remote DB prior to the version upgrade, can you kindly confirm if the below steps are the correct steps to inload the remote DB and proceed with the version upgrade? The reason as to why I am asking is because the only way i can see this working is by moving the remote back to local.

Also, In our environment we do have a proxy configured but what i don't understand is why the upgrade failed but we are able to access the remote DB. Apart from that, from the 3 configuration files below, I can see that the information on the remote DB is correct.

ndo2db.cfg, config.inc.php, settings.php

The I would like to try steps are:
1. Take a database dump of both the local (Nagios DB - MariaDB v5.5) and remote DB (Nagiosxi and Nagiosql DBs both are MySQL v5.7).
2. Upgrade local DB from MariaDB v5.5 to Maria 10.3 - This is to ensure compatibility between the 2 DBs before inloading the remote DB.
3. Take a database dump of the local DB (i.e. now in MariaDB v10.3)
4. Uninstall the local DB (i.e. MariaDB v10.3)
5. Install MySQL v5.7 locally (i.e. in Nagios Prod server). Which is identical to the current version of MySQL in the remote DB.
6. Create 3 new databases in Nagios Prod server i.e Nagios, Nagiosxi and Nagiosql databases.
7. Inload (i.e. restore the data) from the dumps taken in step 1 (for Nagiosxi and Nagiosql DBs) and 3 (for MariaDB v10.3) into the respective newly created DBs.
8. Reconfigure Nagios to connect to the newly created DBs from step 6.
9. Proceed with the upgrade of Nagios from v5.4.13 to 5.6.7.

My concern is, since NagiosXI came together with MariaDB by default, will there be any issues if we were to proceed with the above steps?

Thank You.
Best Regards,
Jason

[safuanmansor]

Hi Support,

Apart from the above question by Jason. The reason we are plan to maintain the mysql is because we still unable to migrate the nagiosql db from mysql to maria event after upgrading the maria to the the higher version.
https://mariadb.com/kb/en/mariadb-vs-my ... atibility/
https://support.nagios.com/forum/viewto ... 16&t=55365

Do you have any other suggestion for this? Can import config files be and alternative for this? What possible impact will be to the current data/services/perfgraph/etc ?

Thanks,
Safuan