Database problems during upgrade to 5.7.0

[jweijters]

During the upgrade from NagiosXI 5.6.14 to 5.7.0 it stopped at
UPGRADE: POST-UPGRADE: NDO post upgrade started...
Performing upgrade...

with a message:
Your dbversion is 1.5.2, and our minimum supported upgrade version is
Upgrade your version of NDOUtils to 2.0.1 before attempting this upgrade

However we allready are running NDOutils 2.1.3

Code: Select all

/usr/local/nagios/bin/ndo2db --version

NDO2DB 2.1.3
Copyright (c) 2009 Nagios Core Development Team and Community Contributors
Copyright (c) 2005-2008 Ethan Galstad
Last Modified: 2017-04-13
License: GPL v2

It seems that our database is still at level 1.5.2

Code: Select all

MariaDB [nagios]> select * from nagios_dbversion;
+----------+---------+
| name     | version |
+----------+---------+
| ndoutils | 1.5.2   |
+----------+---------+
1 row in set (0.00 sec)

In my investigation I find that the upgrade to NagiosXI 5.7.0 needs a database of at least version 2.0.1

In our test environment we succeeded the upgrade of the database by
running the upgradedb script in the ndoutils-2.1.3/db directory and fixing the following problems by hand:
removing the `minimum_importance` from the `nagios_contacts` table
and adjusting the values of `entry_time_usec` in the `nagios_logentries` table so we didn't have duplicate

This gave us an 2.1.3 database. after wich we could run the Nagiosxi upgrade successfully.
However during starting of the Nagios we got errors in the nagios.log telling

[1591796460] NDO-3: Unable to prepare statement for query (27): Unknown column 'check_options' in 'field list'
[1591796460] NDO-3: Unable to prepare statement for query (28): Unknown column 'check_options' in 'field list'

It seemed the database upgrade to 3.0.0 wasn't successful these columns weren't in the database
We manualy ran the upgrade-from-2.1.3.sql script to the database, Now it looks like Nagios is running again.

however we do get NDO-3 errors in the nagios.log::
[1591799013] NDO-3: ndo_get_object_id_name1() - name1 is null
[1591799013] NDO-3: ndo_get_object_id_name1() - name1 is null
[1591799013] NDO-3: ndo_get_object_id_name1() - name1 is null
[1591799013] NDO-3: ndo_get_object_id_name1() - name1 is null
[1591799013] NDO-3: ndo_get_object_id_name1() - name1 is null

This upgrade procedure is not what we want to run in production.

also we want this last error to be fixed.

can you please develop a correct upgrade procedure?

how can we fix these last database errors?

Kind regards,

Joris Weijters

[ssax]

I've reported this to the QA team.

The developers said:

name1 can never be null, we run this by default to clear out nulls when we do the upgrade:

Code: Select all

UPDATE `nagios_objects` SET `name1` = '' WHERE `name1` IS NULL;
UPDATE `nagios_objects` SET `name2` = '' WHERE `name2` IS NULL;

Maybe that was missed somehow, you can try running these commands to resolve it:

Code: Select all

mysql -uroot -pnagiosxi -h 127.0.0.1 nagios -e "UPDATE nagios_objects SET name1 = '' WHERE name1 IS NULL;"
mysql -uroot -pnagiosxi -h 127.0.0.1 nagios -e "UPDATE nagios_objects SET name2 = '' WHERE name2 IS NULL;"

Let us know if that works for you.

[jweijters]

Before adjusting the database I first tried to select this data,

Code: Select all

MariaDB [nagios]> SELECT * FROM nagios_objects WHERE name1 is NULL;
Empty set (0.00 sec)

MariaDB [nagios]> SELECT * FROM nagios_objects WHERE name2 is NULL;
Empty set (0.00 sec)

MariaDB [nagios]>

This delivers an empty set, so it looks like there are no records WHERE name1 or name2 is NULL

[ssax]

You're still getting those errors in the nagios.log after the upgrade? Is there any other information listed around those messages?

[jweijters]

We decided to restore our test environment to the previous level 5.6.14, because the development and testing of new monitors/events cutomers and hosts has to continue.

I saved the "failing" NagiosXI 5.7.0 server.
This is the log around the error message:

Code: Select all

[1591851419] NDO-3: NDO 3.0.0 (c) Copyright 2009-2020 Nagios - Nagios Core Development Team
[1591851419] NDO-3: Database initialized
[1591851419] NDO-3: Database initialized
[1591851419] NDO-3: Callbacks registered
[1591851419] NDO-3: Callbacks registered
[1591851419] Event broker module '/usr/local/nagios/bin/ndo.so' initialized successfully.
[1591851419] mod_gearman: initialized version 3.0.7 (libgearman 0.33)
[1591851419] Event broker module '/usr/lib64/mod_gearman/mod_gearman_nagios4.o' initialized successfully.

then we get some warnings about
[1591851420] Warning: Duplicate definition found for service

then we get

Code: Select all

[1591851420] WARNING: RLIMIT_NPROC is 31193, total max estimated processes is 172660! You should increase your limits (ulimit -u, or limits.conf)
[1591851421] NDO-3: Database initialized
[1591851423] NDO-3: ndo_get_object_id_name1() - name1 is null
[1591851423] NDO-3: ndo_get_object_id_name1() - name1 is null
.
.
[1591851436] NDO-3: ndo_get_object_id_name1() - name1 is null
[1591851436] NDO-3: ndo_get_object_id_name1() - name1 is null
[1591851436] NDO-3: ndo_get_object_id_name1() - name1 is null
[1591851520] Successfully launched command file worker with pid 2952
[1591851546] SERVICE ALERT: e

We get 3623 of lines: NDO-3: ndo_get_object_id_name1() - name1 is null
This is the same amount as we have hosts configured.

Kind regards,

Joris Weijters

[ssax]

The developers found this was related to having a NULL timeperiod that wasn't being handled properly. They said this should be fixed in XI 5.7.1 (released already).