Nagios xi upgrade issue.

[exp-user1]

Hello,

I did the upgrade of Ngaios from version 5.6.12 to 5.7.3 and faced an issue during upgrade:

Shared module ndo.so installed
/usr/local/nagios/etc/nagios.cfg configured to use NDO
UPGRADE: POST-UPGRADE: NDO post upgrade started...
2.0.1
Removing depricated failure_prediction_enabled from NagiosQL
copying updated mysql-upgrade-2.0.0.sql
Current database version: 2.0.1
** DB upgrade required for 2.1.0
Using mysql-upgrade-2.1.0.sql for upgrade...

I followed the link
https://support.nagios.com/kb/article.php?id=139

then redo the upgrade again and it stuck again before i restarted mariadb and it continue "successfully":

Shared module ndo.so installed
/usr/local/nagios/etc/nagios.cfg configured to use NDO
UPGRADE: POST-UPGRADE: NDO post upgrade started...
Performing upgrade...
> Upgrading from version 2.1.3 (/tmp/nagiosxi/subcomponents/ndo/ndo-3.0.3/db/upgrade-from-2.1.3.sql)
ERROR 1053 (08S01) at line 135: Server shutdown in progress
> Upgrading from version 3.0.0 (/tmp/nagiosxi/subcomponents/ndo/ndo-3.0.3/db/upgrade-from-3.0.0.sql)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
> Upgrading from version 3.0.1 (/tmp/nagiosxi/subcomponents/ndo/ndo-3.0.3/db/upgrade-from-3.0.1.sql)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
> Updating dbversion table
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Database upgrade is complete!
UPGRADE: POST-UPGRADE: NDO post upgraded OK.
UPGRADE: NDO upgraded OK.
UPGRADE: Nagios Plugins is being upgraded...

The problem now is the last check is not updated any more since the time of upgrade.
your quick support is appreciated (Prod ENV)

Thanks.

[exp-user1]

please check attached image for last check status.

[exp-user1]

updated info:

last check status is not updated since upgrade on Nagios xi , but updated on Nagios core!!

[benjaminsmith]

Hi,

So it looks ndo is unable to make a database connection after upgrading. Do you have an offloaded database in this setup?

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2

A couple items to check. Open up /usr/local/nagios/etc/nagios.cfg file and make sure ndo3 is getting loaded. This line should be uncommented.

Code: Select all

broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

and this line should be commented.

Code: Select all

#broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Aslo, make sure you have the correct db settings in the following file.

Code: Select all

/usr/local/nagiosxi/html/config.inc.php

If that does not resolve the issue. Please PM me your system profile. Thanks, Benjamin

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.

[exp-user1]

Hi benjamin,

ndo3 settings are good:

[root@olexpnagiosp01 ~]# grep broker /usr/local/nagios/etc/nagios.cfg
# Commented out by NDO 'make install-broker-line' on Sat Sep 19 11:47:34 EDT 2020
# Commented out by NDO 'make install-broker-line' on Sat Sep 19 12:45:44 EDT 2020
##broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg
event_broker_options=-1
# Added by NDO 'make install-broker-line' on Sat Sep 19 11:47:34 EDT 2020
# Commented out by NDO 'make install-broker-line' on Sat Sep 19 12:45:44 EDT 2020
#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
# Added by NDO 'make install-broker-line' on Sat Sep 19 12:45:44 EDT 2020
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
[root@olexpnagiosp01 ~]#

I am not sure about DB setting in /usr/local/nagiosxi/html/config.inc.php . I don't think they are touched, cause nagios status shows all green signs.

system profile will be sent as PM.


My major question is why upgrade stuck for ever with the following output?:

Shared module ndo.so installed
/usr/local/nagios/etc/nagios.cfg configured to use NDO
UPGRADE: POST-UPGRADE: NDO post upgrade started...
2.0.1
Removing depricated failure_prediction_enabled from NagiosQL
copying updated mysql-upgrade-2.0.0.sql
Current database version: 2.0.1
** DB upgrade required for 2.1.0
Using mysql-upgrade-2.1.0.sql for upgrade...

The second question is, after managing to finish upgrade and things seemed good, we realize that last checks on all services are not update since the upgrade. any idea?

[benjaminsmith]

Hi,

Thank you for the sysetem profile. The database log is showing the following errors related to the nagios or ndo databse.

200921 14:10:36 [Warning] Aborted connection 47 to db: 'nagios' user: 'ndoutils' host: 'localhost' (Got an error reading communication packets)
200921 14:23:35 [Warning] Aborted connection 1060 to db: 'nagios' user: 'ndoutils' host: 'localhost' (Got an error reading communication packets)
200921 14:23:35 [Warning] Aborted connection 1058 to db: 'nagios' user: 'ndoutils' host: 'localhost' (Got an error reading communication packets)
200921 14:23:35 [Warning] Aborted connection 1059 to db: 'nagios' user: 'ndoutils' host: 'localhost' (Got an error reading communication packets)
200921 18:20:02 [Warning] Aborted connection 13769 to db: 'nagios' user: 'ndoutils' host: 'localhost' (Got an error reading communication packets)
200921 18:20:02 [Warning] Aborted connection 13768 to db: 'nagios' user: 'ndoutils' host: 'localhost' (Got an error reading communication packets)
200921 18:20:02 [Warning] Aborted connection 13770 to db: 'nagios' user: 'ndoutils' host: 'localhost' (Got an error reading communication packets)

Please try to add the following entry to your /etc/my.cnf file.

Code: Select all

[mysqld]
max_allowed_packet=256M

Then restart the databse and the nagios service and then let me know if the services are updating.

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mariadb
systemctl start nagios
systemctl start npcd
systemctl start crond

[exp-user1]

Hi,

Those warnings are old ones, before upgrade.
Now we cleaned up and reinstalled the previous version of Nagios and restored DB copy before upgrade.

But still wondering why Nagios upgrade process stuck on "Using mysql-upgrade-2.1.0.sql for upgrade..."
we can't upgrade now and face the same scenario.

Thanks.

[benjaminsmith]

Hi,

Ok, so the error below suggests that the upgrade script was unable to update the database tables for ndo3.

> Upgrading from version 3.0.0 (/tmp/nagiosxi/subcomponents/ndo/ndo-3.0.3/db/upgrade-from-3.0.0.sql)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

...and this error message in the nagios log.

[1600726800] NDO-3: Unable to prepare statement for query (28): Unknown column 'check_options' in 'field list'
[1600726800] NDO-3: Error preparing statements
[1600726800] NDO-3: ndo_handle_service_status(ndo-handlers.c:953): Could not reconnect to MySQL database

If you've re-installed the old version it does not require the check_options field, but if you upgrade again. You can the following to manually update the tables.

1. Connect to the nagios database.

Code: Select all

mysql -pnagiosxi nagios

2. Run the following SQL commands to update the tables for ndo3.

Code: Select all

# Add the column 'check_options' to nagios_servicestatus if it doesn't already exist
SET @exist := (SELECT COUNT(*) FROM `information_schema`.`columns` WHERE table_name = 'nagios_servicestatus' AND column_name = 'check_options');
SET @sqlstmt := if( @exist <= 0,
'ALTER TABLE `nagios_servicestatus` ADD `check_options` smallint(6) NOT NULL default ''0'' AFTER `check_type`',
'select ''INFO: Column already exists.''');
PREPARE stmt FROM @sqlstmt;
EXECUTE stmt;

# Add the column 'check_options' to nagios_hoststatus if it doesn't already exist
SET @exist := (SELECT COUNT(*) FROM `information_schema`.`columns` WHERE table_name = 'nagios_hoststatus' AND column_name = 'check_options');
SET @sqlstmt := if (@exist <= 0,
'ALTER TABLE `nagios_hoststatus` ADD `check_options` smallint(6) NOT NULL default ''0'' AFTER `check_type`',
'SELECT ''INFO: Column already exists.''');
PREPARE stmt FROM @sqlstmt;
EXECUTE stmt;

ALTER TABLE nagios_objects MODIFY name1 VARCHAR(1023) CHARACTER SET utf8 COLLATE utf8_bin;
ALTER TABLE nagios_objects MODIFY name2 VARCHAR(1023) CHARACTER SET utf8 COLLATE utf8_bin;

3. Exit Mysql

Code: Select all

exit;

4. Restart the Nagios service.

Code: Select all

systemctl restart nagios

As always, we recommend making changes on a test server first or take backup or VM snapshot before upgrading or making any changes.

Backing Up And Restoring Your Nagios XI System