Page 1 of 2
db stopped after upgrading to 5.7.5
Posted: Tue Jan 26, 2021 3:40 am
by clayye
Nagiosxi 5.6.12-2 on RHEL 7.9 VM (on vmware ), upgrade to 5.7.5-1 via yum update from rpm pakcages at repo.nagios.com ,
after upgrade, Monitoring Engine Process stopped , Admin - System status showed "Database backend " stopped.
made sure:
$ grep ^broker_module /usr/local/nagios/etc/nagios.cfg
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
checked nagios.log , showing :
[1611648584] NDO-3: ndo_handle_service_status(ndo-handlers.c:979): Could not reconnect to MySQL database
[1611648585] NDO-3: Unable to prepare statement for query (27): Unknown column 'check_options' in 'field list'
[1611648585] NDO-3: Error preparing statements
followed
https://support.nagios.com/forum/viewto ... 16&t=60062 , nagios.log showed:
[1611648612] NDO-3: NDO 3.0.4 (c) Copyright 2009-2020 Nagios - Nagios Core Development Team
[1611648612] NDO-3: Database initialized
[1611648612] NDO-3: Database initialized
[1611648612] NDO-3: Callbacks registered
[1611648612] NDO-3: Callbacks registered
[1611648613] NDO-3: Database initialized
but the Database backkend under Admin - system status still showing stopped.
2021-01-25_23-34-27.png
help apprecated .
Re: db stopped after upgrading to 5.7.5
Posted: Tue Jan 26, 2021 3:28 pm
by dchurch
ndo2db, (sometimes called the Database Backend) is no longer needed in Nagios XI.
ndo2db is our older technology that basically listens on a UNIX socket for database inserts, then handles the actual insertion into the database. It has limits, being that it runs into issues when it tries to insert more than the database can handle. In newer versions (
Nagios XI 5.7.0 and later), this was replaced by just writing directly to the database from the Nagios worker threads. In addition to being able to handle more database inserts, this resulted in an overall performance boost, too.
Please run the following commands as root to take care of disabling ndo2db.
Code: Select all
systemctl stop ndo2db
systemctl disable ndo2db
Can you check if the ndo2db broker was actually disabled during the upgrade by pasting the output of the following command?
Code: Select all
grep broker_module /usr/local/nagios/etc/nagios.cfg
I'll also recommend running the database repair script. Run the following as root from the terminal.
Code: Select all
/usr/local/nagiosxi/scripts/repair_databases.sh
Re: db stopped after upgrading to 5.7.5
Posted: Tue Jan 26, 2021 4:45 pm
by clayye
currently I have this in nagios.cfg
# NDOUtils module
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
#broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg line should be commented also ?
Thanks
Re: db stopped after upgrading to 5.7.5
Posted: Tue Jan 26, 2021 4:50 pm
by clayye
I commented out this line in nagios.cfg
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
and ran /usr/local/nagiosxi/scripts/repair_databases.sh, and restarted nagios. Database backend still showing stopped. thanks
Re: db stopped after upgrading to 5.7.5
Posted: Tue Jan 26, 2021 5:03 pm
by clayye
sorry for the duplicate post at
https://support.nagios.com/forum/viewto ... 16&t=60062, didn't see my post after I submitted ( after waiting for while) .
I also tried the steps in
https://support.nagios.com/forum/viewto ... 16&t=60062 , restarted all services and reboot nagios vm.
the "last check state " for all services stopped at
Last Check: 2021-01-26 21:44:56 (GMT time) now for all services
Re: db stopped after upgrading to 5.7.5
Posted: Tue Jan 26, 2021 5:32 pm
by clayye
checking
https://support.nagios.com/forum/viewto ... 16&t=59472 ,
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg should be in nagios.cfg , I put it back and restarted nagios. service check last update time is updated now . but the backend db is still stopped. thanks.
Re: db stopped after upgrading to 5.7.5
Posted: Wed Jan 27, 2021 12:46 pm
by dchurch
Can you please paste the output from the following command?
Code: Select all
mysql -unagiosxi -pn@gweb nagiosxi <<< "select * from xi_sysstat where metric = 'daemons'"
Re: db stopped after upgrading to 5.7.5
Posted: Thu Jan 28, 2021 12:23 am
by clayye
> mysql -unagiosxi -pn@gweb nagiosxi <<< "select * from xi_sysstat where metric = 'daemons'"
sysstat_id metric value update_time
6 daemons a:3:{s:10:"nagioscore";a:4:{s:6:"daemon";s:6:"nagios";s:6:"output";s:87:" ??27113 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg";s:11:"return_code";i:0;s:6:"status";i:0;}s:3:"pnp";a:4:{s:6:"daemon";s:4:"npcd";s:6:"output";s:89:" ??1845 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg";s:11:"return_code";i:0;s:6:"status";i:0;}s:8:"ndoutils";a:4:{s:6:"daemon";s:6:"ndo2db";s:6:"output";s:0:"";s:11:"return_code";i:4;s:6:"status";i:1;}} 2021-01-28 05:23:22
Re: db stopped after upgrading to 5.7.5
Posted: Thu Jan 28, 2021 12:35 pm
by dchurch
Can you inspect the contents of
/usr/local/nagiosxi/cron/sysstat.php? The code shouldn't mention ndo2db at all in 5.7.0 and later.
Please post the output from the following commands:
Code: Select all
grep -25 'daemons =' /usr/local/nagiosxi/cron/sysstat.php
Code: Select all
grep ndo2db /usr/local/nagiosxi/cron/sysstat.php
Re: db stopped after upgrading to 5.7.5
Posted: Thu Jan 28, 2021 4:07 pm
by clayye
> grep -25 'daemons =' /usr/local/nagiosxi/cron/sysstat.php
Code: Select all
'bytes_processed' => $info['bytes_processed'],
'entries_processed' => $info['entries_processed'],
'connect_time' => $info['connect_time'],
'disconnect_time' => $info['disconnect_time']
);
if ($logging) {
echo "DB BACKEND:\n";
print_r($dbbe);
}
$sdata = serialize($dbbe);
update_systat_value("dbbackend", $sdata);
}
function get_daemon_status()
{
global $logging;
global $cfg;
$script_dir = $cfg["script_dir"];
if (substr($script_dir, -1) == '/') {
$script_dir = substr($script_dir, 0, strlen($script_dir) - 1);
}
$daemons = array(
"nagioscore" => array(
"daemon" => "nagios",
"output" => "",
"return_code" => 0,
"status" => SUBSYS_COMPONENT_STATUS_UNKNOWN
),
"pnp" => array(
"daemon" => "npcd",
"output" => "",
"return_code" => 0,
"status" => SUBSYS_COMPONENT_STATUS_UNKNOWN
),
"ndoutils" => array(
"daemon" => "ndo2db",
"output" => "",
"return_code" => 0,
"status" => SUBSYS_COMPONENT_STATUS_UNKNOWN
)
);
foreach ($daemons as $dname => $darr) {
$cmdline = sprintf("sudo %s/manage_services.sh status %s", $script_dir, $darr["daemon"]);
if ($logging) {
> grep ndo2db /usr/local/nagiosxi/cron/sysstat.php
Code: Select all
"daemon" => "ndo2db",
// If the daemon is ndo2db and offloaded, we overwrite the
if ($return_code != 0 && $darr["daemon"] == "ndo2db") {
if (!empty($cfg["ndo2db_offloaded"])) {