db stopped after upgrading to 5.7.5

[clayye]

Nagiosxi 5.6.12-2 on RHEL 7.9 VM (on vmware ), upgrade to 5.7.5-1 via yum update from rpm pakcages at repo.nagios.com ,
after upgrade, Monitoring Engine Process stopped , Admin - System status showed "Database backend " stopped.

made sure:
$ grep ^broker_module /usr/local/nagios/etc/nagios.cfg
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

checked nagios.log , showing :
[1611648584] NDO-3: ndo_handle_service_status(ndo-handlers.c:979): Could not reconnect to MySQL database
[1611648585] NDO-3: Unable to prepare statement for query (27): Unknown column 'check_options' in 'field list'
[1611648585] NDO-3: Error preparing statements

followed https://support.nagios.com/forum/viewto ... 16&t=60062 , nagios.log showed:

[1611648612] NDO-3: NDO 3.0.4 (c) Copyright 2009-2020 Nagios - Nagios Core Development Team
[1611648612] NDO-3: Database initialized
[1611648612] NDO-3: Database initialized
[1611648612] NDO-3: Callbacks registered
[1611648612] NDO-3: Callbacks registered
[1611648613] NDO-3: Database initialized

but the Database backkend under Admin - system status still showing stopped.

2021-01-25_23-34-27.png

help apprecated .

[dchurch]

ndo2db, (sometimes called the Database Backend) is no longer needed in Nagios XI.

ndo2db is our older technology that basically listens on a UNIX socket for database inserts, then handles the actual insertion into the database. It has limits, being that it runs into issues when it tries to insert more than the database can handle. In newer versions (Nagios XI 5.7.0 and later), this was replaced by just writing directly to the database from the Nagios worker threads. In addition to being able to handle more database inserts, this resulted in an overall performance boost, too.

Please run the following commands as root to take care of disabling ndo2db.

Code: Select all

systemctl stop ndo2db
systemctl disable ndo2db

Can you check if the ndo2db broker was actually disabled during the upgrade by pasting the output of the following command?

Code: Select all

grep broker_module /usr/local/nagios/etc/nagios.cfg

I'll also recommend running the database repair script. Run the following as root from the terminal.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

[clayye]

currently I have this in nagios.cfg

# NDOUtils module
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
#broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg line should be commented also ?

Thanks

[clayye]

I commented out this line in nagios.cfg
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

and ran /usr/local/nagiosxi/scripts/repair_databases.sh, and restarted nagios. Database backend still showing stopped. thanks

[clayye]

sorry for the duplicate post at https://support.nagios.com/forum/viewto ... 16&t=60062, didn't see my post after I submitted ( after waiting for while) .

I also tried the steps in https://support.nagios.com/forum/viewto ... 16&t=60062 , restarted all services and reboot nagios vm.

the "last check state " for all services stopped atLast Check: 2021-01-26 21:44:56 (GMT time) now for all services

[clayye]

checking https://support.nagios.com/forum/viewto ... 16&t=59472 ,
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg should be in nagios.cfg , I put it back and restarted nagios. service check last update time is updated now . but the backend db is still stopped. thanks.

[dchurch]

Can you please paste the output from the following command?

Code: Select all

mysql -unagiosxi -pn@gweb nagiosxi <<< "select * from xi_sysstat where metric = 'daemons'"

[clayye]

> mysql -unagiosxi -pn@gweb nagiosxi <<< "select * from xi_sysstat where metric = 'daemons'"
sysstat_id metric value update_time
6 daemons a:3:{s:10:"nagioscore";a:4:{s:6:"daemon";s:6:"nagios";s:6:"output";s:87:" ??27113 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg";s:11:"return_code";i:0;s:6:"status";i:0;}s:3:"pnp";a:4:{s:6:"daemon";s:4:"npcd";s:6:"output";s:89:" ??1845 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg";s:11:"return_code";i:0;s:6:"status";i:0;}s:8:"ndoutils";a:4:{s:6:"daemon";s:6:"ndo2db";s:6:"output";s:0:"";s:11:"return_code";i:4;s:6:"status";i:1;}} 2021-01-28 05:23:22

[dchurch]

Can you inspect the contents of /usr/local/nagiosxi/cron/sysstat.php? The code shouldn't mention ndo2db at all in 5.7.0 and later.

Please post the output from the following commands:

Code: Select all

grep -25 'daemons =' /usr/local/nagiosxi/cron/sysstat.php

Code: Select all

grep ndo2db /usr/local/nagiosxi/cron/sysstat.php

[clayye]

> grep -25 'daemons =' /usr/local/nagiosxi/cron/sysstat.php

Code: Select all

      'bytes_processed' => $info['bytes_processed'],
        'entries_processed' => $info['entries_processed'],
        'connect_time' => $info['connect_time'],
        'disconnect_time' => $info['disconnect_time']
    );

    if ($logging) {
        echo "DB BACKEND:\n";
        print_r($dbbe);
    }

    $sdata = serialize($dbbe);
    update_systat_value("dbbackend", $sdata);
}

function get_daemon_status()
{
    global $logging;
    global $cfg;

    $script_dir = $cfg["script_dir"];
    if (substr($script_dir, -1) == '/') {
        $script_dir = substr($script_dir, 0, strlen($script_dir) - 1);
    }

    $daemons = array(
        "nagioscore" => array(
            "daemon" => "nagios",
            "output" => "",
            "return_code" => 0,
            "status" => SUBSYS_COMPONENT_STATUS_UNKNOWN
        ),
        "pnp" => array(
            "daemon" => "npcd",
            "output" => "",
            "return_code" => 0,
            "status" => SUBSYS_COMPONENT_STATUS_UNKNOWN
        ),
        "ndoutils" => array(
            "daemon" => "ndo2db",
            "output" => "",
            "return_code" => 0,
            "status" => SUBSYS_COMPONENT_STATUS_UNKNOWN
        )
    );

    foreach ($daemons as $dname => $darr) {

        $cmdline = sprintf("sudo %s/manage_services.sh status %s", $script_dir, $darr["daemon"]);

        if ($logging) {

> grep ndo2db /usr/local/nagiosxi/cron/sysstat.php

Code: Select all

            "daemon" => "ndo2db",
        // If the daemon is ndo2db and offloaded, we overwrite the
        if ($return_code != 0 && $darr["daemon"] == "ndo2db") {
            if (!empty($cfg["ndo2db_offloaded"])) {