Message queue disappeared on NagiosXI post upgrade to 5.7.x

[Sampath.Basireddy]

We recently upgraded our NagiosXI instance to 5.7.2 and today to 5.7.5, but since after upgrade, we are noticing multiple issues.

There is no message queue:

Code: Select all

root@devnagios1:[/]: ipcs -q

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

root@devnagios1:[/]:

Also, the Monitoring Engine Event Queue & Monitoring Engine Check Statistics are not showing any data in "Monitoring Engine Status" page.

When we create a new service, it shows in Nagios Core, but not in NagiosXI.

Please suggest.

[dchurch]

This is to be expected because Nagios XI uses NDOUtils instead of the external listener process ndo2db. Let me explain why this is better.

ndo2db is our older technology that basically listens on a UNIX socket for database inserts, then handles the actual insertion into the database. It has limits, being that it runs into issues when it tries to insert more than the database can handle. In newer versions (Nagios XI 5.7.0 and later), this was replaced by just writing directly to the database from the Nagios worker threads. In addition to being able to handle more database inserts, this resulted in an overall performance boost, too.

The reason the message queue is empty is because there aren't tons of pending database writes waiting in the socket IPC formerly used by ndo2db.

[Sampath.Basireddy]

Thanks @dchurch for the response.

But why are we seeing this strange behavior after upgrade? New service created does not reflect on node in XI, but does in Nagios Core.

Similarly, Nagios Service keeps going down unexpectedly and after starting the service, it takes about an hour or so to come back to normal and show the actual node or service status. This wasn't the case before upgrade. Please suggests.

[dchurch]

If you're saying that you modified a service in Core and it's not reflecting back up into Nagios XI, that's to be expected due to how Nagios Core and Nagios XI interact; Nagios XI pushes configuration to Core, not the other way around.

If that's not what you're saying, this sounds like a database issue on the face of it. Try running the database repair script, and let me know if that is successful. Here's how you do that:

Running the Database Repair Script

Run the following as root from the terminal.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

See here for complete instructions: run the database repair

Try this if it doesn't work: Downgrading NDOUtils

If that doesn't fix your issue, what you can try doing is downgrading your version of NDOUtils.

Code: Select all

systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db

Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is uncommented:

Code: Select all

broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Make sure this line is commented:

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service:

Code: Select all

systemctl start nagios

The instructions are different if you have an offloaded database (i.e. a database that isn't on the same machine that XI is running on).

[Sampath.Basireddy]

We are not creating/modifying the service in Core. It is in NagiosXI.

DB Repair did not help, so we did go ahead and downgraded NDOUtils which fixed the issue and everything came back to normal.

Is this a temp solution or is Nagios coming up with a new release to address this issue?

The reason I ask is, the issue we faced is on our Dev Instance of NagiosXI and we are yet to upgrade our Prod NagiosXI to version 5.7.x. Wanted to make sure before we go ahead with our Prod instance, so we don't break the production monitoring.

Please suggest.

[ssax]

XI 5.8 is coming out in a few days, this would be a temporary fix but when you upgrade to XI 5.8 it will leave your NDO2DB downgraded, if you want to upgrade to the new updated version of NDO3 that comes with XI 5.8 you can do this after upgrading:
- NOTE: Since 5.8 isn't released, I'm just guessing at what the download URL is going to be

Code: Select all

rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.8.0.tar.gz
tar zxf xi-5.8.0.tar.gz
cd /tmp/nagiosxi
./init.sh
cd /tmp/nagiosxi/subcomponents/ndo
./install
systemctl restart nagios

As an alternative (you can even do this on your existing system if you want to test the NDO3 version that will be coming in 5.8 on your system) you can take the attached zip file, transfer it to your XI server, and then run these commands as root:

Code: Select all

unzip ndo-master.zip
cd ndo-master
./configure
make all
make install

Comment out the old broker line in your /usr/local/nagios/etc/nagios.cfg by changing it from:

Code: Select all

broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

To:

Code: Select all

#broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Change the new broker line from:

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

To:

Code: Select all

broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Save the file and then restart nagios.

Code: Select all

systemctl restart nagios

Then apply configuration and validate.