Page 1 of 1
Message queue disappeared on NagiosXI post upgrade to 5.7.x
Posted: Thu Jan 07, 2021 2:02 pm
by Sampath.Basireddy
We recently upgraded our NagiosXI instance to 5.7.2 and today to 5.7.5, but since after upgrade, we are noticing multiple issues.
There is no message queue:
Code: Select all
root@devnagios1:[/]: ipcs -q
------ Message Queues --------
key msqid owner perms used-bytes messages
root@devnagios1:[/]:
Also, the
Monitoring Engine Event Queue &
Monitoring Engine Check Statistics are not showing any data in "
Monitoring Engine Status" page.
When we create a new service, it shows in Nagios Core, but not in NagiosXI.
Please suggest.
Re: Message queue disappeared on NagiosXI post upgrade to 5.
Posted: Thu Jan 07, 2021 3:58 pm
by dchurch
This is to be expected because Nagios XI uses NDOUtils instead of the external listener process ndo2db. Let me explain why this is better.
ndo2db is our older technology that basically listens on a UNIX socket for database inserts, then handles the actual insertion into the database. It has limits, being that it runs into issues when it tries to insert more than the database can handle. In newer versions (
Nagios XI 5.7.0 and later), this was replaced by just writing directly to the database from the Nagios worker threads. In addition to being able to handle more database inserts, this resulted in an overall performance boost, too.
The reason the message queue is empty is because there aren't tons of pending database writes waiting in the socket IPC formerly used by ndo2db.
Re: Message queue disappeared on NagiosXI post upgrade to 5.
Posted: Fri Jan 08, 2021 10:51 am
by Sampath.Basireddy
Thanks
@dchurch for the response.
But why are we seeing this strange behavior after upgrade? New service created does not reflect on node in XI, but does in Nagios Core.
Similarly, Nagios Service keeps going down unexpectedly and after starting the service, it takes about an hour or so to come back to normal and show the actual node or service status. This wasn't the case before upgrade. Please suggests.
Re: Message queue disappeared on NagiosXI post upgrade to 5.
Posted: Fri Jan 08, 2021 5:45 pm
by dchurch
If you're saying that you modified a service in Core and it's not reflecting back up into Nagios XI, that's to be expected due to how Nagios Core and Nagios XI interact; Nagios XI pushes configuration to Core, not the other way around.
If that's not what you're saying, this sounds like a database issue on the face of it. Try running the database repair script, and let me know if that is successful. Here's how you do that:
Running the Database Repair Script
Run the following as root from the terminal.
Code: Select all
/usr/local/nagiosxi/scripts/repair_databases.sh
See here for complete instructions:
run the database repair
Try this if it doesn't work: Downgrading NDOUtils
If that doesn't fix your issue, what you can try doing is downgrading your version of NDOUtils.
Code: Select all
systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db
Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is uncommented:
Code: Select all
broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg
Make sure this line is commented:
Code: Select all
#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
Then start the nagios service:
The instructions are different if you have an offloaded database (i.e. a database that isn't on the same machine that XI is running on).
Re: Message queue disappeared on NagiosXI post upgrade to 5.
Posted: Mon Jan 11, 2021 2:59 pm
by Sampath.Basireddy
We are not creating/modifying the service in Core. It is in NagiosXI.
DB Repair did not help, so we did go ahead and downgraded NDOUtils which fixed the issue and everything came back to normal.
Is this a temp solution or is Nagios coming up with a new release to address this issue?
The reason I ask is, the issue we faced is on our Dev Instance of NagiosXI and we are yet to upgrade our Prod NagiosXI to version 5.7.x. Wanted to make sure before we go ahead with our Prod instance, so we don't break the production monitoring.
Please suggest.
Re: Message queue disappeared on NagiosXI post upgrade to 5.
Posted: Tue Jan 12, 2021 2:40 pm
by ssax
XI 5.8 is coming out in a few days, this would be a temporary fix but when you upgrade to XI 5.8 it will leave your NDO2DB downgraded, if you want to upgrade to the new updated version of NDO3 that comes with XI 5.8 you can do this after upgrading:
- NOTE: Since 5.8 isn't released, I'm just guessing at what the download URL is going to be
Code: Select all
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.8.0.tar.gz
tar zxf xi-5.8.0.tar.gz
cd /tmp/nagiosxi
./init.sh
cd /tmp/nagiosxi/subcomponents/ndo
./install
systemctl restart nagios
As an alternative (you can even do this on your existing system if you want to test the NDO3 version that will be coming in 5.8 on your system) you can take the attached zip file, transfer it to your XI server, and then run these commands as root:
Code: Select all
unzip ndo-master.zip
cd ndo-master
./configure
make all
make install
Comment out the old broker line in your
/usr/local/nagios/etc/nagios.cfg by changing it from:
Code: Select all
broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg
To:
Code: Select all
#broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg
Change the new broker line from:
Code: Select all
#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
To:
Code: Select all
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
Save the file and then restart nagios.
Then apply configuration and validate.