Monitoring Engine Event Queue is empty

[lvaillant]

Hello,

I just upgraded my Nagios XI from 5.4.13 to 5.8.6, without any major issues during the steps : 5.4.13 >- 5.5.0 -> 5.6.0 -> 5.7.0 -> 5.8.0 -> 5.8.6.

Nagios Core is polling data, using mod_gearmand2 with other cores...
I see the process and the Nagios logs are running.

But the Nagios XI's Monitoring Engine Event Queue stays empty.
Alerts do not update in Nagios XI and I do not identify specific error in Nagios XI logs.

DB seems to be ok as I used repair_databases script.
process statuses are ok (nagios, nagiosxi, crond, httpd...)

What can be the root cause of this issue ?
Did I miss something ?

Thank you in advance.

[lvaillant]

I restarted several nagios sub parts, and data now feed the RRD files.
I don't know which one helped...

But still no Event Queue, and no log entries.

[lvaillant]

Looking for leads and having a look at Nagios XI - upg v5.6.14 to 5.8.6 issues, my nagios.cfg contains the following lines:

Code: Select all

# grep broker ../etc/nagios.cfg
# Commented out by NDO 'make install-broker-line' on Mon Oct 25 17:14:57 CEST 2021
# Commented out by NDO 'make install-broker-line' on Mon Oct 25 17:37:01 CEST 2021
# Commented out by NDO 'make install-broker-line' on Mon Oct 25 17:50:04 CEST 2021
###broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg
event_broker_options=-1
broker_module=/usr/lib64/mod_gearman2/mod_gearman2.o config=/etc/mod_gearman2/module.conf eventhandler=no
# Added by NDO 'make install-broker-line' on Mon Oct 25 17:14:57 CEST 2021
# Commented out by NDO 'make install-broker-line' on Mon Oct 25 17:37:01 CEST 2021
# Commented out by NDO 'make install-broker-line' on Mon Oct 25 17:50:04 CEST 2021
##broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
# Added by NDO 'make install-broker-line' on Mon Oct 25 17:37:01 CEST 2021
# Commented out by NDO 'make install-broker-line' on Mon Oct 25 17:50:04 CEST 2021
#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
# Added by NDO 'make install-broker-line' on Mon Oct 25 17:50:04 CEST 2021
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

I'm using Mod_Gearman2, as I do not yet upgraded it on master and poller servers.
It works as I see jobs runnings with gearman_top2 and ps commands, and Nagios Core alerts are up2date.

Should I downgrade NDO3 to NDO2db ?

[lvaillant]

After a click to display Scheduling queue, I see recent entries: "next check" date is in the near future of the current time.
(but displaying the table is very slow)

So jobs are really scheduled and running, data are registered by Nagios Core, and graphs updates.
But alerts/notifications do not update, and event queue graph is still empty...

[tgriep]

One question I have is what version of Nagios Core is the server running now?

Run this to get the version.

Code: Select all

/usr/local/nagios/bin/nagios

I am asking for this as there is a compatibility issue with the newer versions of Core and the older Mod Gearman 2.

If the XI upgrade did upgrade Core to 4.4.6, them Mod Gearman will have to upgraded to version 3.
The following link has documentation for upgrading Gearman.
https://assets.nagios.com/downloads/nag ... ios_XI.pdf

If core is still at the older version, 4.2.4, then the new NDO-3 broker will need to be replaced by NDOUTILS which is compatible with the older version of Core.
If you want to go back to ndoutils, do the following.

Follow these instructions to install the ndoutils broker on the server to replace the NDO-3 broker.

Code: Select all

systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi
./init.sh
cd /tmp/nagiosxi/subcomponents/ndoutils
./install

Code: Select all

systemctl enable ndo2db
systemctl start ndo2db

Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is not commented out or if it is not there add it. It should look like the following.

Code: Select all

broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Find the next line and comment it out by adding a # at the beginning of the line.

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service:

Code: Select all

systemctl start nagios

If you have an offloaded MYSQL database you will need to edit your /usr/local/nagios/etc/ndo2db.cfg file and update these settings so the ndo2db daemon will connect to the offloaded MYSQL database.
- You can get the info from your /usr/local/nagios/etc/ndo.cfg or from /usr/local/nagiosxi/html/config.inc.php
db_host
db_port
db_user
db_pass

Thank you.

[lvaillant]

Hello

Code: Select all

# /usr/local/nagios/bin/nagios

Nagios Core 4.2.4
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 12-07-2016
License: GPL
...

So I applied the downgrade for NDO

Code: Select all

# cd /tmp
# cp ~/upgrade/xi-5.6.14.tar.gz .
# tar xzvf xi-5.6.14.tar.gz
# cd nagiosxi/
# ./init.sh
# cd subcomponents/ndoutils/
# ./install
...
INSTALL: POST-INSTALL: Starting NDOUtils post install...
kernel.msgmnb = 786432000
kernel.msgmax = 262144000
kernel.shmmax = 4294967295
kernel.shmall = 268435456
kernel.msgmni = 512000
vm.swappiness = 1
NOTE: If prompted, enter the MySQL root password
NDOUtils database already exists - skipping creation...
INSTALL: POST-INSTALL: NDOUtils post installed OK.
NOTE: If prompted, enter the MySQL root password
NDOUtils database appears OK
NDOUtils installed OK

# systemctl enable ndo2db
Created symlink from /etc/systemd/system/multi-user.target.wants/ndo2db.service to /usr/lib/systemd/system/ndo2db.service.
# vi /usr/local/nagios/etc/ndo2db.cfg
db_servertype=mysql
db_host=localhost
db_port=3306

db_name=nagios
db_prefix=nagios_

#db_user=ndoutils
#db_pass=n@gweb
db_user=XXXXXXXXX
db_pass=YYYYYYYYY

# vi /usr/local/nagios/etc/nagios.cfg
#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

# systemctl start ndo2db
# systemctl status ndo2db
● ndo2db.service - Nagios Data Out Daemon
   Loaded: loaded (/usr/lib/systemd/system/ndo2db.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-10-26 08:02:41 CEST; 9s ago
     Docs: http://www.nagios.org/documentation
  Process: 327416 ExecStartPre=/bin/rm -f /usr/local/nagios/var/ndo.sock (code=exited, status=0/SUCCESS)
  Process: 327412 ExecStartPre=/bin/rm -f /usr/local/nagios/var/ndo2db.lock (code=exited, status=0/SUCCESS)
 Main PID: 327420 (ndo2db)
   CGroup: /system.slice/ndo2db.service
           └─327420 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f

Oct 26 08:02:41 xxx systemd[1]: Starting Nagios Data Out Daemon...
Oct 26 08:02:41 xxx systemd[1]: Started Nagios Data Out Daemon.

# systemctl start nagios

And...
Nothing new.

Nagios Core is running efficiently, checks are ok, alerts too.
But in Nagios XI, graphs update but the alerts do not...

My monitoring team is now blind for hours

[lvaillant]

Having a doubt about initscripts (& systemctl related scripts), I tried to stop nagios via systemctl command.
It did not work (nagios still running) and I had to pkill nagios processus.

Restarting Nagios via cmdline did not completly work, as Monitoring Engine, Performance Grapher & Process state were red in the admin status pages.
I restarted them with the web interface, and now "Event queue" shows activity.
Hosts and Services status updates.

I'll wait some hours to ensure that my plateform now runs as expected.

If I want to upgrade Nagios Core to latest version and NDO2db to NDO again, what is the process please?
Of course, I'll also upgrade mod_gearman from v2 to v3 using https://assets.nagios.com/downloads/nag ... ios_XI.pdf

[tgriep]

Here are the instructions for upgrading Core on your XI server to the latest core version that comes with XI.
https://support.nagios.com/kb/article/n ... e-839.html

Thank you.

[lvaillant]

Thank You.
This thread can be closed.

Regards

[tgriep]

Your Welcome. Locking up the post for you.