NagiosXI very slow to apply changes

[Support_Talea]

Hi,

I have the Nagios XI in version 5.6.9.

It has been very slow to apply any changes for a long time, let's say it can take up to 3 minutes to complete the operation.

Before the database was offsite, now it is back all-in-one and the performances have improved, it is more streamlined and the checks are more efficient.

However, the problem of applying the changes remains.

From the logs I don't see anything strange, there are no errors or anything else, but it waits for 2-3 sometimes 4 minutes.

Server:
VM installed by me
Centos 7
24GB vram
12 vcpu
2hdd flash storage.

Do you have any advice?


Nagios XI - System Info
System
Nagios XI version: 5.6.9
XI installed from: source
XI UUID: 4894dd67-b8d9-4eec-b478-a0b1b6af6913
Release info: XXXXXXXX 3.10.0-957.27.2.el7.x86_64 x86_64
CentOS Linux release 7.6.1810 (Core)
Gnome Installed
Apache Information
PHP Version: 5.4.16
Agent: Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 87.0.4280.88 Safari / 537.36
Server Name: XXXXXX
Server Address: XXXXXXXX
Server Port: 443
Date / Time
PHP Timezone: Europe / Rome
PHP Time: Fri, 11 Dec 2020 17:46:52 +0100
System Time: Fri, 11 Dec 2020 17:46:52 +0100
Nagios XI Date
License ends in: OOPSRT
UUID: 4894dd67-b8d9-4eec-b478-a0b1b6af6913
Install Type: source

└─130681 / usr / local / nagios / libexec / check_icmp -H XXXXX -w 2000.0 50 -c 2500.0 80 -n 10
└─2286 / usr / local / nagios / bin / npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
└─104876 / usr / local / nagios / bin / ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
CPU Load 15: 1.41
Total Hosts: 1076
Total Services: 12646

Best regards,
-Fede

[benjaminsmith]

Hi,

The apply configuration can take some time as it needs to wait for the current checks to finish and stop the Nagios process. It also writes out the current configuration in the database to files, runs a verification on those files and then re-starts the nagios service. It's a pretty I/O intensive operation.

Please send me the system profile and I'll scan the logs for you as well. I would also run a top command during the apply configuration to see which processes are consuming the most resources.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" but

Not directly related, but It wouldn't hurt to increase the PHP settings (e.g. more memory) just to make sure nothing is slowing down into GUI. This will help with reports as well.

Nagios XI - Optimizing The PHP Settings File

Regards,
Benjamin

[Support_Talea]

Hi Benjamin,

thanks for your answer.
When I apply the server runs out of cpu / ram or I / o load

I don't understand why it waits for several minutes.

Best regards,
-f

[benjaminsmith]

Hi,

Thanks for the profile, please go ahead and run through the following commands to restart the services and clear the kernel message queues.

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mariadb
systemctl start ndo2db
systemctl start nagios 
systemctl start npcd
systemctl start crond

If you haven't done so already, follow the steps in the following guide to increase the KMQ settings on this server.

NDOUtils - Message Queue Exceeded

The apply configuration process can take some time, but I would recommend taking steps to free up resources on this server. I noticed that you have Gnome installed on the server, it's going to take more resources to run, so we usually do not recommend that.

Also, Installing a RAMdisk will help boost I/O performance as well.

Utilizing a RAM Disk in Nagios XI

Also, take a look at pages 4-7 to adjust the performance settings as well.

Maximizing Performance In Nagios XI

--Benjamin

[Support_Talea]

Hi Benjamin,

I'm trying to upgrade NagiosXI from 5.6.14 to 5.7.5. I have many problems for the Endo.
I am taking steps.
In the test environment almost everything went well, in the production environment the checks no longer update.
As soon as I can update I proceed with your steps.

Regards,
-f

[benjaminsmith]

Hi,

In the test environment almost everything went well, in the production environment the checks no longer update.

If you hitting the check results update issue, we can either downgrade NDO or I can supply a patched version that will ship out in 5.8 release to try out. Let me know.

To downgrade ndo for a standard install ( local database):

Code: Select all

systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db

Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is uncommented:

Code: Select all

broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Make sure this line is commented:

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service:

Code: Select all

systemctl start nagios

[Support_Talea]

Hi Benjamin,

thank you.

I prefer to go ahead with the NDO version, because disconnecting it from the system service seems like a good idea.
Is the package you say is in this section? https://support.nagios.com/forum/viewto ... 0&start=10
Because I answered in a post where I found a user who - I believe - has the same problem as me.

If you confirm this evening I try to go from 5.6.14 to 5.7.2 following these steps:

cd / tmp
rm -rf / tmp / nagiosxi
wget https://assets.nagios.com/downloads/nag ... 7.2.tar.gz
tar zxf xi-5.7.2.tar.gz
cd / tmp / nagiosxi / subcomponents / ndo
tar zxf ndo-3.0.2.tar.gz
sed -i 's / \ $ versions \ [0 \] / \ $ \ {versions \ [0 \] \} / g' /tmp/nagiosxi/subcomponents/ndo/ndo-3.0.2/db/db-mgmt .sh
rm -f ndo-3.0.2.tar.gz
tar zcvf ndo-3.0.2.tar.gz ndo-3.0.2
rm -rf ndo-3.0.2
cd / tmp / nagiosxi
./upgrade


Then at the end of the upgrade I install the patch "ndo-master.zip" and hopefully, I bring the NagiosXI to 5.7.5. I can't take long with the operations in Production because otherwise I have holes in the reports for our divisions.

Best regards,
-fEde

[benjaminsmith]

HI,

Here are the steps to upgrade to the patched version of NDO 3. You'll copy the ndo-master.zip file to the XI server and run the following to compile, install it and to restart nagios.

Once you have copied it over to your XI server, run the following commands

Code: Select all

unzip ndo-master.zip
cd ndo-master
./configure
make all

Then make sure your loading the new broker module line in /usr/local/nagios/etc/nagios.cfg, as follows.

Code: Select all

broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

And the old broker module line is commented out.

Code: Select all

#broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Then restar the nagios service to load the new version.

Code: Select all

systemctl restart nagios

I've attached the zip file to the thread. Let me know if you need any assistance.

[Support_Talea]

Hi Benjamin,

nothing.
Made the procedure to update from 5.6.14 to 5.7.3, then applied the ndo-master.zip patch unfortunately the problem is not solved.
Checks lag behind the time and do not update the next check time.
Plus I lose all scheduled downtime.
The logs say nothing.
I find no errors but I find the Nagios not working.
Unable to upgrade to 5.7.

Do you have any ideas?

Best regards,
-Fede

[benjaminsmith]

Hi Fede,

Are you able to revert back to 5.6.14 or is the system down right now? Did the new version of ndo3 successfully compile?

It sounds like the monitoring engine is not running, please check the status and restart if needed.

Code: Select all

systemctl status nagios

Pleases download a system profile from the updated server.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button

This can also be done from the command line if needed. Thanks, Benjamin

Code: Select all

rm -rf /usr/local/nagiosxi/var/components/profile*​​
/usr/local/nagiosxi/scripts/components/getprofile.sh SUPPORT

Profile is located at: /usr/local/nagiosxi/var/components/profile.zip​.