Page 1 of 2
Monitoring engine event queue shows zero.
Posted: Fri Aug 28, 2020 6:48 am
by RebeccaIlene
Monitoring engine event queue shows zero.
None of the checks are happening.
Please help.
Re: Monitoring engine event queue shows zero.
Posted: Fri Aug 28, 2020 5:08 pm
by benjaminsmith
Hi Rebecca,
It looks like the monitoring engine has stopped for some reason, can you run through the commands below to do full re-start ( or reboot the server).
Commands are for Cent7 and may need to be adjusted for other operating systems.
Code: Select all
systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mariadb
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond
Let us know if it's working and if the issues persists please PM your system profile.
To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.
Re: Monitoring engine event queue shows zero.
Posted: Mon Aug 31, 2020 11:47 am
by RebeccaIlene
Hi Benjamin,
This issue got auto fixed in five minutes.
What we have noticed is that after updating to 5.7.2, the monitoring engine event queue is going to zero whenever we apply configurations.
After 5- 10 minutes it goes back to 1k-2k checks as usual.
What can we do to fix this permanently?
Re: Monitoring engine event queue shows zero.
Posted: Tue Sep 01, 2020 12:59 pm
by benjaminsmith
Hi Rebecca,
We have had some systems being impacted by delayed check results in 5.7.2, once you apply the configuration, can you verify that checks on coming from the host or service details. Go to Home > Details > Service Status.
service-status.png
If they're not coming in, we may need to downgrade this system to the previous version of ndo2db, let me know.
Benjamin
Re: Monitoring engine event queue shows zero.
Posted: Wed Sep 02, 2020 10:34 pm
by RebeccaIlene
Hi Benjamin,
The checks are coming in but the monitoring engine queue goes to zero and then goes back to 1k checks in five minutes.
Is there a fix for this in the upcoming version update?
Re: Monitoring engine event queue shows zero.
Posted: Thu Sep 03, 2020 1:49 pm
by ssax
There are some fixes coming in XI 5.7.3 (should be released shortly after testing is finished) that should hopefully resolve this.
Please PM me a fresh copy of your profile.zip by clicking the Download Profile button in Admin > System Profile and I'll generate some instructions for downgrading your current NDO3 back to NDO2DB until 5.7.3 is released.
Re: Monitoring engine event queue shows zero.
Posted: Wed Sep 09, 2020 2:31 am
by RebeccaIlene
We have updated the version to 5.7.3 but the issue still persists.
I was wrong about it taking 5- 10 minutes to come up.
The monitoring engine event queue takes 27 - 30 minutes to come up. We timed it now.
This is a serious issue as no checks/ notifications are sent for 30 minutes and this is not ideal.
What is the best way to fix this?
Is it better to downgrade the version of Nagios to 5.6?
All version updates for 5.7 so far have had so many bugs.
Will PM a copy of the profile to you.
Re: Monitoring engine event queue shows zero.
Posted: Wed Sep 09, 2020 4:54 pm
by benjaminsmith
Hi Rebecca,
Let's re-start the Nagios service on your server, then let it run for about 10-15 minutes and send us the nagios.log along with the system profile. This way we should be able to capture some the errors in the log to help troubleshoot this issue for you.
To restart nagios:
Please attach the full nagios log to the thread. Thanks, Benjamin
To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.
Re: Monitoring engine event queue shows zero.
Posted: Thu Sep 10, 2020 5:34 am
by RebeccaIlene
Hi Benjamin,
As a work around we didn't apply configurations today until the end of business.
After applying the configurations, I restarted the nagios service as requested and this time the monitoring engine event queue came back up in 10 minutes!
I have PM you the copy of the profile and the nagios.log file.
Thanks for your help.
Re: Monitoring engine event queue shows zero.
Posted: Fri Sep 11, 2020 10:58 am
by benjaminsmith
Hi Rebecca,
The log entries db table is corrupted, so let's run the repair script.
Sep 10 12:29:39 komov430 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failed'
Log into the terminal as root and run the following:
Code: Select all
/usr/local/nagiosxi/scripts/repair_databases.sh
After running the repair script, let me know if the server is running better.
Benjamin