Monitoring engine event queue shows zero.

[RebeccaIlene]

Monitoring engine event queue shows zero.

None of the checks are happening.

Please help.

[benjaminsmith]

Hi Rebecca,

It looks like the monitoring engine has stopped for some reason, can you run through the commands below to do full re-start ( or reboot the server).

Commands are for Cent7 and may need to be adjusted for other operating systems.

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mariadb
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

Let us know if it's working and if the issues persists please PM your system profile.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.

[RebeccaIlene]

Hi Benjamin,

This issue got auto fixed in five minutes.

What we have noticed is that after updating to 5.7.2, the monitoring engine event queue is going to zero whenever we apply configurations.

After 5- 10 minutes it goes back to 1k-2k checks as usual.

What can we do to fix this permanently?

[benjaminsmith]

Hi Rebecca,

We have had some systems being impacted by delayed check results in 5.7.2, once you apply the configuration, can you verify that checks on coming from the host or service details. Go to Home > Details > Service Status.

service-status.png

If they're not coming in, we may need to downgrade this system to the previous version of ndo2db, let me know.

Benjamin

[RebeccaIlene]

Hi Benjamin,

The checks are coming in but the monitoring engine queue goes to zero and then goes back to 1k checks in five minutes.

Is there a fix for this in the upcoming version update?

[ssax]

There are some fixes coming in XI 5.7.3 (should be released shortly after testing is finished) that should hopefully resolve this.

Please PM me a fresh copy of your profile.zip by clicking the Download Profile button in Admin > System Profile and I'll generate some instructions for downgrading your current NDO3 back to NDO2DB until 5.7.3 is released.

[RebeccaIlene]

We have updated the version to 5.7.3 but the issue still persists.

I was wrong about it taking 5- 10 minutes to come up.
The monitoring engine event queue takes 27 - 30 minutes to come up. We timed it now.
This is a serious issue as no checks/ notifications are sent for 30 minutes and this is not ideal.

What is the best way to fix this?
Is it better to downgrade the version of Nagios to 5.6?
All version updates for 5.7 so far have had so many bugs.

Will PM a copy of the profile to you.

[benjaminsmith]

Hi Rebecca,

Let's re-start the Nagios service on your server, then let it run for about 10-15 minutes and send us the nagios.log along with the system profile. This way we should be able to capture some the errors in the log to help troubleshoot this issue for you.

To restart nagios:

Code: Select all

systemctl restart nagios

Please attach the full nagios log to the thread. Thanks, Benjamin

Code: Select all

/usr/local/nagios/var/nagios.log

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.

[RebeccaIlene]

Hi Benjamin,

As a work around we didn't apply configurations today until the end of business.
After applying the configurations, I restarted the nagios service as requested and this time the monitoring engine event queue came back up in 10 minutes!

I have PM you the copy of the profile and the nagios.log file.

Thanks for your help.

[benjaminsmith]

Hi Rebecca,

The log entries db table is corrupted, so let's run the repair script.

Sep 10 12:29:39 komov430 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failed'

Log into the terminal as root and run the following:

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

After running the repair script, let me know if the server is running better.

Benjamin