Page 1 of 2
pending host checks after apply config
Posted: Wed Aug 05, 2020 7:16 am
by noweda
Hi,
the problem described here came back - dispite having upgraded to 5.7.2:
https://support.nagios.com/forum/viewto ... 16&t=59202
Kind regards,
Tim
Re: pending host checks after apply config
Posted: Wed Aug 05, 2020 4:14 pm
by benjaminsmith
Hi Tim,
Sorry to hear about that, we have had better results with 5.7.2, but have found an issue with check results not updating on a few systems that is being patched right now.
What is your current status, did you revert to a previous version or downgrade ndo3 as mentioned before? Please let me know.
If you're able to send us a profile from this system, that would be helpful for us to analyze the issue further.
To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.
Re: pending host checks after apply config
Posted: Mon Aug 10, 2020 6:25 am
by noweda
Hi Benjamin,
no downgrade was performed, we are running 5.7.2.
Profile was sent by PM
kind regards
simon
Re: pending host checks after apply config
Posted: Tue Aug 11, 2020 8:47 am
by benjaminsmith
Hi Simon,
Thanks for sending over the system profile, that database logs look ok, so that's doesn't seem to be the issue here. However, I would like to check the status of the nagios user account, what is the output of the following command:
There are a number of checks using check_nrpe to, mostly likely, remote hosts running NSClient with errors.
Code: Select all
Aug 10 12:17:50 ux010162 check_nrpe: Remote x.x.x.140 does not support version 3/4 packets
Aug 10 12:17:50 ux010162 check_nrpe: Remote x.x.x.150 does not support version 3/4 packets
Please follow the steps in the KB article below to add a
-2 to the check command to use the version 2 packets in check_nrpe.
https://support.nagios.com/kb/article/n ... s-786.html
Having you noticed any improvement in check results updating? If not,please try to re-start the Nagios software stack and let me know the results. Thanks, Benjamin
Code: Select all
systemctl stop crond
systemctl stop npcd
systemctl stop nagios
pkill -9 -u nagios
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mysqld || systemctl restart mariadb
systemctl start nagios
systemctl start npcd
systemctl start crond
systemctl restart httpd
Re: pending host checks after apply config
Posted: Tue Aug 11, 2020 10:19 am
by noweda
Hi Benjamin,
here's the output from th nagios user account:
Code: Select all
root@ux010162:/# chage -l nagios
Last password change : Sep 25, 2013
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7
I verified check_nrpe at core config mananger under _Commands. -2 is already added to this command. Screenshot attached.
We had issues with the many checks after 14:59. All checks worked until 14:59. After the timestamp we received no new checks. I restarted the nagios stack with your command, but it doesn't worked. I used the command "service servicename restart".
But the issue still persists until 16:38. At this time another user applied the configuration with no changes. After that the checks worked again.
New Profile attached.
Kind regards,
Pasqual
Moderator's Note: The profile has been shared with the support team but has been removed from the public forum.
Re: pending host checks after apply config
Posted: Wed Aug 12, 2020 10:12 am
by benjaminsmith
Hi Pasqual,
After running the Apply Configuration, do you still continue to receive the results of the check, and for how long?
We're working on a fix to the intermittent check updates affecting some systems. As long as the monitoring engine is running, you will still continue to receive notifications. That said, there are a couple of options, if you're able to manage this for now we should have 5.7.3 out in a few weeks to correct the issue.
Otherwise, I would recommend downgrading the backend database ( ndo) to the previous version as described in your previous thread. Be sure to
make a backup before making any changes.
https://support.nagios.com/forum/viewto ... 16&t=59202
My apologies for the inconvenience.
Re: pending host checks after apply config
Posted: Tue Aug 25, 2020 2:25 am
by noweda
Hi Benjamin,
any news about the release date for 5.7.3? We are still having the problem, that the monitoring engine needs about one hour to normalize. Within that period only very few checks are done and the server seems to be totally busy with internal processes (mysqld mainly).
Are there any workarounds to speed this up?
Best regards
Christoph
Re: pending host checks after apply config
Posted: Tue Aug 25, 2020 3:14 pm
by benjaminsmith
Hi Christoph,
While I don't have a firm release date, we are making good progress in 5.7.3. If this is impacting your production instance, I would recommend downgrading the backend database to the previous version. Upgrading will simply re-install the new version again.
Downgrade Instructons (Local Database)
Code: Select all
systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db
Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is uncommented:
Code: Select all
broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg
Make sure this line is commented:
Code: Select all
#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
Then start the nagios service:
Re: pending host checks after apply config
Posted: Mon Sep 07, 2020 8:43 am
by noweda
Hi Benjamin,
today, I installed the maintenance release 5.7.3. Unfortunately, we still have the issue with apply configuration and the latest checks in the dashboard. After apply configuration it takes approximately 1 hour until the checks works again. With command top in the console I saw that the process mysqld consume high cpu 123%! Since the checks works again, mysqld consume approximately 60% cpu.
A new profile comes via PM.
Sincerely,
Pasqual
Re: pending host checks after apply config
Posted: Tue Sep 08, 2020 5:48 pm
by benjaminsmith
Hi Pasqual,
At this point, it would best to move this over to a support ticket, so we can get a remote session setup to take a look at your system to help us troubleshoot.
To open a support ticket go to:
https://support.nagios.com/tickets/
Let me know if you need any assistance.
Benjamin