pending host checks after apply config

[noweda]

Hi,

the problem described here came back - dispite having upgraded to 5.7.2:
https://support.nagios.com/forum/viewto ... 16&t=59202

Kind regards,
Tim

[benjaminsmith]

Hi Tim,

Sorry to hear about that, we have had better results with 5.7.2, but have found an issue with check results not updating on a few systems that is being patched right now.

What is your current status, did you revert to a previous version or downgrade ndo3 as mentioned before? Please let me know.

If you're able to send us a profile from this system, that would be helpful for us to analyze the issue further.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.

[noweda]

Hi Benjamin,

no downgrade was performed, we are running 5.7.2.
Profile was sent by PM

kind regards
simon

[benjaminsmith]

Hi Simon,

Thanks for sending over the system profile, that database logs look ok, so that's doesn't seem to be the issue here. However, I would like to check the status of the nagios user account, what is the output of the following command:

Code: Select all

chage -l nagios

There are a number of checks using check_nrpe to, mostly likely, remote hosts running NSClient with errors.

Code: Select all

Aug 10 12:17:50 ux010162 check_nrpe: Remote x.x.x.140 does not support version 3/4 packets
Aug 10 12:17:50 ux010162 check_nrpe: Remote x.x.x.150 does not support version 3/4 packets

Please follow the steps in the KB article below to add a -2 to the check command to use the version 2 packets in check_nrpe.
https://support.nagios.com/kb/article/n ... s-786.html

Having you noticed any improvement in check results updating? If not,please try to re-start the Nagios software stack and let me know the results. Thanks, Benjamin

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
pkill -9 -u nagios
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mysqld || systemctl restart mariadb
systemctl start nagios
systemctl start npcd
systemctl start crond
systemctl restart httpd

[noweda]

Hi Benjamin,
here's the output from th nagios user account:

Code: Select all

root@ux010162:/# chage -l nagios
Last password change                                    : Sep 25, 2013
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7

I verified check_nrpe at core config mananger under _Commands. -2 is already added to this command. Screenshot attached.

We had issues with the many checks after 14:59. All checks worked until 14:59. After the timestamp we received no new checks. I restarted the nagios stack with your command, but it doesn't worked. I used the command "service servicename restart".

But the issue still persists until 16:38. At this time another user applied the configuration with no changes. After that the checks worked again.

New Profile attached.

Kind regards,
Pasqual

Moderator's Note: The profile has been shared with the support team but has been removed from the public forum.

[benjaminsmith]

Hi Pasqual,

After running the Apply Configuration, do you still continue to receive the results of the check, and for how long?

We're working on a fix to the intermittent check updates affecting some systems. As long as the monitoring engine is running, you will still continue to receive notifications. That said, there are a couple of options, if you're able to manage this for now we should have 5.7.3 out in a few weeks to correct the issue.

Otherwise, I would recommend downgrading the backend database ( ndo) to the previous version as described in your previous thread. Be sure to make a backup before making any changes.

https://support.nagios.com/forum/viewto ... 16&t=59202

My apologies for the inconvenience.

[noweda]

Hi Benjamin,

any news about the release date for 5.7.3? We are still having the problem, that the monitoring engine needs about one hour to normalize. Within that period only very few checks are done and the server seems to be totally busy with internal processes (mysqld mainly).

Are there any workarounds to speed this up?

Best regards
Christoph

[benjaminsmith]

Hi Christoph,

While I don't have a firm release date, we are making good progress in 5.7.3. If this is impacting your production instance, I would recommend downgrading the backend database to the previous version. Upgrading will simply re-install the new version again.

Downgrade Instructons (Local Database)

Code: Select all

systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db

Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is uncommented:

Code: Select all

broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Make sure this line is commented:

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service:

Code: Select all

systemctl start nagios

[noweda]

Hi Benjamin,

today, I installed the maintenance release 5.7.3. Unfortunately, we still have the issue with apply configuration and the latest checks in the dashboard. After apply configuration it takes approximately 1 hour until the checks works again. With command top in the console I saw that the process mysqld consume high cpu 123%! Since the checks works again, mysqld consume approximately 60% cpu.

A new profile comes via PM.

Sincerely,
Pasqual

[benjaminsmith]

Hi Pasqual,

At this point, it would best to move this over to a support ticket, so we can get a remote session setup to take a look at your system to help us troubleshoot.

To open a support ticket go to:

https://support.nagios.com/tickets/

Let me know if you need any assistance.

Benjamin