Services not updating

[J.A.K]

Starting today any changes I make adding hosts or services don't seem to be taking affect. For example I've had one check stuck in unknown, but if I click it then it tells me the object doesn't exist.

both.PNG

I've tried deleting all conf files and rewriting, running the repair DB script, restoring from an old good snapshot.


EDIT: No errors that I can find pointing out what's wrong, but I found if I wait long enough it does eventually catch up. I know another member of my team was adding BPI groups and having to apply config + sync hostgroups in fairly quick succession. Is there some back end timer jobs that has to catch up here?

[benjaminsmith]

Hi J.A.K,

What version of Nagios XI you are currently using and did this issue correspond with any updates or other system changes?

I would re-boot the server or try restarting the Nagios process and let me know if that makes any difference.

Code: Select all

systemctl restart nagios

Can you PM system profile and we'll review the logs as well.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button

Thanks,
Benjmain

[J.A.K]

We're on the latest 5.8.1. Tried rebooting the XI server but only thing that helped was to let it sit for about an hour. Only thing that was happening at the time was my coworker making frequent updates requiring the config being reapplied as well as syncing BPI hostgroups. That's why I was wondering if maybe there was some subsystem command queue that got backed up. System profile is on its way this would have all happened yesterday at the time of this topic's creation.

[benjaminsmith]

Hi J.A.K,

Unfortunately, the database log and the nagios.log did not get zipped up into the profile. There may be some connection issues with the database, so it would great to get the tail output of that log.

See: Nagios XI Log Locations and Descriptions

We've had some issues with check results not updating correctly with the newer backend database application. It's a simple procedure to downgrade to the previous version, and I would recommend running through the steps below, and let me know the issue is resolved.

### STANDARD DOWNGRADE OF NDO3

Code: Select all

systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db

Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is uncommented:

Code: Select all

broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Make sure this line is commented:

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service:

Code: Select all

systemctl start nagios

[J.A.K]

Downgrading (which I'd consider a major change) would require me to plan ahead and get approvals for a change request which could take a few weeks sadly, so I'd need at least sure there's an issue with it. Things seem settled at the moment at least, but I did get those 2 files although my mysql log was at /var/log/mysql/mysqld.log instead of /var/log/mysqld.log like that doc has.

OneDrive_1_2-3-2021.zip

Grabbed Nagios from the day in question and the closest MySQL log which looks to be the next day.

[benjaminsmith]

Hi @J.A.K,

Glad to hear it's working better and thanks for fetching those logs. At the end the database log, there are a number for these errors.

2021-02-03T01:35:26.039685Z 0 [Warning] [MY-010909] [Server] /usr/libexec/mysqld: Forcing close of thread 79191 user: 'ndoutils'.
2021-02-03T01:35:26.039711Z 0 [Warning] [MY-010909] [Server] /usr/libexec/mysqld: Forcing close of thread 79743 user: 'nagiosql'.
2021-02-03T01:35:26.039781Z 0 [Warning] [MY-010909] [Server] /usr/libexec/mysqld: Forcing close of thread 78839 user: 'ndoutils'.

After researching this a little further, it looks like this can be corrected by increasing the max connections for the database. We have a knowledgebase article on the procedure. Let's go ahead and increase that setting.

Nagios XI - MySQL/MariaDB - Max Connections

Then let the server run for a while and run a tail command on the DB log and let me know if that's cleared out.

--Benjamin

[J.A.K]

Looks like even under normal load we were hitting close to max, so that might definitely be it.

[f10000003j@vml1038 ~]$ mysql -uroot -pnagiosxi -e "show variables like 'max_connections';"
mysql: [Warning] Using a password on the command line interface can be insecure.
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 |
+-----------------+-------+

[f10000003j@vml1038 ~]$ mysql -uroot -pnagiosxi -e "show global status like 'Max_used_connections';"
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| Max_used_connections | 107 |
+----------------------+-------+

I increased it using the instructions (although they may need updated since I don't think ndo2db.service is a thing anymore with ndo3db). Let it sit for awhile and don't see those errors. I guess service updates might have been delayed that one day waiting for connections to become available?

[benjaminsmith]

Hi,

That's certainly a possibility. Let us know if you notice an improvement.

Also, thanks for pointing that out with the documentation. We're updating that right now.

Best Regards,
Benjamin

[J.A.K]

So far everything seems solid. I'm a-ok with locking this thread as resolved. Thanks again!

[scottwilkerson]

So far everything seems solid. I'm a-ok with locking this thread as resolved. Thanks again!

Locking thread