Hanging when applying configuration

[paen08]

Hello!

We have an issue when we try to apply configuration in Core Config Manager.

Waiting for configuration verification....................................................................................................................................................................................................................................................................................................................

We have tried steps in https://support.nagios.com/kb/article/n ... es-34.html, but it didn't help.

Best regards
/Patrik

[dchurch]

Question: Does it happen every time, or only sometimes?

If you PM me a system profile I can diagnose further. Get one by going to Admin (top menu) => System Profile (in the left menu), then clicking the blue button.

If you're unable to generate the the profile through the web interface, please try generating it from the command line by running these commands as root:

Code: Select all

rm -rf /usr/local/nagiosxi/var/components/profile*
/usr/local/nagiosxi/scripts/components/getprofile.sh SUPPORT

Then send me the resulting /usr/local/nagiosxi/var/components/profile.zip file.
If the profile script fails, please include the ENTIRE output.

[paen08]

This happens every time. I have sent profile in direct message.

Best regards
/Patrik

[dchurch]

Try following this doc for things to try to help: https://support.nagios.com/kb/article.php?id=34

If that doesn't help...

Instead of using Apply Config, if you go to to Config (top menu) -> Core Config Manager => Config File Management (left menu), then running Delete, Write, then Verify often will reveal more problems with better error messages. This is tremendously helpful when diagnosing config problems. (This is actually the process that the "Apply Config" button does under the hood.)

If it hangs on one of those steps, tell me so I can diagnose further.

Other stuff I'm seeing

This is happening when a certain command is executed. Not sure why it's reading those files (this server is running only Nagios XI, right?), but maybe you want to change the permissions to make sure those files can be read.

Code: Select all

/usr/local/nagioslogserver/mibs/NAGIOS-NOTIFY-MIB.txt: Permission denied
/usr/local/nagioslogserver/mibs/NAGIOS-ROOT-MIB.txt: Permission denied
/usr/local/nagioslogserver/mibs/NAGIOS-NOTIFY-MIB.txt: Permission denied
/usr/local/nagioslogserver/mibs/NAGIOS-ROOT-MIB.txt: Permission denied
/usr/local/nagioslogserver/mibs/NAGIOS-NOTIFY-MIB.txt: Permission denied

[paen08]

We have already done all in the document you stated from Missing/incorrect sudoers Entries and below!

I performed your steps;

Nagios Core 4.4.6
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2020-04-28
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Warning: Duplicate definition found for service 'NTR tablespace' on host 'rts-app1' (config file '/usr/local/nagios/etc/services/NTR_Tablespace_RWTS.cfg', starting on line 16)
Warning: Duplicate definition found for service 'NTR tablespace' on host 'rts-app1' (config file '/usr/local/nagios/etc/services/rts-app1.cfg', starting on line 16)
Error: Invalid max_attempts, check_interval, retry_interval, or notification_interval value for service 'NTR tablespace' on host 'rts-app1'
Error: Could not register service (config file '/usr/local/nagios/etc/services/check_tablespace.cfg', starting on line 16)
Error processing object config files!

I removed all occurances I could find of this service NTR_Tablespace on rts-app1 and deleted, wrote config, verified with no errors and restarted CCM from same page. Then it was no outstanding issue...

HOWEVER when I tried once more to apply configuration without changes, it hangs..

I then make once more delete, write file and verify with the following output:

Nagios Core 4.4.6
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2020-04-28
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 915 services.
Checked 57 hosts.
Checked 7 host groups.
Checked 0 service groups.
Checked 4 contacts.
Checked 7 contact groups.
Checked 136 commands.
Checked 10 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 57 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 10 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check

Apply configuration:

Waiting for configuration verification......................................................................................................................................................................................................................................................................................................................

[paen08]

I don't know if this is related, but every command I am giving currently like force an immediate check seems to end in following message:
"Your request was not processed in a timely manner. It may still execute, as the server may be temporarily busy."

[dchurch]

MySQL on your system is taking up a lot of CPU resources.

I'm noticing now some database errors:

Code: Select all

210301 16:38:52  InnoDB: Error: trying to open a table, but could not
InnoDB: open the tablespace file './nagiosxi/#sql-3d6e_d6.ibd'!
InnoDB: Have you moved InnoDB .ibd files around without using the
InnoDB: commands DISCARD TABLESPACE and IMPORT TABLESPACE?
InnoDB: It is also possible that this is a temporary table #sql...,
InnoDB: and MySQL removed the .ibd file for this.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
210301 16:38:52  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.

Was there an unexpected shutdown?

Not sure why I didn't tell you to do this before, but... Try running the database repair script, and let me know if that is successful. Run the following as root from the terminal.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

[paen08]

This was executed.

===============
REPAIR COMPLETE
===============
DATABASE: nagiosql
TABLE:
/var/lib/mysql/nagiosql ~
DATABASE: nagiosxi
TABLE:
/var/lib/mysql/nagiosxi ~

=======================
nagios database repair succeeded

HOWEVER the problems persist

Force an immediate check gives "Your request was not processed in a timely manner...."

Apply configuration in CCM still hangs: "Waiting for configuration verification........................................................................................................................."

Verify in config file management says:

Nagios Core 4.4.6
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2020-04-28
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 915 services.
Checked 57 hosts.
Checked 7 host groups.
Checked 0 service groups.
Checked 4 contacts.
Checked 7 contact groups.
Checked 136 commands.
Checked 10 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 57 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 10 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check

[benjaminsmith]

Hi paen08,

Helping Dan out with this support case. When did you start to notice this behavior and did it coincide with any system changes?

Please run the following command to repair the database and test once more.

Code: Select all

mysqlcheck -r -f -uroot -pnagiosxi --all-databases	

Then post the output of the following commands if the issue persists. Thanks, Benjamin

Code: Select all

/usr/local/nagiosxi/scripts/reset_config_perms.sh
chage -l nagios
tail -n 200 /var/log/cron
grep nag /etc/group