Stuck on - "Waiting for configuration verification"

[solarisfire]

So we have a brand new airgapped install and imported all of our config successfully.

We're now hitting a roadblock applying the configuration.

The GUI just shows "Waiting for configuration verification" with a line of dots growing forever.

I can't see any obvious issues in the logs.

I've followed instructions in threads like https://support.nagios.com/forum/viewto ... n&start=20

And the DB seems fine.
I've run DB maintenance and it finishes fine.
The script: "/usr/local/nagiosxi/scripts/reconfigure_nagios.sh" throws an error that I can't find in any forum searches:

--- reset_config_perms.sh ------------
> Setting script permissions
> Setting CCM script permissions
> Setting special script permissions
> Setting special component script permissions
> Setting migrate permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting libexec directory permissions
> Setting Nagios XI config permissions
> Setting NOM checkpoint user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------
Could not open input file: /usr/local/nagiosxi/scripts/ccm_import.php

Importing into the CCM failed!

Deleting/Writing/Verifying/Applying manually works fine...

I've updated the port in /usr/local/nagiosxi/html/config.inc.php as I'm using a non-standard port.

[ssax]

What is the output of this command?

Code: Select all

ls -la /usr/local/nagiosxi/scripts/

Can you apply configuration from the command line as the nagios user?

Code: Select all

su - nagios
/usr/local/nagiosxi/scripts/reconfigure_nagios.sh

[solarisfire]

total 532
drwxr-xr-x. 5 root nagios 4096 Jan 18 14:09 .
drwxr-xr-x. 10 root nagios 4096 Nov 2 16:37 ..
drwxr-xr-x. 3 nagios nagios 4096 Nov 2 16:33 automation
-r-xr-x---. 1 root nagios 7861 Nov 2 16:33 backup_xi.sh
-r-xr-x---. 1 nagios nagios 8195 Nov 2 16:33 ccm_delete_object.php
-r-xr-x---. 1 nagios nagios 1041 Nov 2 16:33 ccm_export.php
-r-xr-x---. 1 nagios nagios 1630 Nov 2 16:33 ccm_import.php
-r-xr-x---. 1 nagios nagios 3180 Nov 2 16:33 ccm_snapshot.sh
-r-xr-x---. 1 root nagios 1914 Nov 2 16:33 change_timezone.sh
drwxr-xr-x. 2 root nagios 4096 Nov 2 16:33 components
-rwxr-xr-x. 1 nagios nagios 2839 Nov 2 16:33 contact_notification_handler.php
-rwxr-xr-x. 1 nagios nagios 14051 Nov 2 16:33 deploy_run_job.php
-rw-r--r--. 1 nagios nagios 73 Nov 2 16:33 .gitignore
-rwxr-xr-x. 1 nagios nagios 266 Nov 2 16:33 handle_nagioscore_event.php
-rwxr-xr-x. 1 nagios nagios 3903 Nov 2 16:33 handle_nagioscore.inc.php
-rwxr-xr-x. 1 nagios nagios 267 Nov 2 16:33 handle_nagioscore_notification.php
-r-xr-x---. 1 root nagios 1270 Nov 2 16:33 import_xiconfig.php
-rwxr-xr-x. 1 nagios nagios 705 Nov 2 16:33 initialize_mibs.php
-r-xr-x---. 1 root nagios 3917 Nov 2 16:33 manage_services.sh
-r-xr-x---. 1 root nagios 3820 Nov 2 16:33 manage_ssl_config.sh
drwxr-xr-x. 5 root nagios 4096 Nov 2 16:37 migrate
-rwxr-xr-x. 1 nagios nagios 283461 Nov 2 16:33 nagiosql_defaults.sql
-rwxr-xr-x. 1 nagios nagios 506 Nov 2 16:33 nom_create_nagioscore_checkpoint_cond.sh
-rwxr-xr-x. 1 nagios nagios 1530 Nov 2 16:33 nom_create_nagioscore_checkpoint.sh
-rwxr-xr-x. 1 nagios nagios 859 Nov 2 16:33 nom_create_nagioscore_errorpoint.sh
-rwxr-xr-x. 1 nagios nagios 1474 Nov 2 16:33 nom_restore_nagioscore_checkpoint.sh
-rwxr-xr-x. 1 nagios nagios 1454 Nov 2 16:33 nom_restore_nagioscore_checkpoint_specific.sh
-rwxr-xr-x. 1 nagios nagios 5399 Nov 2 16:33 nom_trim_checkpoints.sh
-rwxr-xr-x. 1 nagios nagios 7779 Nov 2 16:33 nxti_import.php
-rwxr-xr-x. 1 nagios nagios 1351 Nov 2 16:33 nxti.php
-rwxr-xr-x. 1 nagios nagios 13437 Nov 2 16:33 parse_core_eventlog.php
-rwxr-xr-x. 1 nagios nagios 6716 Nov 2 16:33 patch_ndoutils.php
-rwxr-xr-x. 1 nagios nagios 963 Nov 2 16:33 reconfigure_nagios.sh
-rwxr-xr-x. 1 nagios nagios 4162 Nov 2 16:33 remove_historical_data.sh
-r-xr-x---. 1 root nagios 1654 Nov 2 16:33 repair_databases.sh
-r-xr-x---. 1 root nagios 4153 Nov 2 16:33 repairmysql.sh
-r-xr-x---. 1 root nagios 5987 Nov 2 16:33 reset_config_perms.sh
-rwxr-xr-x. 1 nagios nagios 1859 Nov 2 16:33 reset_nagiosadmin_password.php
-rwxr-xr-x. 1 nagios nagios 1881 Nov 2 16:33 restart_nagios_with_export.sh
-rwxr-xr-x. 1 nagios nagios 2841 Nov 2 16:33 restore_defaults.sh
-rwxr-xr-x. 1 nagios nagios 15294 Nov 2 16:33 restore_xi.sh
-rwxr-xr-x. 1 nagios nagios 1908 Nov 2 16:33 send_to_auditlog.php
-r-xr-x---. 1 root nagios 1534 Nov 2 16:33 send_to_nls.php
-rwxr-xr-x. 1 nagios nagios 1345 Nov 2 16:33 unlock_user_account.php
-rwxr-xr-x. 1 nagios nagios 722 Nov 2 16:33 update_check.php
-r-xr-x---. 1 root nagios 2914 Nov 2 16:33 upgrade_to_latest.sh

Works fine as the Nagios user:


--- reset_config_perms.sh ------------
> Setting script permissions
> Setting CCM script permissions
> Setting special script permissions
> Setting special component script permissions
> Setting migrate permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting libexec directory permissions
> Setting Nagios XI config permissions
> Setting NOM checkpoint user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------

--- ccm_import.php -------------------
> Setting import directory: /usr/local/nagios/etc/import/
> Importing config files into the CCM
- Importing: /usr/local/nagios/etc/import/PmBJ9tjS.tmp.cfg .. SUCCESS
--------------------------------------

--- ccm_export.php -------------------
> Writing CCM configuration to Nagios files
Finished writing out configuraton
--------------------------------------

--------------------------------------
> Verifying configuration with Nagios Core
> Output:
Nagios Core 4.4.6
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2020-04-28
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 539 services.
Checked 35 hosts.
Checked 16 host groups.
Checked 0 service groups.
Checked 5 contacts.
Checked 3 contact groups.
Checked 166 commands.
Checked 9 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 35 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 9 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
> Return Code: 0
--------------------------------------

However the WebGUI still won't apply config changes...

Also when trying to submit a command, like forcing an immediate check of a host, it will return:

"Your request was not processed in a timely manner. It may still execute, as the server may be temporarily busy".

[ssax]

What is the output of these commands?

Code: Select all

sestatus
cat /etc/sudoers
cat /etc/sudoers.d/*

[solarisfire]

selinux is disabled.

SELinux status: disabled

Our sudoers file is confidential, it's mounted on NFS and shared between every server and contains config for every application, so I can't just share it on a public forum.

It does contain a section that says:

#----------
#CONFIG FOR NAGIOSXI
#----------

User_Alias NAGIOSXI=nagios
User_Alias NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/send_to_nls.php *
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/migrate/migrate.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/components/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_ssl_config.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/components/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh

And I've tested and both the apache and nagios users can run everything they need to with no password.

[solarisfire]

As a test I configured:

nagios ALL = NOPASSWD: ALL
apache ALL = NOPASSWD: ALL

and "Waiting for configuration varification" still gets stuck. So I don't think it's the sudoers config.

[pbroste]

Hello @solarisfire

When you run through the " applying the configuration" on the web console do you see any errors or warnings while running this:

Start this command first then 'Apply Configuration":

• Code: Select all

  tail -F /usr/local/nagiosxi/var/cmdsubsys.log /var/log/httpd/error_log /var/log/httpd/ssl_error_log

Also observe the Network Tab in Development tools on your web browser to find out what is spinning:

• network_dev_tools.png

Let us know how things look,
Perry

[solarisfire]

The network tab just seems to be calling:

ajaxhelper.php?cmd=getcommandstatus&opts=7&nsp=9547d2775f72c8932fdb3227cc15c3030314501c671c6feb4709ed656d8a2a60

about once a second... which returns a 200...

Weirdly that returns a json object.

{"command_id":"7","status_code":"0","result_code":"0","submission_time":"2022-01-17 07:22:43","event_time":"2022-01-17 07:22:43","processing_time":null,"result":null}

Is something stuck in the system submitted 2 days ago?

[solarisfire]

/usr/local/nagiosxi/var/cmdsubsys.log only contains:
.............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
..........................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
.............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
..........................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
...........................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
...........................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known

[solarisfire]

The SSL error log contain a lot of errors... But not sure how many pertain to this issue...

cat /var/log/httpd/ssl_error_log | awk -F] '{print $5}' | awk -F, '{print $1}' | sort | uniq -c
1 AH01617: user nagiosadmin: authentication failure for "/nagios/": Password Mismatch
3 Could not resolve host: api.nagios.com; Name or service not known
4 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/birdseye/ajaxreqs.php on line 78
24226 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-utils.inc.php on line 1032
24226 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-utils.inc.php on line 1033
24226 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-utils.inc.php on line 1034
24226 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-utils.inc.php on line 1035
24226 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-utils.inc.php on line 1036
24226 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-utils.inc.php on line 1037
24226 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-utils.inc.php on line 1038
24226 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-utils.inc.php on line 1043
24226 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-utils.inc.php on line 1044
24226 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-utils.inc.php on line 1045
24226 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-utils.inc.php on line 1046
24226 PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-utils.inc.php on line 1047
24226 PHP Notice: Undefined index: problem_acknowledged in /usr/local/nagiosxi/html/includes/utils-status.inc.php on line 184
24226 PHP Notice: Undefined index: service_id in /usr/local/nagiosxi/html/includes/utils-status.inc.php on line 234
1 PHP Notice: Undefined variable: count in /usr/local/nagiosxi/html/includes/components/tracerouteaction/traceroute.php on line 66
24226 PHP Notice: Undefined variable: display_service_display_name in /usr/local/nagiosxi/html/includes/utils-status.inc.php on line 222
1 PHP Notice: Undefined variable: search in /usr/local/nagiosxi/html/admin/users.php on line 173
4 PHP Warning: Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/birdseye/ajaxreqs.php on line 78