Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Stuck on - "Waiting for configuration verification"

https://support.nagios.com/forum/viewtopic.php?t=64359

Page 2 of 3

Re: Stuck on - "Waiting for configuration verification"

Posted: Thu Jan 20, 2022 10:19 am
by solarisfire
So I've removed everything from the xi_commands table in the SQL DB on the off chance that something was stuck in there.

I tried submitting a new host check command, and the entry pops up in the DB.

Code: Select all

MariaDB [nagiosxi]> select * from xi_commands \G;
*************************** 1. row ***************************
        command_id: 1
          group_id: 0
      submitter_id: 1
    beneficiary_id: 0
           command: 16
   submission_time: 2022-01-20 15:07:12
        event_time: 2022-01-20 15:07:12
    frequency_type: 0
   frequency_units: 0
frequency_interval: 0
   processing_time: NULL
       status_code: 0
       result_code: 0
      command_data: {"host_name":"***","cmd":98,"start_time":1642691226}
            result: NULL
1 row in set (0.00 sec)
But nothing happens...

Is there some process that is meant to consume commands from this list and actually run them? That possibly isn't running...

Struggling to find documentation everywhere.

We've paid money for NagiosXI to try and simplify the management of our Nagios monitoring over Nagios Core, and it's proving to be a whole lot more hastle than Nagios Core ever was...

Re: Stuck on - "Waiting for configuration verification"

Posted: Thu Jan 20, 2022 1:12 pm
by pbroste
Hello @solarisfire

Want to take a look at your Nagios XI System Profile so we can see what is going on and get this ironed out.

To send us your system profile.
  • Login to the Nagios XI GUI using a web browser.
  • Click the "Admin" > "System Profile" Menu
  • Click the "Download Profile" button
  • Save the profile.zip file and share
or

Code: Select all

rm -rf /usr/local/nagiosxi/var/components/profile.zip
/usr/local/nagiosxi/scripts/components/getprofile.sh SUPPORT

Then send the resulting /usr/local/nagiosxi/var/components/profile.zip​ file via Private Message.

Thanks,
Perry

Re: Stuck on - "Waiting for configuration verification"

Posted: Thu Jan 20, 2022 5:49 pm
by solarisfire
profile.zip sent via PM.

Had to build on the cli. WebGUI returned:

PROFILE BUILD FAILED
Array
(
)
CODE: 1

Re: Stuck on - "Waiting for configuration verification"

Posted: Fri Jan 21, 2022 10:23 am
by pbroste
Hello @solarisfire

Thanks for sending over the System Profile, a couple of things that are of interest.

Want to touch on and circle back to message 'Could not resolve host: api.nagios.com; Name or service not known'.

Are we able to get there?

Code: Select all

hostname
ping api.nagios.com -c 4
ping api.localhost -c 4
wget api.nagios.com
Let's take a look at the output results on the api command for 'applyconfig':

Code: Select all

curl -XPOST "https://yourhostnameaddresshere/nagiosxi/api/v1/system/applyconfig?apikey=yourapikeyhere" -k -vvv
Example results should look like this:
....redacted......
< Transfer-Encoding: chunked
< Content-Type: application/json
<
{"success":"Apply config command has been sent to the backend.","command_id":403}
* Connection #0 to host xxx.xxx.xxx.xxx left intact
Please let use know how things look.

Thanks,
Perry

Re: Stuck on - "Waiting for configuration verification"

Posted: Fri Jan 21, 2022 11:31 am
by solarisfire
This server is on an airgapped offline network.

It shouldn't be able to resolve the DNS for api.nagios.com, nor should it ever be able contact that address...

I've sent the output of the curl command in a PM.

Re: Stuck on - "Waiting for configuration verification"

Posted: Fri Jan 21, 2022 4:14 pm
by pbroste
Hello @solarisfire

Thanks for bringing airgap to my attention again,

By running the api applyconfig command we have verified that the command was completed successfully and can be verified by watching: tail -F /usr/local/nagiosxi/var/cmdsubsys.log
{"success":"Apply config command has been sent to the backend.","command_id":3}
* Connection #0 to host localhost left intact
Let's go ahead and look at the Apache logs to what is going on there:

Code: Select all

tar -czvf /tmp/apache2logs.tar.gz /var/log/httpd/
Please send over the '/tmp/apache2log.tar.gz' when you get a chance.

Thanks,
Perry

Re: Stuck on - "Waiting for configuration verification"

Posted: Mon Jan 24, 2022 9:13 am
by solarisfire
It's not being processed successfully though...

cmdsubsys.log just says:

cat /usr/local/nagiosxi/var/cmdsubsys.log
.............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
..........................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
.............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
..........................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
...........................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
...........................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known

What process normally writes to this log file? How can I check that the command subsystem is running?

I'll send over the apache logs in a PM.

Re: Stuck on - "Waiting for configuration verification"

Posted: Mon Jan 24, 2022 12:01 pm
by solarisfire
Okay, I've finally managed to work out the answer to my own question...

The commands in the nagiosxi_commands table, are processed by a cron job that runs every minute...

* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1

That command was not being executed due to the nagios user not having a valid homedir configured in /etc/passwd.

A lot of pain for a very simple configuration issue.

Re: Stuck on - "Waiting for configuration verification"

Posted: Mon Jan 24, 2022 5:13 pm
by pbroste
Hello @solarisfire

Thanks for sending over the Apache logs, on review did not see any show stoppers. Let's take a look at the group/own/permissions:

Code: Select all

ls -lahrtR /usr/local/nagiosxi/* > /tmp/nagiosxidir.txt

Code: Select all

ls -lahrtR /usr/local/nagios/* > /tmp/nagiosdir.txt
Please send over the results,
Perry

Re: Stuck on - "Waiting for configuration verification"

Posted: Tue Jan 25, 2022 5:21 am
by solarisfire
If you see my previous message, setting the correct home directory for the Nagios user fixed all of these issues.

The cron jobs seem to be core to how nagiosxi works, and things are very broken if nagiosxi cron is unable to run.
All times are UTC-05:00
Page 2 of 3

Powered by phpBB® Forum Software © phpBB Limited