Stuck on - "Waiting for configuration verification"

[solarisfire]

So I've removed everything from the xi_commands table in the SQL DB on the off chance that something was stuck in there.

I tried submitting a new host check command, and the entry pops up in the DB.

Code: Select all

MariaDB [nagiosxi]> select * from xi_commands \G;
*************************** 1. row ***************************
        command_id: 1
          group_id: 0
      submitter_id: 1
    beneficiary_id: 0
           command: 16
   submission_time: 2022-01-20 15:07:12
        event_time: 2022-01-20 15:07:12
    frequency_type: 0
   frequency_units: 0
frequency_interval: 0
   processing_time: NULL
       status_code: 0
       result_code: 0
      command_data: {"host_name":"***","cmd":98,"start_time":1642691226}
            result: NULL
1 row in set (0.00 sec)

But nothing happens...

Is there some process that is meant to consume commands from this list and actually run them? That possibly isn't running...

Struggling to find documentation everywhere.

We've paid money for NagiosXI to try and simplify the management of our Nagios monitoring over Nagios Core, and it's proving to be a whole lot more hastle than Nagios Core ever was...

[pbroste]

Hello @solarisfire

Want to take a look at your Nagios XI System Profile so we can see what is going on and get this ironed out.

To send us your system profile.

• Login to the Nagios XI GUI using a web browser.
• Click the "Admin" > "System Profile" Menu
• Click the "Download Profile" button
• Save the profile.zip file and share

or

Code: Select all

rm -rf /usr/local/nagiosxi/var/components/profile.zip
/usr/local/nagiosxi/scripts/components/getprofile.sh SUPPORT

Then send the resulting /usr/local/nagiosxi/var/components/profile.zip​ file via Private Message.

Thanks,
Perry

[solarisfire]

profile.zip sent via PM.

Had to build on the cli. WebGUI returned:

PROFILE BUILD FAILED
Array
(
)
CODE: 1

[pbroste]

Hello @solarisfire

Thanks for sending over the System Profile, a couple of things that are of interest.

Want to touch on and circle back to message 'Could not resolve host: api.nagios.com; Name or service not known'.

Are we able to get there?

Code: Select all

hostname
ping api.nagios.com -c 4
ping api.localhost -c 4
wget api.nagios.com

Let's take a look at the output results on the api command for 'applyconfig':

Code: Select all

curl -XPOST "https://yourhostnameaddresshere/nagiosxi/api/v1/system/applyconfig?apikey=yourapikeyhere" -k -vvv

Example results should look like this:

....redacted......
< Transfer-Encoding: chunked
< Content-Type: application/json
<
{"success":"Apply config command has been sent to the backend.","command_id":403}
* Connection #0 to host xxx.xxx.xxx.xxx left intact

Please let use know how things look.

Thanks,
Perry

[solarisfire]

This server is on an airgapped offline network.

It shouldn't be able to resolve the DNS for api.nagios.com, nor should it ever be able contact that address...

I've sent the output of the curl command in a PM.

[pbroste]

Hello @solarisfire

Thanks for bringing airgap to my attention again,

By running the api applyconfig command we have verified that the command was completed successfully and can be verified by watching: tail -F /usr/local/nagiosxi/var/cmdsubsys.log

{"success":"Apply config command has been sent to the backend.","command_id":3}
* Connection #0 to host localhost left intact

Let's go ahead and look at the Apache logs to what is going on there:

Code: Select all

tar -czvf /tmp/apache2logs.tar.gz /var/log/httpd/

Please send over the '/tmp/apache2log.tar.gz' when you get a chance.

Thanks,
Perry

[solarisfire]

It's not being processed successfully though...

cmdsubsys.log just says:

cat /usr/local/nagiosxi/var/cmdsubsys.log
.............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
..........................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
.............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
..........................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
...........................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
...........................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known
............................................................
PROCESSED 0 COMMANDS
Could not resolve host: api.nagios.com; Name or service not known

What process normally writes to this log file? How can I check that the command subsystem is running?

I'll send over the apache logs in a PM.

[solarisfire]

Okay, I've finally managed to work out the answer to my own question...

The commands in the nagiosxi_commands table, are processed by a cron job that runs every minute...

* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1

That command was not being executed due to the nagios user not having a valid homedir configured in /etc/passwd.

A lot of pain for a very simple configuration issue.

[pbroste]

Hello @solarisfire

Thanks for sending over the Apache logs, on review did not see any show stoppers. Let's take a look at the group/own/permissions:

Code: Select all

ls -lahrtR /usr/local/nagiosxi/* > /tmp/nagiosxidir.txt

Code: Select all

ls -lahrtR /usr/local/nagios/* > /tmp/nagiosdir.txt

Please send over the results,
Perry

[solarisfire]

If you see my previous message, setting the correct home directory for the Nagios user fixed all of these issues.

The cron jobs seem to be core to how nagiosxi works, and things are very broken if nagiosxi cron is unable to run.