upgrade to 5.7.3 hosts and services stay Pending

[jweijters]

Hi I upgraded my NagiosXI test environment from 5.6.14 to 5.7.3, however in the nagiosXI console the status of the hosts and services stay pending.

services_pending.JPG

In NagiosXI I see the Last Check date of services and hosts 1970-01-01 00:00:01

In NagiosCore I do see the services being updated. I also see that gearman is handling the checks.
I also see that events are being handled.

Some research in the database shows:
I selected the hostcheck of a random host and the hoststatus of this same host.
the hostcheck shows the host check is succesful, however the hoststatus is pendng with dates at 1970-01-01 00:00:01

Code: Select all

MariaDB [nagios]> select * from nagios_hostchecks where host_object_id = '88950' ;
+--------------+-------------+----------------+------------+--------------+-----------------------+--------------------+-------+------------+---------------------+-----------------+---------------------+---------------+-------------------+-----------------------------+--------------+---------+---------------+----------------+--------------------+-------------+--------------------------------------------+-------------+--------------------------------------------------------------+
| hostcheck_id | instance_id | host_object_id | check_type | is_raw_check | current_check_attempt | max_check_attempts | state | state_type | start_time          | start_time_usec | end_time            | end_time_usec | command_object_id | command_args                | command_line | timeout | early_timeout | execution_time | latency            | return_code | output                                     | long_output | perfdata                                                     |
+--------------+-------------+----------------+------------+--------------+-----------------------+--------------------+-------+------------+---------------------+-----------------+---------------------+---------------+-------------------+-----------------------------+--------------+---------+---------------+----------------+--------------------+-------------+--------------------------------------------+-------------+--------------------------------------------------------------+
|        67325 |           1 |          88950 |          0 |            0 |                     1 |                 10 |     0 |          1 | 2020-10-09 09:00:21 |          956815 | 2020-10-09 09:00:25 |        990462 |             23359 | 500,50%!700,70% -t 30!!!!!! |              |      30 |             0 |       4.033647 | 22.895861864089966 |           0 | PING OK - Packet loss = 0%, RTA = 23.56 ms |             | rta=23.563000ms;500.000000;700.000000;0.000000 pl=0%;50;70;0 |
+--------------+-------------+----------------+------------+--------------+-----------------------+--------------------+-------+------------+---------------------+-----------------+---------------------+---------------+-------------------+-----------------------------+--------------+---------+---------------+----------------+--------------------+-------------+--------------------------------------------+-------------+--------------------------------------------------------------+
1 row in set (0.00 sec)

MariaDB [nagios]> select * from nagios_hoststatus where  host_object_id = '88950' ;
+---------------+-------------+----------------+---------------------+--------------------------+-------------+----------+---------------+------------------+---------------------+-----------------------+--------------------+---------------------+---------------------+------------+---------------+---------------------+------------------------+-----------------+---------------------+---------------------+-----------------------+------------+---------------------+---------------------+-----------------------+-----------------------+-------------------------------+----------------------+-----------------------------+------------------------+-----------------------+-----------------------+------------------------+-------------+----------------------+---------+----------------+--------------------------+----------------------------+--------------------------+------------------+--------------------------+---------------+---------------+-----------------------+----------------------+----------------------------+
| hoststatus_id | instance_id | host_object_id | status_update_time  | output                   | long_output | perfdata | current_state | has_been_checked | should_be_scheduled | current_check_attempt | max_check_attempts | last_check          | next_check          | check_type | check_options | last_state_change   | last_hard_state_change | last_hard_state | last_time_up        | last_time_down      | last_time_unreachable | state_type | last_notification   | next_notification   | no_more_notifications | notifications_enabled | problem_has_been_acknowledged | acknowledgement_type | current_notification_number | passive_checks_enabled | active_checks_enabled | event_handler_enabled | flap_detection_enabled | is_flapping | percent_state_change | latency | execution_time | scheduled_downtime_depth | failure_prediction_enabled | process_performance_data | obsess_over_host | modified_host_attributes | event_handler | check_command | normal_check_interval | retry_check_interval | check_timeperiod_object_id |
+---------------+-------------+----------------+---------------------+--------------------------+-------------+----------+---------------+------------------+---------------------+-----------------------+--------------------+---------------------+---------------------+------------+---------------+---------------------+------------------------+-----------------+---------------------+---------------------+-----------------------+------------+---------------------+---------------------+-----------------------+-----------------------+-------------------------------+----------------------+-----------------------------+------------------------+-----------------------+-----------------------+------------------------+-------------+----------------------+---------+----------------+--------------------------+----------------------------+--------------------------+------------------+--------------------------+---------------+---------------+-----------------------+----------------------+----------------------------+
|       2717611 |           1 |          88950 | 2020-10-09 07:28:04 | Host check is pending... |             |          |             0 |                0 |                   0 |                     0 |                  0 | 1970-01-01 00:00:01 | 1970-01-01 00:00:01 |          0 |             0 | 1970-01-01 00:00:01 | 1970-01-01 00:00:01    |               0 | 1970-01-01 00:00:01 | 1970-01-01 00:00:01 | 1970-01-01 00:00:01   |          1 | 1970-01-01 00:00:01 | 1970-01-01 00:00:01 |                     0 |                     1 |                             0 |                    0 |                           0 |                      0 |                     1 |                     0 |                      0 |           0 |                    0 |       0 |              0 |                        0 |                          0 |                        0 |                0 |                        0 |               |               |                     0 |                    0 |                          0 |
+---------------+-------------+----------------+---------------------+--------------------------+-------------+----------+---------------+------------------+---------------------+-----------------------+--------------------+---------------------+---------------------+------------+---------------+---------------------+------------------------+-----------------+---------------------+---------------------+-----------------------+------------+---------------------+---------------------+-----------------------+-----------------------+-------------------------------+----------------------+-----------------------------+------------------------+-----------------------+-----------------------+------------------------+-------------+----------------------+---------+----------------+--------------------------+----------------------------+--------------------------+------------------+--------------------------+---------------+---------------+-----------------------+----------------------+----------------------------+
1 row in set (0.00 sec)

How can I fix this problem?

kind regards,
Joris Weijters

[dchurch]

Hi!

We have found an issue with check results not being updated if there is a crashed database table, so I would go ahead and run the repair script.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

Let us know the results. If the issue persists, please attach or PM a System Profile tarball? You can download it if you go to Admin (top of screen), then click System Config (left menu) => System Profile, then click Download Profile.

Thank you.

[jweijters]

Hi,

I updated my NagiosXI test again to the latest version 5.7.4
and ran the repair database. however hosts and services are still pending
The last check is at 1970-01-01 00:00:01
In the attachement the output of the database repair.
I uploaded the profile

Moderator's Note: The profile has been shared with the support team but has been removed from the public forum.

[dchurch]

The reason the service checks are in a pending state is because NDOUtils is in a state where it can't make updates to the database.

The recommended course of action is to downgrade NDOUtils to a previous version.

DOWNGRADE NDO WITH OFFLOADED DB

Code: Select all

service nagios stop
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi

If you have an offloaded DB you'll need to do these things:

• Edit /tmp/nagiosxi/xi-sys.cfg and update 'mysqlpass' value.
• Edit /tmp/nagiosxi/subcomponents/ndoutils/mods/cfg/ndo2db.cfg and update 'db_host', 'db_user', and 'db_pass' values.
• Edit /tmp/nagiosxi/subcomponents/ndoutils/install and /tmp/nagiosxi/subcomponents/ndoutils/post-install to update all calls to mysql to include -h <db_ip>

Then continue as normal.

Code: Select all

cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db
service ndo2db start

Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is uncommented:

Code: Select all

broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Make sure this line is commented:

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service:

Code: Select all

systemctl start nagios