Cannot add hosts to services in NagiosXI 5.8.5

[TreviPay]

Hosts cannot be added to services in NagiosXI 5.8.5

see nag1. This is an example service.

see nag2. I add 8 hosts to the service and click "save".

see nag3. Changes cannot be applied because service has no hosts.

see nag4. I go back to confirm that the recently added hosts are indeed there, but there are no assigned hosts.

Can you help us correct whatever is causing this?

[pbroste]

Hello @TreviPay

There may be an issue writing to the database. However, to start, let's go to the Core Configuration Manager - CCM to make sure that things are good.

To start let's manually delete the tmp's we see sitting in the import directory:

Code: Select all

rm -rf /usr/local/nagios/etc/import/*

Then:

• 1: Terminal command list all running /bin/nagios -> ps -aux | grep -E '/bin/nagios'
• 2: Terminal command -> killall -9 nagios (or pkill nagios)
• 3: Terminal command check to see if /bin/nagios processes are stopped
• 4: Restart nagios.service by terminal command: systemctl restart nagios
• 5: Head over to the Nagios XI web console ==> Core Configuration Manager (CCM) ==> Config File Management ==> [Delete Files] ==> [Write Files] ==> [Verify Files]
• 6: Core Configuration Manager (CCM) ==> Under Quick Tools ==> "Apply Configuration"
• 7: Restart nagios.service by terminal command: systemctl restart nagios
• 8: Verify

If the issue still persists, then let's run through the database repair.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

Bounce the nagios.service and let us know the results after you head over to CCM and verify that the host and services look good and verify that there are no errors in core by:

Code: Select all

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

If the issue persists, please PM your system profile for us to review.

To send us your system profile.

• Login to the Nagios XI GUI using a web browser.
• Click the "Admin" > "System Profile" Menu
• Click the "Download Profile" button
• Save the profile.zip file and share in a private message or upload it to the post/ticket

Thanks,
Perry

[TreviPay]

I tried your suggestions, but the issue persists. I sent you a PM with the profile attached.

[pbroste]

Hello @TreviPay

Thanks for following up and providing the System Profile. After review, we see that there are several attempts made to delete the TOMCAT.cfg but we see that it still resides in the nagiosql database on line item 994 in tbl_service.

Typically, running through the CCM > Delete > Write > Verify and the database repair will clean things up and resolve this issue. Would like to head over to the database repair and do some more granular repairs and checks.

Code: Select all

service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f nagios_service
service mysqld start

Clear the locks:

Code: Select all

rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php

Run through all again and bounce the nagios and database services.

Please let us know how things good,
Perry

[TreviPay]

I ran through the steps you provided, but the issue still persists.

Unfortunately, now there are new problems:

The *Apply Configuration button in the CCM no longer works and instead outputs "SQL Error [nagiosxi] : Incorrect datetime value: '0000-00-00 00:00:00' for column 'processing_time' at row 1"

I can no longer restart the nagios monitoring engine using the Admin > System Status action.

Also, I can no longer force immediate checks on services; it only outputs "Your request was not processed in a timely manner. It may still execute, as the server may be temporarily busy."

I also tried deleting the TOMCAT.cfg entries from the CCM, applying the changes via Delete Files, Write Files, Verify, and Restart Nagios and then adding the specific TOMCAT service that we need back, but when I add hosts to the service, they don't get saved when I click the save button.

When I was executing the myisamchk command I couldn't find "nagios_service" so I ran it on "nagios_services" instead. Hopefully that was the right one.

It seems like Nagios is behaving as if it didn't have access to whichever database(s) it is trying to read/write to...

[TreviPay]

I just realized that "database repair" was a link to documentation on a database repair process. I'll run through that and let you know where that lands us.

[TreviPay]

I went through the database repair documentation and I didn't see any errors in the output. That didn't seem to fix anything, though. The problem remains

[pbroste]

Hello @TreviPay

Thanks for following up with the details, appears that we are receiving a date/time issue and want to hit up that by entering the following:

Code: Select all

mysql -u root -p'nagiosxi' -e "SELECT @@GLOBAL.sql_mode;"

Please bounce the database service and then run through the CCM > Delete > Write > Verify. Add your TOMCAT host and service config through CCM.

You can verify the TOMCAT config in the database by executing the command to verify MySQL is alive.

Code: Select all

mysql -uroot -pnagiosxi nagiosql

Code: Select all

select * from tbl_service where id = 994;

Thanks,
Perry

[TreviPay]

Code: Select all

mysql -u root -p'nagiosxi' -e "SELECT @@GLOBAL.sql_mode;"

had extra output (I would paste what it was but my scrollback buffer ate it.)
now it is set only to:

Code: Select all

+--------------------------------------------+
| @@GLOBAL.sql_mode                          |
+--------------------------------------------+
| STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION |
+--------------------------------------------+

That fixed the issue with the apply configuration button and I can once again force immediate checks of services.

Unfortunately the original problem persists. It affect any service config file that I try to add hosts to or remove hosts from.
For example, /usr/local/nagios/etc/services/DISK.cfg contains the following:

Code: Select all

define service {
    host_name              opweb14.multiservice.com
    service_description    DISK-VAR-TOMCAT7
    use                    generic-service
    hostgroup_name         DISK-VAR-TOMCAT7
    check_command          check-disk-nrpe!/var/tomcat7!50%!25%!!!!!
    contact_groups         webadmin-group-notify,webmaster-group-email
    register               1
}

If I try to add another host to that service, the file entry becomes:

Code: Select all

define service {
    service_description    DISK-VAR-TOMCAT7
    use                    generic-service
    check_command          check-disk-nrpe!/var/tomcat7!50%!25%!!!!!
    contact_groups         webadmin-group-notify,webmaster-group-email
    register               1
}

the host_name line goes missing (as well as the hostgroup_name line which I have now noticed.)

Code: Select all

mysql> select * from tbl_service where id = 994;
Empty set (0.00 sec)

The above command returns nothing, but maybe that's because I have removed the TOMCAT service and re-added it (maybe the id number has changed?).

[TreviPay]

I re-added the example TOMCAT service and this is the output (The service is temporarily disabled so that the verification check passes)

Code: Select all

mysql> select * from tbl_service where id = 1815;
+------+-------------+-----------+----------------------+----------------+---------------------------+---------------------+--------------+---------------+--------------------------+--------------+-------------------------+-------------------------------+-------------+---------------+--------------------+----------------+----------------+-----------------------+------------------------+--------------+-------------------+---------------------+-----------------+---------------------+---------------+-----------------------+--------------------+---------------------+------------------------+------------------------+-------------------+---------------------------+------------------------------+-----------------------+--------------------------+---------------------+----------------------+-----------------------+----------+---------------------+----------------+---------------------------+------------------+-------+-----------+------------+------------+----------------+---------------+------+--------+---------------------+---------------+-----------+
| id   | config_name | host_name | host_name_tploptions | hostgroup_name | hostgroup_name_tploptions | service_description | display_name | servicegroups | servicegroups_tploptions | use_template | use_template_tploptions | check_command                 | is_volatile | initial_state | max_check_attempts | check_interval | retry_interval | active_checks_enabled | passive_checks_enabled | check_period | parallelize_check | obsess_over_service | check_freshness | freshness_threshold | event_handler | event_handler_enabled | low_flap_threshold | high_flap_threshold | flap_detection_enabled | flap_detection_options | process_perf_data | retain_status_information | retain_nonstatus_information | notification_interval | first_notification_delay | notification_period | notification_options | notifications_enabled | contacts | contacts_tploptions | contact_groups | contact_groups_tploptions | stalking_options | notes | notes_url | action_url | icon_image | icon_image_alt | use_variables | name | active | last_modified       | access_rights | config_id |
+------+-------------+-----------+----------------------+----------------+---------------------------+---------------------+--------------+---------------+--------------------------+--------------+-------------------------+-------------------------------+-------------+---------------+--------------------+----------------+----------------+-----------------------+------------------------+--------------+-------------------+---------------------+-----------------+---------------------+---------------+-----------------------+--------------------+---------------------+------------------------+------------------------+-------------------+---------------------------+------------------------------+-----------------------+--------------------------+---------------------+----------------------+-----------------------+----------+---------------------+----------------+---------------------------+------------------+-------+-----------+------------+------------+----------------+---------------+------+--------+---------------------+---------------+-----------+
| 1815 | TOMCAT      |         1 |                    2 |              0 |                         2 | TOMCAT-phonestats   |              |             0 |                        2 |            1 |                       2 | 227!:8080/queuemetrics!!!!!!! |           2 |               |               NULL |           NULL |           NULL |                     2 |                      2 |            0 |                 2 |
 2 |               2 |                NULL |             0 |                     2 |               NULL |                NULL |                      2 |                        |                 2 |
 2 |                            2 |                  NULL |                     NULL |                   0 |                      |                     2 |        0 |                   2 |              0 |
         2 |                  |       |           |            |            |                |             0 |      | 0      | 2021-08-17 10:05:37 | NULL          |         1 |
+------+-------------+-----------+----------------------+----------------+---------------------------+---------------------+--------------+---------------+--------------------------+--------------+-------------------------+-------------------------------+-------------+---------------+--------------------+----------------+----------------+-----------------------+------------------------+--------------+-------------------+---------------------+-----------------+---------------------+---------------+-----------------------+--------------------+---------------------+------------------------+------------------------+-------------------+---------------------------+------------------------------+-----------------------+--------------------------+---------------------+----------------------+-----------------------+----------+---------------------+----------------+---------------------------+------------------+-------+-----------+------------+------------+----------------+---------------+------+--------+---------------------+---------------+-----------+
1 row in set (0.01 sec)