Apply config failing with no error - table issue in mysql db

[optionstechnology]

Hi,

I'm currently unable to apply configuration on one of our hosts and its not showing any errors in the output. Have tried numerous fixes found online but no luck (Delete, Verify, Write etc):
See screenshot attached

The only issue I can find anywhere (after running the /usr/local/nagiosxi/scripts/repair_databases.sh script) is that the nagios_commenthistory table is having issues

See the output from various commands below:

[root@HOSTNAME nagios]# cd /usr/local/nagiosxi/scripts
[root@HOSTNAME scripts]# ./reconfigure_nagios.sh

--- reset_config_perms.sh ------------
> Setting script permissions
> Setting CCM script permissions
> Setting special script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting libexec directory permissions
> Setting Nagios XI config permissions
> Setting NOM checkpoint user:group permissions
> + Setting Nagios Core corelog.newobjects user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------

--- ccm_import.php -------------------
> Setting import directory: /usr/local/nagios/etc/import/
> Importing config files into the CCM
No files to import
--------------------------------------

--- ccm_export.php -------------------
> Writing CCM configuration to Nagios files
Finished writing out configuraton
--------------------------------------

--------------------------------------
> Verifying configuration with Nagios Core
> Output:
Nagios Core 4.4.5
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2019-08-20
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...

###HERE WE JUST HAD SOME WARNINGS ABOUT DUPLICATES

> Return Code: 139
--------------------------------------
/usr/local/nagiosxi/nom/checkpoints/nagioscore/errors /usr/local/nagiosxi/scripts
tar: Removing leading `/' from member names
/usr/local/nagiosxi/scripts
LATEST NOM SNAPSHOT: /usr/local/nagiosxi/nom/checkpoints/nagioscore/1607592543.tar.gz
/ /usr/local/nagiosxi/scripts
RESTORING NOM SNAPSHOT : /usr/local/nagiosxi/nom/checkpoints/nagioscore/1607592543.tar.gz
/usr/local/nagiosxi/scripts

--- reset_config_perms.sh ------------
> Setting script permissions
> Setting CCM script permissions
> Setting special script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting libexec directory permissions
> Setting Nagios XI config permissions
> Setting NOM checkpoint user:group permissions
> + Setting Nagios Core corelog.newobjects user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------



[root@HOSTNAME scripts]# tail /var/log/mysqld.log
201210 12:58:32 [Note] /usr/libexec/mysqld: Shutdown complete

201210 12:58:32 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
201210 12:59:18 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
201210 12:59:18 InnoDB: Initializing buffer pool, size = 8.0M
201210 12:59:18 InnoDB: Completed initialization of buffer pool
201210 12:59:18 InnoDB: Started; log sequence number 0 1966086051
201210 12:59:18 [Note] Event Scheduler: Loaded 0 events
201210 12:59:18 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

[root@HOSTNAME nagios]# perror 139
MySQL error code 139: Too big row



[root@HOSTNAME scripts]# /usr/local/nagiosxi/scripts/repair_databases.sh
DATABASE: nagios
TABLE:
/var/lib/mysql/nagios /usr/local/nagiosxi/scripts
Stopping mysqld: [ OK ]
- recovering (with sort) MyISAM-table 'nagios_acknowledgements'
Data records: 48502
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_commands'
Data records: 218
- Fixing index 1
- Fixing index 2

---------

/usr/bin/myisamchk: File 'nagios_commenthistory.MYD' not found (Errcode: 2)
myisamchk: error: File 'nagios_commenthistory' doesn't exist

---------

...............

Starting mysqld: [ OK ]
/usr/local/nagiosxi/scripts

===============
REPAIR COMPLETE
===============
Stopping ndo2db: done.
Starting ndo2db: done.
Stopping nagios: .done.
Starting nagios: done.

=======================
nagios database repair FAILED, please check output above!
nagiosql database repair succeeded
nagiosxi database repair succeeded

[root@HOSTNAME scripts]# cd /var/lib/mysql/nagios
[root@HOSTNAME nagios]# ls -l
total 5670356
-rw-r--r-- 1 root root 13970 Jun 20 2018 check_network_device
-rw-rw---- 1 mysql mysql 61 Nov 1 2017 db.opt
-rw-rw---- 1 mysql mysql 9100 Nov 1 2017 nagios_acknowledgements.frm
-rw-rw---- 1 mysql mysql 3350864 Dec 10 13:12 nagios_acknowledgements.MYD
-rw-rw---- 1 mysql mysql 1499136 Dec 10 13:12 nagios_acknowledgements.MYI
-rw-rw---- 1 mysql mysql 8746 Nov 1 2017 nagios_commands.frm
-rw-rw---- 1 mysql mysql 25340 Dec 10 13:15 nagios_commands.MYD
-rw-rw---- 1 mysql mysql 9216 Dec 10 13:15 nagios_commands.MYI
-rw-rw---- 1 mysql mysql 9340 Nov 1 2017 nagios_commenthistory.frm
-rw-rw---- 1 mysql mysql 2493596 Jul 9 09:36 nagios_commenthistory.MYD-2ac8_a49
-rw-r--r-- 1 root root 317440 Jul 9 09:36 nagios_commenthistory.MYI
-rw-rw---- 1 mysql mysql 9220 Nov 1 2017 nagios_comments.frm
-rw-rw---- 1 mysql mysql 1078908 Dec 10 13:15 nagios_comments.MYD
-rw-rw---- 1 mysql mysql 274432 Dec 10 13:15 nagios_comments.MYI
-rw-rw---- 1 mysql mysql 8726 Nov 1 2017 nagios_configfiles.frm
-rw-rw---- 1 mysql mysql 44 Dec 10 13:15 nagios_configfiles.MYD
-rw-rw---- 1 mysql mysql 5120 Dec 10 13:15 nagios_configfiles.MYI
-rw-rw---- 1 mysql mysql 8760 Nov 1 2017 nagios_configfilevariables.frm
-rw-rw---- 1 mysql mysql 9132 Dec 10 13:15 nagios_configfilevariables.MYD
-rw-rw---- 1 mysql mysql 4096 Dec 10 13:15 nagios_configfilevariables.MYI
-rw-rw---- 1 mysql mysql 9262 Nov 1 2017 nagios_conninfo.frm
-rw-rw---- 1 mysql mysql 467032 Dec 10 13:22 nagios_conninfo.MYD
-rw-rw---- 1 mysql mysql 57344 Dec 10 13:22 nagios_conninfo.MYI
-rw-rw---- 1 mysql mysql 8760 Nov 1 2017 nagios_contact_addresses.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:12 nagios_contact_addresses.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:12 nagios_contact_addresses.MYI
-rw-rw---- 1 mysql mysql 8748 Nov 1 2017 nagios_contactgroup_members.frm
-rw-rw---- 1 mysql mysql 270 Dec 10 13:15 nagios_contactgroup_members.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:15 nagios_contactgroup_members.MYI
-rw-rw---- 1 mysql mysql 8768 Nov 1 2017 nagios_contactgroups.frm
-rw-rw---- 1 mysql mysql 352 Dec 10 13:15 nagios_contactgroups.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:15 nagios_contactgroups.MYI
-rw-rw---- 1 mysql mysql 8856 Nov 1 2017 nagios_contact_notificationcommands.frm
-rw-rw---- 1 mysql mysql 4248 Dec 10 13:15 nagios_contact_notificationcommands.MYD
-rw-rw---- 1 mysql mysql 7168 Dec 10 13:15 nagios_contact_notificationcommands.MYI
-rw-rw---- 1 mysql mysql 9000 Nov 1 2017 nagios_contactnotificationmethods.frm
-rw-rw---- 1 mysql mysql 41324 Dec 10 13:12 nagios_contactnotificationmethods.MYD
-rw-rw---- 1 mysql mysql 48128 Dec 10 13:12 nagios_contactnotificationmethods.MYI
-rw-rw---- 1 mysql mysql 8928 Nov 1 2017 nagios_contactnotifications.frm
-rw-rw---- 1 mysql mysql 36777 Dec 10 13:12 nagios_contactnotifications.MYD
-rw-rw---- 1 mysql mysql 59392 Dec 10 13:12 nagios_contactnotifications.MYI
-rw-rw---- 1 mysql mysql 10026 Nov 1 2017 nagios_contacts.frm
-rw-rw---- 1 mysql mysql 1992 Dec 10 13:15 nagios_contacts.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:15 nagios_contacts.MYI
-rw-rw---- 1 mysql mysql 9240 Nov 1 2017 nagios_contactstatus.frm
-rw-rw---- 1 mysql mysql 816 Dec 10 13:15 nagios_contactstatus.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:15 nagios_contactstatus.MYI
-rw-rw---- 1 mysql mysql 8844 Nov 1 2017 nagios_customvariables.frm
-rw-rw---- 1 mysql mysql 273012 Dec 10 13:15 nagios_customvariables.MYD
-rw-rw---- 1 mysql mysql 342016 Dec 10 13:16 nagios_customvariables.MYI
-rw-rw---- 1 mysql mysql 8870 Nov 1 2017 nagios_customvariablestatus.frm
-rw-rw---- 1 mysql mysql 297964 Dec 10 13:23 nagios_customvariablestatus.MYD
-rw-rw---- 1 mysql mysql 307200 Dec 10 13:23 nagios_customvariablestatus.MYI
-rw-rw---- 1 mysql mysql 8596 Nov 1 2017 nagios_dbversion.frm
-rw-rw---- 1 mysql mysql 20 Dec 10 13:12 nagios_dbversion.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:12 nagios_dbversion.MYI
-rw-rw---- 1 mysql mysql 9490 Nov 1 2017 nagios_downtimehistory.frm
-rw-rw---- 1 mysql mysql 552984 Dec 10 13:15 nagios_downtimehistory.MYD
-rw-rw---- 1 mysql mysql 188416 Dec 10 13:20 nagios_downtimehistory.MYI
-rw-rw---- 1 mysql mysql 9380 Nov 1 2017 nagios_eventhandlers.frm
-rw-rw---- 1 mysql mysql 65844 Dec 10 13:23 nagios_eventhandlers.MYD
-rw-rw---- 1 mysql mysql 12288 Dec 10 13:23 nagios_eventhandlers.MYI
-rw-rw---- 1 mysql mysql 8812 Nov 1 2017 nagios_externalcommands.frm
-rw-rw---- 1 mysql mysql 11765708 Dec 10 13:23 nagios_externalcommands.MYD
-rw-rw---- 1 mysql mysql 399360 Dec 10 13:23 nagios_externalcommands.MYI
-rw-rw---- 1 mysql mysql 9166 Nov 1 2017 nagios_flappinghistory.frm
-rw-rw---- 1 mysql mysql 7515105 Dec 10 13:12 nagios_flappinghistory.MYD
-rw-rw---- 1 mysql mysql 1180672 Dec 10 13:12 nagios_flappinghistory.MYI
-rw-rw---- 1 mysql mysql 9660 Nov 1 2017 nagios_hostchecks.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:12 nagios_hostchecks.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:12 nagios_hostchecks.MYI
-rw-rw---- 1 mysql mysql 8738 Nov 1 2017 nagios_host_contactgroups.frm
-rw-rw---- 1 mysql mysql 15 Dec 10 13:15 nagios_host_contactgroups.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:15 nagios_host_contactgroups.MYI
-rw-rw---- 1 mysql mysql 8718 Nov 1 2017 nagios_host_contacts.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:12 nagios_host_contacts.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:12 nagios_host_contacts.MYI
-rw-rw---- 1 mysql mysql 9108 Nov 1 2017 nagios_hostdependencies.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:12 nagios_hostdependencies.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:12 nagios_hostdependencies.MYI
-rw-rw---- 1 mysql mysql 8778 Nov 1 2017 nagios_hostescalation_contactgroups.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:12 nagios_hostescalation_contactgroups.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:12 nagios_hostescalation_contactgroups.MYI
-rw-rw---- 1 mysql mysql 8758 Nov 1 2017 nagios_hostescalation_contacts.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:12 nagios_hostescalation_contacts.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:12 nagios_hostescalation_contacts.MYI
-rw-rw---- 1 mysql mysql 9148 Nov 1 2017 nagios_hostescalations.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:12 nagios_hostescalations.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:12 nagios_hostescalations.MYI
-rw-rw---- 1 mysql mysql 8730 Nov 1 2017 nagios_hostgroup_members.frm
-rw-rw---- 1 mysql mysql 26970 Dec 10 13:15 nagios_hostgroup_members.MYD
-rw-rw---- 1 mysql mysql 55296 Dec 10 13:15 nagios_hostgroup_members.MYI
-rw-rw---- 1 mysql mysql 8756 Nov 1 2017 nagios_hostgroups.frm
-rw-rw---- 1 mysql mysql 14816 Dec 10 13:15 nagios_hostgroups.MYD
-rw-rw---- 1 mysql mysql 12288 Dec 10 13:15 nagios_hostgroups.MYI
-rw-rw---- 1 mysql mysql 8732 Nov 1 2017 nagios_host_parenthosts.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:12 nagios_host_parenthosts.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:12 nagios_host_parenthosts.MYI
-rw-rw---- 1 mysql mysql 11932 Nov 1 2017 nagios_hosts.frm
-rw-rw---- 1 mysql mysql 54296 Dec 10 13:15 nagios_hosts.MYD
-rw-rw---- 1 mysql mysql 16384 Dec 10 13:16 nagios_hosts.MYI
-rw-rw---- 1 mysql mysql 15368 Nov 1 2017 nagios_hoststatus.frm
-rw-rw---- 1 mysql mysql 98068 Dec 10 13:23 nagios_hoststatus.MYD
-rw-rw---- 1 mysql mysql 86016 Dec 10 13:23 nagios_hoststatus.MYI
-rw-rw---- 1 mysql mysql 8684 Nov 1 2017 nagios_instances.frm
-rw-rw---- 1 mysql mysql 20 Dec 10 13:12 nagios_instances.MYD
-rw-rw---- 1 mysql mysql 2048 Dec 10 13:12 nagios_instances.MYI
-rw-rw---- 1 mysql mysql 8972 Nov 19 2019 nagios_logentries.frm
-rw-rw---- 1 mysql mysql 1346747312 Dec 10 13:23 nagios_logentries.MYD
-rw-rw---- 1 mysql mysql 853286912 Dec 10 13:23 nagios_logentries.MYI
-rw-rw---- 1 mysql mysql 9168 Nov 1 2017 nagios_notifications.frm
-rw-rw---- 1 mysql mysql 241686548 Dec 10 13:23 nagios_notifications.MYD
-rw-rw---- 1 mysql mysql 70907904 Dec 10 13:23 nagios_notifications.MYI
-rw-rw---- 1 mysql mysql 8766 Nov 1 2017 nagios_objects.frm
-rw-rw---- 1 mysql mysql 814452 Dec 10 13:15 nagios_objects.MYD
-rw-rw---- 1 mysql mysql 732160 Dec 10 13:16 nagios_objects.MYI
-rw-rw---- 1 mysql mysql 8948 Nov 1 2017 nagios_processevents.frm
-rw-rw---- 1 mysql mysql 648012 Dec 10 13:15 nagios_processevents.MYD
-rw-rw---- 1 mysql mysql 130048 Dec 10 13:19 nagios_processevents.MYI
-rw-rw---- 1 mysql mysql 10174 Nov 1 2017 nagios_programstatus.frm
-rw-rw---- 1 mysql mysql 136 Dec 10 13:23 nagios_programstatus.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:23 nagios_programstatus.MYI
-rw-rw---- 1 mysql mysql 8706 Nov 1 2017 nagios_runtimevariables.frm
-rw-rw---- 1 mysql mysql 1016 Dec 10 13:15 nagios_runtimevariables.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:20 nagios_runtimevariables.MYI
-rw-rw---- 1 mysql mysql 9332 Nov 19 2019 nagios_scheduleddowntime.frm
-rw-rw---- 1 mysql mysql 1100 Dec 10 13:15 nagios_scheduleddowntime.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:20 nagios_scheduleddowntime.MYI
-rw-rw---- 1 mysql mysql 9626 Nov 1 2017 nagios_servicechecks.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:14 nagios_servicechecks.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:14 nagios_servicechecks.MYI
-rw-rw---- 1 mysql mysql 8750 Nov 1 2017 nagios_service_contactgroups.frm
-rw-rw---- 1 mysql mysql 375 Dec 10 13:15 nagios_service_contactgroups.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:16 nagios_service_contactgroups.MYI
-rw-rw---- 1 mysql mysql 8730 Nov 1 2017 nagios_service_contacts.frm
-rw-rw---- 1 mysql mysql 720 Dec 10 13:15 nagios_service_contacts.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:16 nagios_service_contacts.MYI
-rw-rw---- 1 mysql mysql 9178 Nov 1 2017 nagios_servicedependencies.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:14 nagios_servicedependencies.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:14 nagios_servicedependencies.MYI
-rw-rw---- 1 mysql mysql 8790 Nov 1 2017 nagios_serviceescalation_contactgroups.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:14 nagios_serviceescalation_contactgroups.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:14 nagios_serviceescalation_contactgroups.MYI
-rw-rw---- 1 mysql mysql 8770 Nov 1 2017 nagios_serviceescalation_contacts.frm
-rw-rw---- 1 mysql mysql 1245 Dec 10 13:15 nagios_serviceescalation_contacts.MYD
-rw-rw---- 1 mysql mysql 5120 Dec 10 13:16 nagios_serviceescalation_contacts.MYI
-rw-rw---- 1 mysql mysql 9220 Nov 1 2017 nagios_serviceescalations.frm
-rw-rw---- 1 mysql mysql 851 Dec 10 13:15 nagios_serviceescalations.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:16 nagios_serviceescalations.MYI
-rw-rw---- 1 mysql mysql 8748 Nov 1 2017 nagios_servicegroup_members.frm
-rw-rw---- 1 mysql mysql 37710 Dec 10 13:15 nagios_servicegroup_members.MYD
-rw-rw---- 1 mysql mysql 71680 Dec 10 13:16 nagios_servicegroup_members.MYI
-rw-rw---- 1 mysql mysql 8768 Nov 1 2017 nagios_servicegroups.frm
-rw-rw---- 1 mysql mysql 200 Dec 10 13:15 nagios_servicegroups.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:16 nagios_servicegroups.MYI
-rw-rw---- 1 mysql mysql 8756 Nov 1 2017 nagios_service_parentservices.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:14 nagios_service_parentservices.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:14 nagios_service_parentservices.MYI
-rw-rw---- 1 mysql mysql 11766 Nov 1 2017 nagios_services.frm
-rw-rw---- 1 mysql mysql 1483724 Dec 10 13:15 nagios_services.MYD
-rw-rw---- 1 mysql mysql 302080 Dec 10 13:16 nagios_services.MYI
-rw-rw---- 1 mysql mysql 15448 Nov 1 2017 nagios_servicestatus.frm
-rw-rw---- 1 mysql mysql 3402024 Dec 10 13:23 nagios_servicestatus.MYD
-rw-rw---- 1 mysql mysql 1782784 Dec 10 13:23 nagios_servicestatus.MYI
-rw-rw---- 1 mysql mysql 9174 Nov 1 2017 nagios_statehistory.frm
-rw-rw---- 1 mysql mysql 2898181688 Dec 10 13:23 nagios_statehistory.MYD
-rw-rw---- 1 mysql mysql 352049152 Dec 10 13:23 nagios_statehistory.MYI
-rw-rw---- 1 mysql mysql 9110 Nov 1 2017 nagios_systemcommands.frm
-rw-rw---- 1 mysql mysql 20624 Dec 10 13:23 nagios_systemcommands.MYD
-rw-rw---- 1 mysql mysql 9216 Dec 10 13:23 nagios_systemcommands.MYI
-rw-rw---- 1 mysql mysql 8914 Nov 1 2017 nagios_timedeventqueue.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:15 nagios_timedeventqueue.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:15 nagios_timedeventqueue.MYI
-rw-rw---- 1 mysql mysql 9104 Nov 1 2017 nagios_timedevents.frm
-rw-rw---- 1 mysql mysql 0 Dec 10 13:15 nagios_timedevents.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 10 13:15 nagios_timedevents.MYI
-rw-rw---- 1 mysql mysql 8760 Nov 1 2017 nagios_timeperiods.frm
-rw-rw---- 1 mysql mysql 1032 Dec 10 13:15 nagios_timeperiods.MYD
-rw-rw---- 1 mysql mysql 3072 Dec 10 13:15 nagios_timeperiods.MYI
-rw-rw---- 1 mysql mysql 8794 Nov 1 2017 nagios_timeperiod_timeranges.frm
-rw-rw---- 1 mysql mysql 2163 Dec 10 13:15 nagios_timeperiod_timeranges.MYD
-rw-rw---- 1 mysql mysql 8192 Dec 10 13:15 nagios_timeperiod_timeranges.MYI

[root@HOSTNAME nagios]# mysql -u ndoutils -pn@gweb nagios -e 'TRUNCATE TABLE nagios_commenthistory'
ERROR 1 (HY000) at line 1: Can't create/write to file '/var/lib/mysql/nagios/nagios_commenthistory.MYI' (Errcode: 13)

[root@HOSTNAME nagios]# perror 13
OS error code 13: Permission denied

[root@HOSTNAME nagios]# echo 'repair table nagios_commenthistory use_frm;' | mysql -t -u root -pnagiosxi nagios
+------------------------------+--------+----------+------------------+
| Table | Op | Msg_type | Msg_text |
+------------------------------+--------+----------+------------------+
| nagios.nagios_commenthistory | repair | Error | Can't open table |
| nagios.nagios_commenthistory | repair | error | Corrupt |
+------------------------------+--------+----------+------------------+

[dchurch]

Try this:

Code: Select all

chown -R mysql:mysql /var/lib/mysql/nagios/

Then you should be able to do this:

Code: Select all

echo 'repair table nagios_commenthistory use_frm;' | mysql -t -u root -pnagiosxi nagios

[dchurch]

Just checking in since we haven't heard from you in a while. Did that solve your problem?

[optionstechnology]

Hi,

Not seeing that issue any more but I still cannot Apply Config - error is still blank. See output from the below files when applying config:

[root@HOSTNAME nagios]# tail -Fn0 /usr/local/nagiosxi/var/*.log /usr/local/nagios/var/*.log /var/log/messages /var/log/cron
==> /usr/local/nagiosxi/var/cleaner.log <==

==> /usr/local/nagiosxi/var/cmdsubsys.log <==

==> /usr/local/nagiosxi/var/dbmaint.log <==

==> /usr/local/nagiosxi/var/deadpool.log <==

==> /usr/local/nagiosxi/var/event_handler.log <==

==> /usr/local/nagiosxi/var/eventman.log <==

==> /usr/local/nagiosxi/var/feedproc.log <==

==> /usr/local/nagiosxi/var/load_url.log <==

==> /usr/local/nagiosxi/var/nom.log <==

==> /usr/local/nagiosxi/var/perfdataproc.log <==

==> /usr/local/nagiosxi/var/recurringdowntime.log <==

==> /usr/local/nagiosxi/var/reportengine.log <==

==> /usr/local/nagiosxi/var/snmptt_service_results.log <==

==> /usr/local/nagiosxi/var/sysstat.log <==

==> /usr/local/nagios/var/livestatus.log <==

==> /usr/local/nagios/var/nagios.log <==

==> /usr/local/nagios/var/npcd.log <==

==> /usr/local/nagios/var/perfdata.log <==

==> /var/log/messages <==

==> /var/log/cron <==

==> /usr/local/nagiosxi/var/eventman.log <==
.
==> /usr/local/nagiosxi/var/cmdsubsys.log <==
...
==> /usr/local/nagiosxi/var/eventman.log <==
.
==> /usr/local/nagiosxi/var/cmdsubsys.log <==
PROCESSING COMMAND ID 14085...
PROCESS COMMAND: CMD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
sudo: unknown defaults entry `maxseq'

--- reset_config_perms.sh ------------
> Setting script permissions
> Setting CCM script permissions
> Setting special script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting libexec directory permissions
> Setting Nagios XI config permissions
> Setting NOM checkpoint user:group permissions
> + Setting Nagios Core corelog.newobjects user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------

--- ccm_import.php -------------------
> Setting import directory: /usr/local/nagios/etc/import/
> Importing config files into the CCM
No files to import
--------------------------------------

--- ccm_export.php -------------------
> Writing CCM configuration to Nagios files
Finished writing out configuraton
--------------------------------------

==> /var/log/messages <==
Jan 13 05:01:12 HOSTNAME kernel: nagios[98808]: segfault at 0 ip 0000000000452940 sp 00007fff60b65da0 error 6 in nagios[400000+a7000]

==> /usr/local/nagiosxi/var/cmdsubsys.log <==

--------------------------------------
> Verifying configuration with Nagios Core
> Output:
Nagios Core 4.4.5
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2019-08-20
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Warning: Duplicate definition found for service 'SERVICENAME' on host 'HOSTNAME' (config file '/usr/local/nagios/etc/services/HOSTNAME.cfg', starting on line 52)
MORE LINES ABOUT DUPLICATE DEFINITIONS
> Return Code: 139
--------------------------------------
/usr/local/nagiosxi/nom/checkpoints/nagioscore/errors /usr/local/nagiosxi/scripts

==> /var/log/messages <==
Jan 13 05:01:12 HOSTNAME kernel: nagios[98817]: segfault at 0 ip 0000000000452940 sp 00007fff3c511100 error 6 in nagios[400000+a7000]

==> /usr/local/nagiosxi/var/cmdsubsys.log <==
tar: Removing leading `/' from member names
/usr/local/nagiosxi/scripts
LATEST NOM SNAPSHOT: /usr/local/nagiosxi/nom/checkpoints/nagioscore/1610482684.tar.gz
/ /usr/local/nagiosxi/scripts
RESTORING NOM SNAPSHOT : /usr/local/nagiosxi/nom/checkpoints/nagioscore/1610482684.tar.gz
tar: usr/local/nagios/etc/hosts: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/hosts: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: usr/local/nagios/etc/pnp/pages: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/pnp/pages: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: usr/local/nagios/etc/pnp/check_commands: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/pnp/check_commands: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: usr/local/nagios/etc/pnp: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/pnp: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: usr/local/nagios/etc/import: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: usr/local/nagios/etc/objects: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/objects: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: usr/local/nagios/etc/services: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/services: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: usr/local/nagios/etc/static: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/static: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: usr/local/nagios/etc: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: Exiting with failure status due to previous errors
/usr/local/nagiosxi/scripts
sudo: unknown defaults entry `maxseq'

--- reset_config_perms.sh ------------
> Setting script permissions
> Setting CCM script permissions
> Setting special script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting libexec directory permissions
> Setting Nagios XI config permissions
> Setting NOM checkpoint user:group permissions
> + Setting Nagios Core corelog.newobjects user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------
OUTPUT=--------------------------------------
RETURNCODE=1
PROCESSING COMMAND ID 14086...
PROCESS COMMAND: CMD=1150, DATA=remove
CMDLINE=php /usr/local/nagiosxi/html/includes/components/nagiosbpi/api_tool.php --cmd=syncall

==> /usr/local/nagiosxi/var/eventman.log <==
..^C





Have already tried resetting perms (sudo /usr/local/nagiosxi/scripts/reset_config_perms.sh) followed by Delete Files -> Write Config -> Verify Files

[dchurch]

Jan 13 05:01:12 HOSTNAME kernel: nagios[98808]: segfault at 0 ip 0000000000452940 sp 00007fff60b65da0 error 6 in nagios[400000+a7000]

It would seem that Nagios Core is crashing when Nagios XI tells it to verify configuration. This would explain the empty error message.

I checked with my team, and they've seen this happen once in a blue moon due to a configuration error. It is mostly caused by an * used in an object that would create a lot of objects, like millions, and that caused it to sigsegv.

I would step back over any configuration changes made recently that's causing this to happen and examine the differences.

If you PM me a system profile and I can step thru your configurations to see if I can replicate it on my end.

Further questions

If your system has been running for a while and has seen changes, this might be from a bug that's already been fixed in Nagios Core, but we won't know until I determine the following:

• What distro are you running?
• What version of Nagios XI is installed?
• How was Nagios XI installed?
• Were you told to downgrade Nagios Core at some point?
• If so, what version of Nagios Core did you install?

[optionstechnology]

The only changes that've been made were for removing config (i.e. Hosts or Services, service escalations etc.).

I've reviewed recent changes and can't see any reason why. Also, surely this is a bug with Nagios itself if its not giving anything in the output? Shouldn't it at least point me in the right direction?

What distro are you running? CentOS 6.10 (Final)
What version of Nagios XI is installed? 5.6.14
How was Nagios XI installed?
Were you told to downgrade Nagios Core at some point? No, we're on core version 4.4.5
If so, what version of Nagios Core did you install?

I've sent a profile to you on PM

[dchurch]

Profile received.

While I'm not able to get it to crash while verifying configs, I was able to cause a verification to happen successfully (with a lot of errors).

Can you manually verify the configs by going to Config (top menu) => Core Config Manager => Tools => Config File Management (left panel) and:
1. Click Delete Files
2. Click Write Configs
3. Click Verify Files

If you get any error messages at each step, please post them here.

Full text of the errors I'm seeing when I verify on CentOS 7 / Nagios Xi 5.7.5 (I had to abbreviate some things to make it fit in a forum post):

Code: Select all

Error: Could not expand hostgroups and/or hosts specified in host escalation (config file '.../hostescalations.cfg' line 49)
Error: Service escalation host name or description is NULL
Error: Could not register service escalation (config file '.../serviceescalations.cfg' line 425)
... Moderator's Note: Edited to protect customer data.

[optionstechnology]

No errors in the output when 'Verifying' the file, although the output seems to be truncated at the end (see attached)

[dchurch]

Has your team updated the Nagios XI configs recently? Was there a time when the config validated? In general, it's recommended to make tiny changes to ensure nothing breaks between applying the configs.

I found some things you can fix in your configuration to make the current configuration give fewer errors, but it's not a good idea to post hostnames on the public forum. If you escalate this to a ticket I can better communicate how to fix your config. To open a support ticket: https://support.nagios.com/tickets/