Table './nagios/nagios_contactnotifications' is marked as cr

[saptech]

it is for more than 2 weeks that this error appears in the nagios.log

Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed SQL: SQL Error [ndoutils] : Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed

we used the following document
https://assets.nagios.com/downloads/nag ... tabase.pdf
to clenaup the tables but after 1 or 2 days the errors occur again

any advice?

[rkennedy]

I would repair once again, does it clear up the issue? Generally just running /usr/local/nagiosxi/scripts/repair_databases.sh will fix the issue.

Between the initial repair and now, was anything done to the machine? Usually I see crashed tables occur from a sudden shutdown or a service ending abruptly.

[saptech]

it is now about 2 weeks that we repeat the repair every couple of days....
I don't experience crashes or anything else

[rkennedy]

Could you PM over a profile for me to review? (Admin -> System Profile -> Download Profile)

This should have information related to the issue you're seeing.

[saptech]

here attached

EDIT: Removed attachment and saved internally.

[tgriep]

You may want to run the following command to do an extended repair and to repair missing or corrupt headers in the tables for all of the databases on the server.

Code: Select all

mysqlcheck -f -r -u root -pnagiosxi --all-databases --use_frm

Try that and post if it works or not.

One thing I see is that the nagiosramdisk is almost full and the size should be increased.
To do that, login as root to the XI server and do the following

Code: Select all

service nagios stop
service npcd stop
umount /var/nagiosramdisk

Then edit the /etc/sysconfig/nagios file and change the following line from

Code: Select all

RAMDISK_SIZE=100

to

Code: Select all

RAMDISK_SIZE=200

Save the file and run the following

Code: Select all

service nagios start
service npcd start

That will increase it to 200 Meg.

[saptech]

same error:

SQL: SQL Error [ndoutils] : Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed SQL: SQL Error [ndoutils] : Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed

Here output of the repair command:

Code: Select all

mysql.columns_priv                                 OK
mysql.db
warning  : Number of rows changed from 0 to 4
status   : OK
mysql.event                                        OK
mysql.func                                         OK
mysql.general_log                                  OK
mysql.help_category
warning  : Number of rows changed from 0 to 40
status   : OK
mysql.help_keyword
warning  : Number of rows changed from 0 to 453
status   : OK
mysql.help_relation
warning  : Number of rows changed from 0 to 1009
status   : OK
mysql.help_topic
warning  : Number of rows changed from 0 to 510
status   : OK
mysql.host                                         OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.servers                                      OK
mysql.slow_log                                     OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user
warning  : Number of rows changed from 0 to 7
status   : OK
nagios.nagios_acknowledgements
warning  : Number of rows changed from 0 to 1
status   : OK
nagios.nagios_commands
warning  : Number of rows changed from 0 to 119
status   : OK
nagios.nagios_commenthistory
warning  : Number of rows changed from 0 to 12
status   : OK
nagios.nagios_comments
warning  : Number of rows changed from 0 to 1
status   : OK
nagios.nagios_configfiles
warning  : Number of rows changed from 0 to 1
status   : OK
nagios.nagios_configfilevariables
warning  : Number of rows changed from 0 to 131
status   : OK
nagios.nagios_conninfo
warning  : Number of rows changed from 0 to 290
status   : OK
nagios.nagios_contact_addresses                    OK
nagios.nagios_contact_notificationcommands
warning  : Number of rows changed from 0 to 96
status   : OK
nagios.nagios_contactgroup_members
warning  : Number of rows changed from 0 to 13
status   : OK
nagios.nagios_contactgroups
warning  : Number of rows changed from 0 to 3
status   : OK
nagios.nagios_contactnotificationmethods
info     : Delete link points outside datafile at 20168
warning  : Number of rows changed from 0 to 2276
status   : OK
nagios.nagios_contactnotifications
warning  : Number of rows changed from 0 to 160
status   : OK
nagios.nagios_contacts
warning  : Number of rows changed from 0 to 8
status   : OK
nagios.nagios_contactstatus
warning  : Number of rows changed from 0 to 8
status   : OK
nagios.nagios_customvariables
warning  : Number of rows changed from 0 to 1
status   : OK
nagios.nagios_customvariablestatus
warning  : Number of rows changed from 0 to 1
status   : OK
nagios.nagios_dbversion
warning  : Number of rows changed from 0 to 1
status   : OK
nagios.nagios_downtimehistory                      OK
nagios.nagios_eventhandlers
info     : Delete link points outside datafile at 0
status   : OK
nagios.nagios_externalcommands
info     : Delete link points outside datafile at 2729812
info     : Delete link points outside datafile at 4856848
info     : Delete link points outside datafile at 4860132
warning  : Number of rows changed from 0 to 32273
status   : OK
nagios.nagios_flappinghistory
warning  : Number of rows changed from 0 to 32
status   : OK
nagios.nagios_host_contactgroups
warning  : Number of rows changed from 0 to 22
status   : OK
nagios.nagios_host_contacts
warning  : Number of rows changed from 0 to 2
status   : OK
nagios.nagios_host_parenthosts                     OK
nagios.nagios_hostchecks                           OK
nagios.nagios_hostdependencies                     OK
nagios.nagios_hostescalation_contactgroups         OK
nagios.nagios_hostescalation_contacts              OK
nagios.nagios_hostescalations                      OK
nagios.nagios_hostgroup_members
warning  : Number of rows changed from 0 to 17
status   : OK
nagios.nagios_hostgroups
warning  : Number of rows changed from 0 to 3
status   : OK
nagios.nagios_hosts
warning  : Number of rows changed from 0 to 22
status   : OK
nagios.nagios_hoststatus
info     : Delete link points outside datafile at 6364
warning  : Number of rows changed from 0 to 22
status   : OK
nagios.nagios_instances
warning  : Number of rows changed from 0 to 1
status   : OK
nagios.nagios_logentries
info     : Delete link points outside datafile at 979168
info     : Delete link points outside datafile at 10282056
info     : Delete link points outside datafile at 10284580
warning  : Number of rows changed from 0 to 147158
status   : OK
nagios.nagios_notifications
warning  : Number of rows changed from 0 to 2485
status   : OK
nagios.nagios_objects
warning  : Number of rows changed from 0 to 572
status   : OK
nagios.nagios_processevents
warning  : Number of rows changed from 0 to 504
status   : OK
nagios.nagios_programstatus
warning  : Number of rows changed from 0 to 1
status   : OK
nagios.nagios_runtimevariables
warning  : Number of rows changed from 0 to 18
status   : OK
nagios.nagios_scheduleddowntime                    OK
nagios.nagios_service_contactgroups
warning  : Number of rows changed from 0 to 337
status   : OK
nagios.nagios_service_contacts                     OK
nagios.nagios_service_parentservices               OK
nagios.nagios_servicechecks                        OK
nagios.nagios_servicedependencies                  OK
nagios.nagios_serviceescalation_contactgroups      OK
nagios.nagios_serviceescalation_contacts           OK
nagios.nagios_serviceescalations                   OK
nagios.nagios_servicegroup_members                 OK
nagios.nagios_servicegroups                        OK
nagios.nagios_services
warning  : Number of rows changed from 0 to 337
status   : OK
nagios.nagios_servicestatus
info     : Delete link points outside datafile at 92256
info     : Delete link points outside datafile at 108416
info     : Found block that points outside data file at 108444
info     : Found block that points outside data file at 108484
info     : Found block that points outside data file at 108544
info     : Found block that points outside data file at 108668
warning  : Number of rows changed from 0 to 337
status   : OK
nagios.nagios_statehistory
warning  : Number of rows changed from 0 to 50712
status   : OK
nagios.nagios_systemcommands
info     : Delete link points outside datafile at 5844
info     : Found block with too small length at 5976; Skipped
info     : Delete link points outside datafile at 13648
info     : Found block with too small length at 15732; Skipped
info     : Found block with too small length at 16092; Skipped
info     : Found block with too small length at 16852; Skipped
info     : Found block with too small length at 18716; Skipped
info     : Found block with too small length at 20972; Skipped
info     : Found block with too small length at 21744; Skipped
info     : Delete link points outside datafile at 24152
info     : Found block with too small length at 25044; Skipped
info     : Delete link points outside datafile at 26008
warning  : Number of rows changed from 0 to 96
status   : OK
nagios.nagios_timedeventqueue                      OK
nagios.nagios_timedevents                          OK
nagios.nagios_timeperiod_timeranges
warning  : Number of rows changed from 0 to 31
status   : OK
nagios.nagios_timeperiods
warning  : Number of rows changed from 0 to 8
status   : OK
nagiosql.tbl_command
warning  : Number of rows changed from 0 to 125
status   : OK
nagiosql.tbl_contact
warning  : Number of rows changed from 0 to 7
status   : OK
nagiosql.tbl_contactgroup
warning  : Number of rows changed from 0 to 3
status   : OK
nagiosql.tbl_contacttemplate
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_domain
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_host
warning  : Number of rows changed from 0 to 22
status   : OK
nagiosql.tbl_hostdependency                        OK
nagiosql.tbl_hostescalation                        OK
nagiosql.tbl_hostextinfo                           OK
nagiosql.tbl_hostgroup
warning  : Number of rows changed from 0 to 4
status   : OK
nagiosql.tbl_hosttemplate
warning  : Number of rows changed from 0 to 49
status   : OK
nagiosql.tbl_info
warning  : Number of rows changed from 0 to 218
status   : OK
nagiosql.tbl_lnkContactToCommandHost
warning  : Number of rows changed from 0 to 7
status   : OK
nagiosql.tbl_lnkContactToCommandService
warning  : Number of rows changed from 0 to 7
status   : OK
nagiosql.tbl_lnkContactToContactgroup
warning  : Number of rows changed from 0 to 5
status   : OK
nagiosql.tbl_lnkContactToContacttemplate
warning  : Number of rows changed from 0 to 6
status   : OK
nagiosql.tbl_lnkContactToVariabledefinition        OK
nagiosql.tbl_lnkContactgroupToContact
warning  : Number of rows changed from 0 to 11
status   : OK
nagiosql.tbl_lnkContactgroupToContactgroup         OK
nagiosql.tbl_lnkContacttemplateToCommandHost
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_lnkContacttemplateToCommandService
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_lnkContacttemplateToContactgroup
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_lnkContacttemplateToContacttemplate   OK
nagiosql.tbl_lnkContacttemplateToVariabledefinition OK
nagiosql.tbl_lnkHostToContact
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_lnkHostToContactgroup
warning  : Number of rows changed from 0 to 21
status   : OK
nagiosql.tbl_lnkHostToHost                         OK
nagiosql.tbl_lnkHostToHostgroup
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_lnkHostToHosttemplate
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_lnkHostToVariabledefinition
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_lnkHostdependencyToHost_DH            OK
nagiosql.tbl_lnkHostdependencyToHost_H             OK
nagiosql.tbl_lnkHostdependencyToHostgroup_DH       OK
nagiosql.tbl_lnkHostdependencyToHostgroup_H        OK
nagiosql.tbl_lnkHostescalationToContact            OK
nagiosql.tbl_lnkHostescalationToContactgroup       OK
nagiosql.tbl_lnkHostescalationToHost               OK
nagiosql.tbl_lnkHostescalationToHostgroup          OK
nagiosql.tbl_lnkHostgroupToHost
warning  : Number of rows changed from 0 to 16
status   : OK
nagiosql.tbl_lnkHostgroupToHostgroup               OK
nagiosql.tbl_lnkHosttemplateToContact              OK
nagiosql.tbl_lnkHosttemplateToContactgroup
warning  : Number of rows changed from 0 to 4
status   : OK
nagiosql.tbl_lnkHosttemplateToHost                 OK
nagiosql.tbl_lnkHosttemplateToHostgroup
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_lnkHosttemplateToHosttemplate
warning  : Number of rows changed from 0 to 47
status   : OK
nagiosql.tbl_lnkHosttemplateToVariabledefinition   OK
nagiosql.tbl_lnkServiceToContact                   OK
nagiosql.tbl_lnkServiceToContactgroup
warning  : Number of rows changed from 0 to 93
status   : OK
nagiosql.tbl_lnkServiceToHost
warning  : Number of rows changed from 0 to 339
status   : OK
nagiosql.tbl_lnkServiceToHostgroup
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_lnkServiceToServicegroup              OK
nagiosql.tbl_lnkServiceToServicetemplate
warning  : Number of rows changed from 0 to 8
status   : OK
nagiosql.tbl_lnkServiceToVariabledefinition        OK
nagiosql.tbl_lnkServicedependencyToHost_DH         OK
nagiosql.tbl_lnkServicedependencyToHost_H          OK
nagiosql.tbl_lnkServicedependencyToHostgroup_DH    OK
nagiosql.tbl_lnkServicedependencyToHostgroup_H     OK
nagiosql.tbl_lnkServicedependencyToService_DS      OK
nagiosql.tbl_lnkServicedependencyToService_S       OK
nagiosql.tbl_lnkServiceescalationToContact         OK
nagiosql.tbl_lnkServiceescalationToContactgroup    OK
nagiosql.tbl_lnkServiceescalationToHost            OK
nagiosql.tbl_lnkServiceescalationToHostgroup       OK
nagiosql.tbl_lnkServiceescalationToService         OK
nagiosql.tbl_lnkServicegroupToService              OK
nagiosql.tbl_lnkServicegroupToServicegroup         OK
nagiosql.tbl_lnkServicetemplateToContact           OK
nagiosql.tbl_lnkServicetemplateToContactgroup
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_lnkServicetemplateToHost              OK
nagiosql.tbl_lnkServicetemplateToHostgroup         OK
nagiosql.tbl_lnkServicetemplateToServicegroup      OK
nagiosql.tbl_lnkServicetemplateToServicetemplate
warning  : Number of rows changed from 0 to 74
status   : OK
nagiosql.tbl_lnkServicetemplateToVariabledefinition OK
nagiosql.tbl_lnkTimeperiodToTimeperiod             OK
nagiosql.tbl_logbook                               OK
nagiosql.tbl_mainmenu
warning  : Number of rows changed from 0 to 7
status   : OK
nagiosql.tbl_service
warning  : Number of rows changed from 0 to 102
status   : OK
nagiosql.tbl_servicedependency                     OK
nagiosql.tbl_serviceescalation                     OK
nagiosql.tbl_serviceextinfo                        OK
nagiosql.tbl_servicegroup                          OK
nagiosql.tbl_servicetemplate
warning  : Number of rows changed from 0 to 76
status   : OK
nagiosql.tbl_settings
warning  : Number of rows changed from 0 to 12
status   : OK
nagiosql.tbl_submenu
warning  : Number of rows changed from 0 to 29
status   : OK
nagiosql.tbl_timedefinition
warning  : Number of rows changed from 0 to 37
status   : OK
nagiosql.tbl_timeperiod
warning  : Number of rows changed from 0 to 7
status   : OK
nagiosql.tbl_user
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_variabledefinition
warning  : Number of rows changed from 0 to 1
status   : OK

[tgriep]

It looks like the repair fixed the database tables. Are the messages gone now?

[saptech]

as per my previous message after the repair the message is still coming out:

SQL: SQL Error [ndoutils] : Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed SQL: SQL Error [ndoutils] : Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed

[tgriep]

Lets try shutting the daemons down while the repair is run and then start them up as data could of been written to that table while the repair was run.
Login as root to the XI server and run the following.

Code: Select all

service nagios stop
killall -9 nagios
service crond stop
service ndo2db stop
mysqlcheck -f -r -u root -pnagiosxi --all-databases --use_frm
service mysqld restart
service ndo2db start
service crond start
service nagios start

Try that and see if that fixes the table for you.