Page 1 of 4
Table './nagios/nagios_contactnotifications' is marked as cr
Posted: Fri Oct 21, 2016 6:09 am
by saptech
it is for more than 2 weeks that this error appears in the nagios.log
Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed SQL: SQL Error [ndoutils] : Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed
we used the following document
https://assets.nagios.com/downloads/nag ... tabase.pdf
to clenaup the tables but after 1 or 2 days the errors occur again
any advice?
Re: Table './nagios/nagios_contactnotifications' is marked a
Posted: Fri Oct 21, 2016 9:30 am
by rkennedy
I would repair once again, does it clear up the issue? Generally just running /usr/local/nagiosxi/scripts/repair_databases.sh will fix the issue.
Between the initial repair and now, was anything done to the machine? Usually I see crashed tables occur from a sudden shutdown or a service ending abruptly.
Re: Table './nagios/nagios_contactnotifications' is marked a
Posted: Mon Oct 24, 2016 3:09 am
by saptech
it is now about 2 weeks that we repeat the repair every couple of days....
I don't experience crashes or anything else
Re: Table './nagios/nagios_contactnotifications' is marked a
Posted: Mon Oct 24, 2016 11:44 am
by rkennedy
Could you PM over a profile for me to review? (Admin -> System Profile -> Download Profile)
This should have information related to the issue you're seeing.
Re: Table './nagios/nagios_contactnotifications' is marked a
Posted: Tue Oct 25, 2016 5:32 am
by saptech
here attached
EDIT: Removed attachment and saved internally.
Re: Table './nagios/nagios_contactnotifications' is marked a
Posted: Tue Oct 25, 2016 4:04 pm
by tgriep
You may want to run the following command to do an extended repair and to repair missing or corrupt headers in the tables for all of the databases on the server.
Code: Select all
mysqlcheck -f -r -u root -pnagiosxi --all-databases --use_frm
Try that and post if it works or not.
One thing I see is that the nagiosramdisk is almost full and the size should be increased.
To do that, login as root to the XI server and do the following
Code: Select all
service nagios stop
service npcd stop
umount /var/nagiosramdisk
Then edit the /etc/sysconfig/nagios file and change the following line from
to
Save the file and run the following
Code: Select all
service nagios start
service npcd start
That will increase it to 200 Meg.
Re: Table './nagios/nagios_contactnotifications' is marked a
Posted: Tue Nov 01, 2016 4:22 am
by saptech
same error:
SQL: SQL Error [ndoutils] : Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed SQL: SQL Error [ndoutils] : Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed
Here output of the repair command:
Code: Select all
mysql.columns_priv OK
mysql.db
warning : Number of rows changed from 0 to 4
status : OK
mysql.event OK
mysql.func OK
mysql.general_log OK
mysql.help_category
warning : Number of rows changed from 0 to 40
status : OK
mysql.help_keyword
warning : Number of rows changed from 0 to 453
status : OK
mysql.help_relation
warning : Number of rows changed from 0 to 1009
status : OK
mysql.help_topic
warning : Number of rows changed from 0 to 510
status : OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.servers OK
mysql.slow_log OK
mysql.tables_priv OK
mysql.time_zone OK
mysql.time_zone_leap_second OK
mysql.time_zone_name OK
mysql.time_zone_transition OK
mysql.time_zone_transition_type OK
mysql.user
warning : Number of rows changed from 0 to 7
status : OK
nagios.nagios_acknowledgements
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_commands
warning : Number of rows changed from 0 to 119
status : OK
nagios.nagios_commenthistory
warning : Number of rows changed from 0 to 12
status : OK
nagios.nagios_comments
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_configfiles
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_configfilevariables
warning : Number of rows changed from 0 to 131
status : OK
nagios.nagios_conninfo
warning : Number of rows changed from 0 to 290
status : OK
nagios.nagios_contact_addresses OK
nagios.nagios_contact_notificationcommands
warning : Number of rows changed from 0 to 96
status : OK
nagios.nagios_contactgroup_members
warning : Number of rows changed from 0 to 13
status : OK
nagios.nagios_contactgroups
warning : Number of rows changed from 0 to 3
status : OK
nagios.nagios_contactnotificationmethods
info : Delete link points outside datafile at 20168
warning : Number of rows changed from 0 to 2276
status : OK
nagios.nagios_contactnotifications
warning : Number of rows changed from 0 to 160
status : OK
nagios.nagios_contacts
warning : Number of rows changed from 0 to 8
status : OK
nagios.nagios_contactstatus
warning : Number of rows changed from 0 to 8
status : OK
nagios.nagios_customvariables
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_customvariablestatus
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_dbversion
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_downtimehistory OK
nagios.nagios_eventhandlers
info : Delete link points outside datafile at 0
status : OK
nagios.nagios_externalcommands
info : Delete link points outside datafile at 2729812
info : Delete link points outside datafile at 4856848
info : Delete link points outside datafile at 4860132
warning : Number of rows changed from 0 to 32273
status : OK
nagios.nagios_flappinghistory
warning : Number of rows changed from 0 to 32
status : OK
nagios.nagios_host_contactgroups
warning : Number of rows changed from 0 to 22
status : OK
nagios.nagios_host_contacts
warning : Number of rows changed from 0 to 2
status : OK
nagios.nagios_host_parenthosts OK
nagios.nagios_hostchecks OK
nagios.nagios_hostdependencies OK
nagios.nagios_hostescalation_contactgroups OK
nagios.nagios_hostescalation_contacts OK
nagios.nagios_hostescalations OK
nagios.nagios_hostgroup_members
warning : Number of rows changed from 0 to 17
status : OK
nagios.nagios_hostgroups
warning : Number of rows changed from 0 to 3
status : OK
nagios.nagios_hosts
warning : Number of rows changed from 0 to 22
status : OK
nagios.nagios_hoststatus
info : Delete link points outside datafile at 6364
warning : Number of rows changed from 0 to 22
status : OK
nagios.nagios_instances
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_logentries
info : Delete link points outside datafile at 979168
info : Delete link points outside datafile at 10282056
info : Delete link points outside datafile at 10284580
warning : Number of rows changed from 0 to 147158
status : OK
nagios.nagios_notifications
warning : Number of rows changed from 0 to 2485
status : OK
nagios.nagios_objects
warning : Number of rows changed from 0 to 572
status : OK
nagios.nagios_processevents
warning : Number of rows changed from 0 to 504
status : OK
nagios.nagios_programstatus
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_runtimevariables
warning : Number of rows changed from 0 to 18
status : OK
nagios.nagios_scheduleddowntime OK
nagios.nagios_service_contactgroups
warning : Number of rows changed from 0 to 337
status : OK
nagios.nagios_service_contacts OK
nagios.nagios_service_parentservices OK
nagios.nagios_servicechecks OK
nagios.nagios_servicedependencies OK
nagios.nagios_serviceescalation_contactgroups OK
nagios.nagios_serviceescalation_contacts OK
nagios.nagios_serviceescalations OK
nagios.nagios_servicegroup_members OK
nagios.nagios_servicegroups OK
nagios.nagios_services
warning : Number of rows changed from 0 to 337
status : OK
nagios.nagios_servicestatus
info : Delete link points outside datafile at 92256
info : Delete link points outside datafile at 108416
info : Found block that points outside data file at 108444
info : Found block that points outside data file at 108484
info : Found block that points outside data file at 108544
info : Found block that points outside data file at 108668
warning : Number of rows changed from 0 to 337
status : OK
nagios.nagios_statehistory
warning : Number of rows changed from 0 to 50712
status : OK
nagios.nagios_systemcommands
info : Delete link points outside datafile at 5844
info : Found block with too small length at 5976; Skipped
info : Delete link points outside datafile at 13648
info : Found block with too small length at 15732; Skipped
info : Found block with too small length at 16092; Skipped
info : Found block with too small length at 16852; Skipped
info : Found block with too small length at 18716; Skipped
info : Found block with too small length at 20972; Skipped
info : Found block with too small length at 21744; Skipped
info : Delete link points outside datafile at 24152
info : Found block with too small length at 25044; Skipped
info : Delete link points outside datafile at 26008
warning : Number of rows changed from 0 to 96
status : OK
nagios.nagios_timedeventqueue OK
nagios.nagios_timedevents OK
nagios.nagios_timeperiod_timeranges
warning : Number of rows changed from 0 to 31
status : OK
nagios.nagios_timeperiods
warning : Number of rows changed from 0 to 8
status : OK
nagiosql.tbl_command
warning : Number of rows changed from 0 to 125
status : OK
nagiosql.tbl_contact
warning : Number of rows changed from 0 to 7
status : OK
nagiosql.tbl_contactgroup
warning : Number of rows changed from 0 to 3
status : OK
nagiosql.tbl_contacttemplate
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_domain
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_host
warning : Number of rows changed from 0 to 22
status : OK
nagiosql.tbl_hostdependency OK
nagiosql.tbl_hostescalation OK
nagiosql.tbl_hostextinfo OK
nagiosql.tbl_hostgroup
warning : Number of rows changed from 0 to 4
status : OK
nagiosql.tbl_hosttemplate
warning : Number of rows changed from 0 to 49
status : OK
nagiosql.tbl_info
warning : Number of rows changed from 0 to 218
status : OK
nagiosql.tbl_lnkContactToCommandHost
warning : Number of rows changed from 0 to 7
status : OK
nagiosql.tbl_lnkContactToCommandService
warning : Number of rows changed from 0 to 7
status : OK
nagiosql.tbl_lnkContactToContactgroup
warning : Number of rows changed from 0 to 5
status : OK
nagiosql.tbl_lnkContactToContacttemplate
warning : Number of rows changed from 0 to 6
status : OK
nagiosql.tbl_lnkContactToVariabledefinition OK
nagiosql.tbl_lnkContactgroupToContact
warning : Number of rows changed from 0 to 11
status : OK
nagiosql.tbl_lnkContactgroupToContactgroup OK
nagiosql.tbl_lnkContacttemplateToCommandHost
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_lnkContacttemplateToCommandService
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_lnkContacttemplateToContactgroup
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkContacttemplateToContacttemplate OK
nagiosql.tbl_lnkContacttemplateToVariabledefinition OK
nagiosql.tbl_lnkHostToContact
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_lnkHostToContactgroup
warning : Number of rows changed from 0 to 21
status : OK
nagiosql.tbl_lnkHostToHost OK
nagiosql.tbl_lnkHostToHostgroup
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkHostToHosttemplate
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_lnkHostToVariabledefinition
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkHostdependencyToHost_DH OK
nagiosql.tbl_lnkHostdependencyToHost_H OK
nagiosql.tbl_lnkHostdependencyToHostgroup_DH OK
nagiosql.tbl_lnkHostdependencyToHostgroup_H OK
nagiosql.tbl_lnkHostescalationToContact OK
nagiosql.tbl_lnkHostescalationToContactgroup OK
nagiosql.tbl_lnkHostescalationToHost OK
nagiosql.tbl_lnkHostescalationToHostgroup OK
nagiosql.tbl_lnkHostgroupToHost
warning : Number of rows changed from 0 to 16
status : OK
nagiosql.tbl_lnkHostgroupToHostgroup OK
nagiosql.tbl_lnkHosttemplateToContact OK
nagiosql.tbl_lnkHosttemplateToContactgroup
warning : Number of rows changed from 0 to 4
status : OK
nagiosql.tbl_lnkHosttemplateToHost OK
nagiosql.tbl_lnkHosttemplateToHostgroup
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkHosttemplateToHosttemplate
warning : Number of rows changed from 0 to 47
status : OK
nagiosql.tbl_lnkHosttemplateToVariabledefinition OK
nagiosql.tbl_lnkServiceToContact OK
nagiosql.tbl_lnkServiceToContactgroup
warning : Number of rows changed from 0 to 93
status : OK
nagiosql.tbl_lnkServiceToHost
warning : Number of rows changed from 0 to 339
status : OK
nagiosql.tbl_lnkServiceToHostgroup
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkServiceToServicegroup OK
nagiosql.tbl_lnkServiceToServicetemplate
warning : Number of rows changed from 0 to 8
status : OK
nagiosql.tbl_lnkServiceToVariabledefinition OK
nagiosql.tbl_lnkServicedependencyToHost_DH OK
nagiosql.tbl_lnkServicedependencyToHost_H OK
nagiosql.tbl_lnkServicedependencyToHostgroup_DH OK
nagiosql.tbl_lnkServicedependencyToHostgroup_H OK
nagiosql.tbl_lnkServicedependencyToService_DS OK
nagiosql.tbl_lnkServicedependencyToService_S OK
nagiosql.tbl_lnkServiceescalationToContact OK
nagiosql.tbl_lnkServiceescalationToContactgroup OK
nagiosql.tbl_lnkServiceescalationToHost OK
nagiosql.tbl_lnkServiceescalationToHostgroup OK
nagiosql.tbl_lnkServiceescalationToService OK
nagiosql.tbl_lnkServicegroupToService OK
nagiosql.tbl_lnkServicegroupToServicegroup OK
nagiosql.tbl_lnkServicetemplateToContact OK
nagiosql.tbl_lnkServicetemplateToContactgroup
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkServicetemplateToHost OK
nagiosql.tbl_lnkServicetemplateToHostgroup OK
nagiosql.tbl_lnkServicetemplateToServicegroup OK
nagiosql.tbl_lnkServicetemplateToServicetemplate
warning : Number of rows changed from 0 to 74
status : OK
nagiosql.tbl_lnkServicetemplateToVariabledefinition OK
nagiosql.tbl_lnkTimeperiodToTimeperiod OK
nagiosql.tbl_logbook OK
nagiosql.tbl_mainmenu
warning : Number of rows changed from 0 to 7
status : OK
nagiosql.tbl_service
warning : Number of rows changed from 0 to 102
status : OK
nagiosql.tbl_servicedependency OK
nagiosql.tbl_serviceescalation OK
nagiosql.tbl_serviceextinfo OK
nagiosql.tbl_servicegroup OK
nagiosql.tbl_servicetemplate
warning : Number of rows changed from 0 to 76
status : OK
nagiosql.tbl_settings
warning : Number of rows changed from 0 to 12
status : OK
nagiosql.tbl_submenu
warning : Number of rows changed from 0 to 29
status : OK
nagiosql.tbl_timedefinition
warning : Number of rows changed from 0 to 37
status : OK
nagiosql.tbl_timeperiod
warning : Number of rows changed from 0 to 7
status : OK
nagiosql.tbl_user
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_variabledefinition
warning : Number of rows changed from 0 to 1
status : OK
Re: Table './nagios/nagios_contactnotifications' is marked a
Posted: Tue Nov 01, 2016 9:30 am
by tgriep
It looks like the repair fixed the database tables. Are the messages gone now?
Re: Table './nagios/nagios_contactnotifications' is marked a
Posted: Wed Nov 02, 2016 7:51 am
by saptech
as per my previous message after the repair the message is still coming out:
SQL: SQL Error [ndoutils] : Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed SQL: SQL Error [ndoutils] : Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed
Re: Table './nagios/nagios_contactnotifications' is marked a
Posted: Wed Nov 02, 2016 9:24 am
by tgriep
Lets try shutting the daemons down while the repair is run and then start them up as data could of been written to that table while the repair was run.
Login as root to the XI server and run the following.
Code: Select all
service nagios stop
killall -9 nagios
service crond stop
service ndo2db stop
mysqlcheck -f -r -u root -pnagiosxi --all-databases --use_frm
service mysqld restart
service ndo2db start
service crond start
service nagios start
Try that and see if that fixes the table for you.