A database connection error...

GldRush98 · Post by **GldRush98** » Mon Oct 05, 2015 8:36 am

Not how I was wanting to start my Monday morning

Web page gives the:

Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh
Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh
SQL: SQL Error [nagiosxi] : Database connection failed SQL: SQL Error [nagiosxi] : Database connection failed SQL: SQL Error [nagiosxi] : Database connection failedMessage: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh

So running the repair script, everything is looking fine to me:

Code: Select all

[root@nagios scripts]# ./repair_databases.sh
mysqld (pid 6428) is running...
mysqld (pid 6428) is running...
DATABASE: nagios
TABLE:
/var/lib/mysql/nagios /usr/local/nagiosxi/scripts
Stopping MySQL:                                            [  OK  ]
- recovering (with sort) MyISAM-table 'nagios_acknowledgements.MYI'
Data records: 1004
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_commands.MYI'
Data records: 133
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_commenthistory.MYI'
Data records: 1804
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_comments.MYI'
Data records: 9
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_configfiles.MYI'
Data records: 1
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_configfilevariables.MYI'
Data records: 131
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_conninfo.MYI'
Data records: 968
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_contact_addresses.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contactgroup_members.MYI'
Data records: 25
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contactgroups.MYI'
Data records: 5
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contact_notificationcommands.MYI'
Data records: 120
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contactnotificationmethods.MYI'
Data records: 13371
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4

---------

- recovering (with sort) MyISAM-table 'nagios_contactnotifications.MYI'
Data records: 13371
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5

---------

- recovering (with sort) MyISAM-table 'nagios_contacts.MYI'
Data records: 10
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contactstatus.MYI'
Data records: 10
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_customvariables.MYI'
Data records: 746
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_customvariablestatus.MYI'
Data records: 746
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with keycache) MyISAM-table 'nagios_dbversion.MYI'
Data records: 1

---------

- recovering (with sort) MyISAM-table 'nagios_downtimehistory.MYI'
Data records: 343
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_eventhandlers.MYI'
Data records: 263
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_externalcommands.MYI'
Data records: 28
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_flappinghistory.MYI'
Data records: 4862
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_hostchecks.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_host_contactgroups.MYI'
Data records: 89
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_host_contacts.MYI'
Data records: 2
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostdependencies.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostescalation_contactgroups.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostescalation_contacts.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostescalations.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostgroup_members.MYI'
Data records: 86
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostgroups.MYI'
Data records: 5
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_host_parenthosts.MYI'
Data records: 38
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hosts.MYI'
Data records: 87
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_hoststatus.MYI'
Data records: 87
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6
- Fixing index 7
- Fixing index 8
- Fixing index 9
- Fixing index 10
- Fixing index 11
- Fixing index 12
- Fixing index 13
- Fixing index 14
- Fixing index 15
- Fixing index 16
- Fixing index 17
- Fixing index 18
- Fixing index 19

---------

- recovering (with sort) MyISAM-table 'nagios_instances.MYI'
Data records: 1
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_logentries.MYI'
Data records: 116439
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5

---------

- recovering (with sort) MyISAM-table 'nagios_notifications.MYI'
Data records: 2724
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5

---------

- recovering (with sort) MyISAM-table 'nagios_objects.MYI'
Data records: 1473
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5

---------

- recovering (with sort) MyISAM-table 'nagios_processevents.MYI'
Data records: 1380
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_programstatus.MYI'
Data records: 1
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_runtimevariables.MYI'
Data records: 18
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_scheduleddowntime.MYI'
Data records: 4
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_servicechecks.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4

---------

- recovering (with sort) MyISAM-table 'nagios_service_contactgroups.MYI'
Data records: 691
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_service_contacts.MYI'
Data records: 13
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_servicedependencies.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalation_contactgroups.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalation_contacts.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalations.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_servicegroup_members.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_servicegroups.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_service_parentservices.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_services.MYI'
Data records: 676
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_servicestatus.MYI'
Data records: 676
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6
- Fixing index 7
- Fixing index 8
- Fixing index 9
- Fixing index 10
- Fixing index 11
- Fixing index 12
- Fixing index 13
- Fixing index 14
- Fixing index 15
- Fixing index 16
- Fixing index 17
- Fixing index 18
- Fixing index 19

---------

- recovering (with sort) MyISAM-table 'nagios_statehistory.MYI'
Data records: 258398
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_systemcommands.MYI'
Data records: 46356
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_timedeventqueue.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6

---------

- recovering (with sort) MyISAM-table 'nagios_timedevents.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6

---------

- recovering (with sort) MyISAM-table 'nagios_timeperiods.MYI'
Data records: 18
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_timeperiod_timeranges.MYI'
Data records: 96
- Fixing index 1
- Fixing index 2
Starting MySQL:                                            [  OK  ]
/usr/local/nagiosxi/scripts

===============
REPAIR COMPLETE
===============
mysqld (pid 6878) is running...
DATABASE: nagiosql
TABLE:
/var/lib/mysql/nagiosql /usr/local/nagiosxi/scripts
Stopping MySQL:                                            [  OK  ]
- recovering (with sort) MyISAM-table 'tbl_command.MYI'
Data records: 137
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_contactgroup.MYI'
Data records: 5
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_contact.MYI'
Data records: 9
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_contacttemplate.MYI'
Data records: 2
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_domain.MYI'
Data records: 1
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_hostdependency.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_hostescalation.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_hostextinfo.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_hostgroup.MYI'
Data records: 6
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_host.MYI'
Data records: 88
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_hosttemplate.MYI'
Data records: 48
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_info.MYI'
Data records: 218
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactgroupToContactgroup.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactgroupToContact.MYI'
Data records: 16
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkContacttemplateToCommandHost.MYI'
Data records: 2
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkContacttemplateToCommandService.MYI'
Data records: 2
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkContacttemplateToContactgroup.MYI'
Data records: 1
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkContacttemplateToContacttemplate.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkContacttemplateToVariabledefinition.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactToCommandHost.MYI'
Data records: 1
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactToCommandService.MYI'
Data records: 1
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactToContactgroup.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactToContacttemplate.MYI'
Data records: 9
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactToVariabledefinition.MYI'
Data records: 16
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostdependencyToHost_DH.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostdependencyToHostgroup_DH.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostdependencyToHostgroup_H.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostdependencyToHost_H.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostescalationToContactgroup.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostescalationToContact.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostescalationToHostgroup.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostescalationToHost.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostgroupToHostgroup.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostgroupToHost.MYI'
Data records: 1
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHosttemplateToContactgroup.MYI'
Data records: 4
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHosttemplateToContact.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHosttemplateToHostgroup.MYI'
Data records: 1
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHosttemplateToHost.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHosttemplateToHosttemplate.MYI'
Data records: 46
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHosttemplateToVariabledefinition.MYI'
Data records: 3
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostToContactgroup.MYI'
Data records: 89
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostToContact.MYI'
Data records: 2
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostToHostgroup.MYI'
Data records: 86
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostToHost.MYI'
Data records: 38
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostToHosttemplate.MYI'
Data records: 87
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostToVariabledefinition.MYI'
Data records: 86
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicedependencyToHost_DH.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicedependencyToHostgroup_DH.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicedependencyToHostgroup_H.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicedependencyToHost_H.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicedependencyToService_DS.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicedependencyToService_S.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceescalationToContactgroup.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceescalationToContact.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceescalationToHostgroup.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceescalationToHost.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceescalationToService.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicegroupToServicegroup.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicegroupToService.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToContactgroup.MYI'
Data records: 1
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToContact.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToHostgroup.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToHost.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToServicegroup.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToServicetemplate.MYI'
Data records: 73
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToVariabledefinition.MYI'
Data records: 2
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToContactgroup.MYI'
Data records: 695
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToContact.MYI'
Data records: 12
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToHostgroup.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToHost.MYI'
Data records: 690
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToServicegroup.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToServicetemplate.MYI'
Data records: 683
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToVariabledefinition.MYI'
Data records: 674
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_lnkTimeperiodToTimeperiod.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_logbook.MYI'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_mainmenu.MYI'
Data records: 7
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_servicedependency.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_serviceescalation.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_serviceextinfo.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_servicegroup.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_service.MYI'
Data records: 690
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_servicetemplate.MYI'
Data records: 75
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_settings.MYI'
Data records: 12
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_submenu.MYI'
Data records: 29
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_timedefinition.MYI'
Data records: 102
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'tbl_timeperiod.MYI'
Data records: 17
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_user.MYI'
Data records: 4
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'tbl_variabledefinition.MYI'
Data records: 922
- Fixing index 1
Starting MySQL:                                            [  OK  ]
/usr/local/nagiosxi/scripts

===============
REPAIR COMPLETE
===============
Stopping ndo2db: done.
Starting ndo2db: done.
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
[root@nagios scripts]#

Manually running mysqlcheck is showing everything is ok

Code: Select all

[root@nagios scripts]# mysqlcheck --repair --all-databases -u root -p
Enter password:
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.func                                         OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.host                                         OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
nagios.nagios_acknowledgements                     OK
nagios.nagios_commands                             OK
nagios.nagios_commenthistory                       OK
nagios.nagios_comments                             OK
nagios.nagios_configfiles                          OK
nagios.nagios_configfilevariables                  OK
nagios.nagios_conninfo                             OK
nagios.nagios_contact_addresses                    OK
nagios.nagios_contact_notificationcommands         OK
nagios.nagios_contactgroup_members                 OK
nagios.nagios_contactgroups                        OK
nagios.nagios_contactnotificationmethods           OK
nagios.nagios_contactnotifications                 OK
nagios.nagios_contacts                             OK
nagios.nagios_contactstatus                        OK
nagios.nagios_customvariables                      OK
nagios.nagios_customvariablestatus                 OK
nagios.nagios_dbversion                            OK
nagios.nagios_downtimehistory                      OK
nagios.nagios_eventhandlers                        OK
nagios.nagios_externalcommands                     OK
nagios.nagios_flappinghistory                      OK
nagios.nagios_host_contactgroups                   OK
nagios.nagios_host_contacts                        OK
nagios.nagios_host_parenthosts                     OK
nagios.nagios_hostchecks                           OK
nagios.nagios_hostdependencies                     OK
nagios.nagios_hostescalation_contactgroups         OK
nagios.nagios_hostescalation_contacts              OK
nagios.nagios_hostescalations                      OK
nagios.nagios_hostgroup_members                    OK
nagios.nagios_hostgroups                           OK
nagios.nagios_hosts                                OK
nagios.nagios_hoststatus                           OK
nagios.nagios_instances                            OK
nagios.nagios_logentries                           OK
nagios.nagios_notifications                        OK
nagios.nagios_objects                              OK
nagios.nagios_processevents                        OK
nagios.nagios_programstatus                        OK
nagios.nagios_runtimevariables                     OK
nagios.nagios_scheduleddowntime                    OK
nagios.nagios_service_contactgroups                OK
nagios.nagios_service_contacts                     OK
nagios.nagios_service_parentservices               OK
nagios.nagios_servicechecks                        OK
nagios.nagios_servicedependencies                  OK
nagios.nagios_serviceescalation_contactgroups      OK
nagios.nagios_serviceescalation_contacts           OK
nagios.nagios_serviceescalations                   OK
nagios.nagios_servicegroup_members                 OK
nagios.nagios_servicegroups                        OK
nagios.nagios_services                             OK
nagios.nagios_servicestatus                        OK
nagios.nagios_statehistory                         OK
nagios.nagios_systemcommands                       OK
nagios.nagios_timedeventqueue                      OK
nagios.nagios_timedevents                          OK
nagios.nagios_timeperiod_timeranges                OK
nagios.nagios_timeperiods                          OK
nagiosql.tbl_command                               OK
nagiosql.tbl_contact                               OK
nagiosql.tbl_contactgroup                          OK
nagiosql.tbl_contacttemplate                       OK
nagiosql.tbl_domain                                OK
nagiosql.tbl_host                                  OK
nagiosql.tbl_hostdependency                        OK
nagiosql.tbl_hostescalation                        OK
nagiosql.tbl_hostextinfo                           OK
nagiosql.tbl_hostgroup                             OK
nagiosql.tbl_hosttemplate                          OK
nagiosql.tbl_info                                  OK
nagiosql.tbl_lnkContactToCommandHost               OK
nagiosql.tbl_lnkContactToCommandService            OK
nagiosql.tbl_lnkContactToContactgroup              OK
nagiosql.tbl_lnkContactToContacttemplate           OK
nagiosql.tbl_lnkContactToVariabledefinition        OK
nagiosql.tbl_lnkContactgroupToContact              OK
nagiosql.tbl_lnkContactgroupToContactgroup         OK
nagiosql.tbl_lnkContacttemplateToCommandHost       OK
nagiosql.tbl_lnkContacttemplateToCommandService    OK
nagiosql.tbl_lnkContacttemplateToContactgroup      OK
nagiosql.tbl_lnkContacttemplateToContacttemplate   OK
nagiosql.tbl_lnkContacttemplateToVariabledefinition OK
nagiosql.tbl_lnkHostToContact                      OK
nagiosql.tbl_lnkHostToContactgroup                 OK
nagiosql.tbl_lnkHostToHost                         OK
nagiosql.tbl_lnkHostToHostgroup                    OK
nagiosql.tbl_lnkHostToHosttemplate                 OK
nagiosql.tbl_lnkHostToVariabledefinition           OK
nagiosql.tbl_lnkHostdependencyToHost_DH            OK
nagiosql.tbl_lnkHostdependencyToHost_H             OK
nagiosql.tbl_lnkHostdependencyToHostgroup_DH       OK
nagiosql.tbl_lnkHostdependencyToHostgroup_H        OK
nagiosql.tbl_lnkHostescalationToContact            OK
nagiosql.tbl_lnkHostescalationToContactgroup       OK
nagiosql.tbl_lnkHostescalationToHost               OK
nagiosql.tbl_lnkHostescalationToHostgroup          OK
nagiosql.tbl_lnkHostgroupToHost                    OK
nagiosql.tbl_lnkHostgroupToHostgroup               OK
nagiosql.tbl_lnkHosttemplateToContact              OK
nagiosql.tbl_lnkHosttemplateToContactgroup         OK
nagiosql.tbl_lnkHosttemplateToHost                 OK
nagiosql.tbl_lnkHosttemplateToHostgroup            OK
nagiosql.tbl_lnkHosttemplateToHosttemplate         OK
nagiosql.tbl_lnkHosttemplateToVariabledefinition   OK
nagiosql.tbl_lnkServiceToContact                   OK
nagiosql.tbl_lnkServiceToContactgroup              OK
nagiosql.tbl_lnkServiceToHost                      OK
nagiosql.tbl_lnkServiceToHostgroup                 OK
nagiosql.tbl_lnkServiceToServicegroup              OK
nagiosql.tbl_lnkServiceToServicetemplate           OK
nagiosql.tbl_lnkServiceToVariabledefinition        OK
nagiosql.tbl_lnkServicedependencyToHost_DH         OK
nagiosql.tbl_lnkServicedependencyToHost_H          OK
nagiosql.tbl_lnkServicedependencyToHostgroup_DH    OK
nagiosql.tbl_lnkServicedependencyToHostgroup_H     OK
nagiosql.tbl_lnkServicedependencyToService_DS      OK
nagiosql.tbl_lnkServicedependencyToService_S       OK
nagiosql.tbl_lnkServiceescalationToContact         OK
nagiosql.tbl_lnkServiceescalationToContactgroup    OK
nagiosql.tbl_lnkServiceescalationToHost            OK
nagiosql.tbl_lnkServiceescalationToHostgroup       OK
nagiosql.tbl_lnkServiceescalationToService         OK
nagiosql.tbl_lnkServicegroupToService              OK
nagiosql.tbl_lnkServicegroupToServicegroup         OK
nagiosql.tbl_lnkServicetemplateToContact           OK
nagiosql.tbl_lnkServicetemplateToContactgroup      OK
nagiosql.tbl_lnkServicetemplateToHost              OK
nagiosql.tbl_lnkServicetemplateToHostgroup         OK
nagiosql.tbl_lnkServicetemplateToServicegroup      OK
nagiosql.tbl_lnkServicetemplateToServicetemplate   OK
nagiosql.tbl_lnkServicetemplateToVariabledefinition OK
nagiosql.tbl_lnkTimeperiodToTimeperiod             OK
nagiosql.tbl_logbook                               OK
nagiosql.tbl_mainmenu                              OK
nagiosql.tbl_service                               OK
nagiosql.tbl_servicedependency                     OK
nagiosql.tbl_serviceescalation                     OK
nagiosql.tbl_serviceextinfo                        OK
nagiosql.tbl_servicegroup                          OK
nagiosql.tbl_servicetemplate                       OK
nagiosql.tbl_settings                              OK
nagiosql.tbl_submenu                               OK
nagiosql.tbl_timedefinition                        OK
nagiosql.tbl_timeperiod                            OK
nagiosql.tbl_user                                  OK
nagiosql.tbl_variabledefinition                    OK

The only recent change to the machine was replacing its expiring SSL certs... that happened over a week ago though and I don't see how that could be related to this.

I'm still working on it but figured I would pop this in here for some assistance as I'm running low on ideas.

hsmith · Post by **hsmith** » Mon Oct 05, 2015 9:15 am

How are you doing on disk space?

Code: Select all

df -h
df -ih

GldRush98 · Post by **GldRush98** » Mon Oct 05, 2015 9:16 am

Yeah, first thing I checked actually

Code: Select all

[root@nagios scripts]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       74G  8.9G   61G  13% /
/dev/xvda1             99M   22M   72M  24% /boot
tmpfs                 1.0G     0  1.0G   0% /dev/shm
tmpfs                  50M  1.8M   49M   4% /var/nagiosramdisk
[root@nagios scripts]# df -ih
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/VolGroup00-LogVol00
                         19M    112K     19M    1% /
/dev/xvda1               26K      45     26K    1% /boot
tmpfs                   256K       1    256K    1% /dev/shm
tmpfs                   256K       6    256K    1% /var/nagiosramdisk

GldRush98 · Post by **GldRush98** » Mon Oct 05, 2015 9:40 am

Could there be some kind of table/database size limit config I'm hitting in either MySQL or Postgres?

GldRush98 · Post by **GldRush98** » Mon Oct 05, 2015 9:49 am

Checking our failover monitor box, this seems to be the first error we got:

State: CRITICAL
Info:
Error: Could not parse XML from https://[redacted]:443/nagiosxi ( SQL: SQL Error [nagiosxi] :/b ERROR: database is not accepting commands to avoid wraparound data loss in database postgres
Date/Time: 10/03/2015 14:10:07

Post by **tgriep** » Mon Oct 05, 2015 9:57 am

The error on your failover system looks like the postgres database needs to be VACUUMed. Run the following in a shell to see if that resolves the errors.

Code: Select all

service postgresql stop
su postgres
echo "VACUUM FULL;" > /tmp/fix.sql
postgres -D /var/lib/pgsql/data nagiosxi < /tmp/fix.sql
echo "VACUUM FULL;" > /tmp/fix.sql
postgres -D /var/lib/pgsql/data postgres < /tmp/fix.sql
echo "VACUUM FULL;" > /tmp/fix.sql
postgres -D /var/lib/pgsql/data template1 < /tmp/fix.sql
exit
service postgresql start

GldRush98 · Post by **GldRush98** » Mon Oct 05, 2015 10:03 am

tgriep wrote:The error on your failover system looks like the postgres database needs to be VACUUMed. Run the following in a shell to see if that resolves the errors.
Code: Select all
service postgresql stop
su postgres
echo "VACUUM FULL;" > /tmp/fix.sql
postgres -D /var/lib/pgsql/data nagiosxi < /tmp/fix.sql
echo "VACUUM FULL;" > /tmp/fix.sql
postgres -D /var/lib/pgsql/data postgres < /tmp/fix.sql
echo "VACUUM FULL;" > /tmp/fix.sql
postgres -D /var/lib/pgsql/data template1 < /tmp/fix.sql
exit
service postgresql start

This fixed it!

So I'm reading around, there is supposed to be an autovacuum process... I suppose I need to look in to that and see why that's not running or something?

Post by **tgriep** » Mon Oct 05, 2015 2:05 pm

Vacuuming may not be enabled / supported in all versions of postgress.
The config files are in this folder.

Code: Select all

/var/lib/pgsql/data

Is it alright to close this post?

GldRush98 · Post by **GldRush98** » Tue Oct 06, 2015 9:21 am

Ah, ok. Sounds good. I'll probably just leave it alone for now and keep this fix in my back pocket in case it happens again.
Thanks, yes you may close

Nagios Support Forum

A database connection error...

A database connection error...

Re: A database connection error...

Re: A database connection error...

Re: A database connection error...

Re: A database connection error...

Re: A database connection error...

Re: A database connection error...

Re: A database connection error...

Re: A database connection error...