Database Error

[ksafa]

Hello,
We keep getting database error message on Operations Center screen. We've tried running repair_database script several times but still no luck. Only getting message when accessing Operations Center view. Other views appear to work fine as well as detail views.

Thanks

[npolovenko]

Hello, @ksafa. Please try running the following query to repair the database:

mysqlcheck -r -f -uroot -pnagiosxi --all-databases --use_frm

If the issue still persists please take a screenshot of the page with errors in XI and share it with us in the ticket.

[ksafa]

Below is output of the command. Screenshot attached.

Code: Select all

mysql.columns_priv                                 OK
mysql.db
warning  : Number of rows changed from 0 to 5
status   : OK
mysql.event                                        OK
mysql.func                                         OK
mysql.help_category
warning  : Number of rows changed from 0 to 39
status   : OK
mysql.help_keyword
warning  : Number of rows changed from 0 to 464
status   : OK
mysql.help_relation
warning  : Number of rows changed from 0 to 1028
status   : OK
mysql.help_topic
warning  : Number of rows changed from 0 to 508
status   : OK
mysql.host                                         OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv
warning  : Number of rows changed from 0 to 2
status   : OK
mysql.servers                                      OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user
warning  : Number of rows changed from 0 to 9
status   : OK
nagios.nagios_acknowledgements
warning  : Number of rows changed from 0 to 390
status   : OK
nagios.nagios_commands
warning  : Number of rows changed from 0 to 144
status   : OK
nagios.nagios_commenthistory
warning  : Number of rows changed from 0 to 85376
status   : OK
nagios.nagios_comments
warning  : Number of rows changed from 0 to 551
status   : OK
nagios.nagios_configfiles
warning  : Number of rows changed from 0 to 1
status   : OK
nagios.nagios_configfilevariables
warning  : Number of rows changed from 0 to 131
status   : OK
nagios.nagios_conninfo
warning  : Number of rows changed from 0 to 172
status   : OK
nagios.nagios_contact_addresses                    OK
nagios.nagios_contact_notificationcommands
warning  : Number of rows changed from 0 to 228
status   : OK
nagios.nagios_contactgroup_members
warning  : Number of rows changed from 0 to 19
status   : OK
nagios.nagios_contactgroups
warning  : Number of rows changed from 0 to 2
status   : OK
nagios.nagios_contactnotificationmethods
warning  : Number of rows changed from 0 to 99860
status   : OK
nagios.nagios_contactnotifications
warning  : Number of rows changed from 0 to 99849
status   : OK
nagios.nagios_contacts
warning  : Number of rows changed from 0 to 19
status   : OK
nagios.nagios_contactstatus
warning  : Number of rows changed from 0 to 19
status   : OK
nagios.nagios_customvariables
warning  : Number of rows changed from 0 to 3239
status   : OK
nagios.nagios_customvariablestatus
warning  : Number of rows changed from 0 to 3239
status   : OK
nagios.nagios_dbversion
warning  : Number of rows changed from 0 to 1
status   : OK
nagios.nagios_downtimehistory
warning  : Number of rows changed from 0 to 7211
status   : OK
nagios.nagios_eventhandlers                        OK
nagios.nagios_externalcommands
info     : Delete link points outside datafile at 74500
info     : Delete link points outside datafile at 889904
warning  : Number of rows changed from 0 to 1247223
status   : OK
nagios.nagios_flappinghistory
warning  : Number of rows changed from 0 to 18286
status   : OK
nagios.nagios_host_contactgroups
warning  : Number of rows changed from 0 to 3
status   : OK
nagios.nagios_host_contacts
warning  : Number of rows changed from 0 to 237
status   : OK
nagios.nagios_host_parenthosts                     OK
nagios.nagios_hostchecks                           OK
nagios.nagios_hostdependencies                     OK
nagios.nagios_hostescalation_contactgroups         OK
nagios.nagios_hostescalation_contacts              OK
nagios.nagios_hostescalations                      OK
nagios.nagios_hostgroup_members
warning  : Number of rows changed from 0 to 126
status   : OK
nagios.nagios_hostgroups
warning  : Number of rows changed from 0 to 2
status   : OK
nagios.nagios_hosts
warning  : Number of rows changed from 0 to 268
status   : OK
nagios.nagios_hoststatus
warning  : Number of rows changed from 0 to 268
status   : OK
nagios.nagios_instances
warning  : Number of rows changed from 0 to 1
status   : OK
nagios.nagios_logentries
warning  : Number of rows changed from 0 to 4357
status   : OK
nagios.nagios_notifications
warning  : Number of rows changed from 0 to 111004
status   : OK
nagios.nagios_objects
warning  : Number of rows changed from 0 to 5133
status   : OK
nagios.nagios_processevents
warning  : Number of rows changed from 0 to 734
status   : OK
nagios.nagios_programstatus
warning  : Number of rows changed from 0 to 1
status   : OK
nagios.nagios_runtimevariables
warning  : Number of rows changed from 0 to 18
status   : OK
nagios.nagios_scheduleddowntime
warning  : Number of rows changed from 0 to 476
status   : OK
nagios.nagios_service_contactgroups
warning  : Number of rows changed from 0 to 170
status   : OK
nagios.nagios_service_contacts
warning  : Number of rows changed from 0 to 3617
status   : OK
nagios.nagios_service_parentservices               OK
nagios.nagios_servicechecks                        OK
nagios.nagios_servicedependencies                  OK
nagios.nagios_serviceescalation_contactgroups      OK
nagios.nagios_serviceescalation_contacts           OK
nagios.nagios_serviceescalations                   OK
nagios.nagios_servicegroup_members
warning  : Number of rows changed from 0 to 933
status   : OK
nagios.nagios_servicegroups
warning  : Number of rows changed from 0 to 10
status   : OK
nagios.nagios_services
warning  : Number of rows changed from 0 to 3901
status   : OK
nagios.nagios_servicestatus
warning  : Number of rows changed from 0 to 3901
status   : OK
nagios.nagios_statehistory
warning  : Number of rows changed from 0 to 1325803
status   : OK
nagios.nagios_systemcommands                       OK
nagios.nagios_timedeventqueue                      OK
nagios.nagios_timedevents                          OK
nagios.nagios_timeperiod_timeranges
warning  : Number of rows changed from 0 to 152
status   : OK
nagios.nagios_timeperiods
warning  : Number of rows changed from 0 to 25
status   : OK
nagiosql.tbl_command
warning  : Number of rows changed from 0 to 147
status   : OK
nagiosql.tbl_contact
warning  : Number of rows changed from 0 to 18
status   : OK
nagiosql.tbl_contactgroup
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_contacttemplate
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_domain
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_host
warning  : Number of rows changed from 0 to 276
status   : OK
nagiosql.tbl_hostdependency                        OK
nagiosql.tbl_hostescalation                        OK
nagiosql.tbl_hostextinfo                           OK
nagiosql.tbl_hostgroup
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_hosttemplate
warning  : Number of rows changed from 0 to 50
status   : OK
nagiosql.tbl_info
warning  : Number of rows changed from 0 to 218
status   : OK
nagiosql.tbl_lnkContactToCommandHost
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_lnkContactToCommandService
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_lnkContactToContactgroup              OK
nagiosql.tbl_lnkContactToContacttemplate
warning  : Number of rows changed from 0 to 18
status   : OK
nagiosql.tbl_lnkContactToVariabledefinition        OK
nagiosql.tbl_lnkContactgroupToContact
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_lnkContactgroupToContactgroup         OK
nagiosql.tbl_lnkContacttemplateToCommandHost
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_lnkContacttemplateToCommandService
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_lnkContacttemplateToContactgroup
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_lnkContacttemplateToContacttemplate   OK
nagiosql.tbl_lnkContacttemplateToVariabledefinition OK
nagiosql.tbl_lnkHostToContact
warning  : Number of rows changed from 0 to 245
status   : OK
nagiosql.tbl_lnkHostToContactgroup                 OK
nagiosql.tbl_lnkHostToHost                         OK
nagiosql.tbl_lnkHostToHostgroup                    OK
nagiosql.tbl_lnkHostToHosttemplate
warning  : Number of rows changed from 0 to 278
status   : OK
nagiosql.tbl_lnkHostToVariabledefinition
warning  : Number of rows changed from 0 to 273
status   : OK
nagiosql.tbl_lnkHostdependencyToHost_DH            OK
nagiosql.tbl_lnkHostdependencyToHost_H             OK
nagiosql.tbl_lnkHostdependencyToHostgroup_DH       OK
nagiosql.tbl_lnkHostdependencyToHostgroup_H        OK
nagiosql.tbl_lnkHostescalationToContact            OK
nagiosql.tbl_lnkHostescalationToContactgroup       OK
nagiosql.tbl_lnkHostescalationToHost               OK
nagiosql.tbl_lnkHostescalationToHostgroup          OK
nagiosql.tbl_lnkHostgroupToHost
warning  : Number of rows changed from 0 to 126
status   : OK
nagiosql.tbl_lnkHostgroupToHostgroup               OK
nagiosql.tbl_lnkHosttemplateToContact              OK
nagiosql.tbl_lnkHosttemplateToContactgroup
warning  : Number of rows changed from 0 to 4
status   : OK
nagiosql.tbl_lnkHosttemplateToHost                 OK
nagiosql.tbl_lnkHosttemplateToHostgroup
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_lnkHosttemplateToHosttemplate
warning  : Number of rows changed from 0 to 48
status   : OK
nagiosql.tbl_lnkHosttemplateToVariabledefinition   OK
nagiosql.tbl_lnkServiceToContact
warning  : Number of rows changed from 0 to 2577
status   : OK
nagiosql.tbl_lnkServiceToContactgroup              OK
nagiosql.tbl_lnkServiceToHost
warning  : Number of rows changed from 0 to 3293
status   : OK
nagiosql.tbl_lnkServiceToHostgroup
warning  : Number of rows changed from 0 to 10
status   : OK
nagiosql.tbl_lnkServiceToServicegroup              OK
nagiosql.tbl_lnkServiceToServicetemplate
warning  : Number of rows changed from 0 to 2950
status   : OK
nagiosql.tbl_lnkServiceToVariabledefinition
warning  : Number of rows changed from 0 to 2834
status   : OK
nagiosql.tbl_lnkServicedependencyToHost_DH         OK
nagiosql.tbl_lnkServicedependencyToHost_H          OK
nagiosql.tbl_lnkServicedependencyToHostgroup_DH    OK
nagiosql.tbl_lnkServicedependencyToHostgroup_H     OK
nagiosql.tbl_lnkServicedependencyToService_DS      OK
nagiosql.tbl_lnkServicedependencyToService_S       OK
nagiosql.tbl_lnkServiceescalationToContact         OK
nagiosql.tbl_lnkServiceescalationToContactgroup    OK
nagiosql.tbl_lnkServiceescalationToHost            OK
nagiosql.tbl_lnkServiceescalationToHostgroup       OK
nagiosql.tbl_lnkServiceescalationToService         OK
nagiosql.tbl_lnkServicegroupToService
warning  : Number of rows changed from 0 to 626
status   : OK
nagiosql.tbl_lnkServicegroupToServicegroup
warning  : Number of rows changed from 0 to 4
status   : OK
nagiosql.tbl_lnkServicetemplateToContact           OK
nagiosql.tbl_lnkServicetemplateToContactgroup
warning  : Number of rows changed from 0 to 1
status   : OK
nagiosql.tbl_lnkServicetemplateToHost              OK
nagiosql.tbl_lnkServicetemplateToHostgroup         OK
nagiosql.tbl_lnkServicetemplateToServicegroup      OK
nagiosql.tbl_lnkServicetemplateToServicetemplate
warning  : Number of rows changed from 0 to 81
status   : OK
nagiosql.tbl_lnkServicetemplateToVariabledefinition OK
nagiosql.tbl_lnkTimeperiodToTimeperiod             OK
nagiosql.tbl_logbook
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_mainmenu
warning  : Number of rows changed from 0 to 7
status   : OK
nagiosql.tbl_service
warning  : Number of rows changed from 0 to 2890
status   : OK
nagiosql.tbl_servicedependency                     OK
nagiosql.tbl_serviceescalation                     OK
nagiosql.tbl_serviceextinfo                        OK
nagiosql.tbl_servicegroup
warning  : Number of rows changed from 0 to 10
status   : OK
nagiosql.tbl_servicetemplate
warning  : Number of rows changed from 0 to 83
status   : OK
nagiosql.tbl_session                               OK
nagiosql.tbl_session_locks                         OK
nagiosql.tbl_settings
warning  : Number of rows changed from 0 to 12
status   : OK
nagiosql.tbl_submenu
warning  : Number of rows changed from 0 to 29
status   : OK
nagiosql.tbl_timedefinition
warning  : Number of rows changed from 0 to 159
status   : OK
nagiosql.tbl_timeperiod
warning  : Number of rows changed from 0 to 25
status   : OK
nagiosql.tbl_user
warning  : Number of rows changed from 0 to 2
status   : OK
nagiosql.tbl_variabledefinition
warning  : Number of rows changed from 0 to 3107
status   : OK
nagiosxi.xi_auditlog
info     : Delete link points outside datafile at 61400
warning  : Number of rows changed from 0 to 9239
status   : OK
nagiosxi.xi_commands
warning  : Number of rows changed from 0 to 17
status   : OK
nagiosxi.xi_eventqueue
info     : Delete link points outside datafile at 0
info     : Found block that points outside data file at 3096
status   : OK
nagiosxi.xi_events
warning  : Number of rows changed from 0 to 1134
status   : OK
nagiosxi.xi_incidents
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_meta
info     : Delete link points outside datafile at 150808
info     : Delete link points outside datafile at 155624
info     : Delete link points outside datafile at 506368
warning  : Number of rows changed from 0 to 1140
status   : OK
nagiosxi.xi_options
warning  : Number of rows changed from 0 to 157
status   : OK
nagiosxi.xi_sysstat
warning  : Number of rows changed from 0 to 17
status   : OK
nagiosxi.xi_usermeta
warning  : Number of rows changed from 0 to 1467
status   : OK
nagiosxi.xi_users
warning  : Number of rows changed from 0 to 40
status   : OK

[npolovenko]

@ksafa, I see. Could you upload the DB log? Hopefully, it will show us some more details.

/var/log/mariadb/mariadb.log
/var/log/mysqld.log

[ksafa]

mariadb.log attached. Could not find mysqld.log.

[tgriep]

If you look at the bottom of the log file, is shows that the tables are repaired.
But, if this is an older XI server or an older XI server that was migrated, it could be still using the Postgres database and that one is causing the problems.

If the server is using Postgres, there should be log files in this folder

Code: Select all

/var/lib/pgsql/data/pg_log/

Get the following file from the server and upload it to the post.

Code: Select all

postgresql-Fri.log

Typically, the Postgres database just needs to be vacuumed to remove the old transaction entries.
This KB article has instructions for doing that.

https://support.nagios.com/kb/article/n ... ce-25.html


If that still does not work, I would need you to run the follwoing as root and post the output here.

Code: Select all

df -h
df -i
chage -l nagios
ps -ef --cols=300
mysql -u root -pnagiosxi -e "show global status like '%used_connections%'; show variables like 'max_connections';"

[ksafa]

There is nothing under /var/lib/pgsql/data/ folder.

Not seeing the database error anymore. I did notice at the time we were having the issue that the load was quite high on the server so i shutdown the monitoring engine just to see what would happen. Is it possible we ran into some sort of load/connection limit. I ran the commands you requested and will PM the output to you.

Thanks

[tgriep]

It could be that there was a stuck Nagios process running and stopping them stopped the error.
The data in the PM does show that the nagios process is not running so it did not startup.
Run this to start it and check the status.

Code: Select all

service nagios start
service nagios status

Post the output.

[ksafa]

#service nagios status
nagios (pid 15006) is running...

[tgriep]

The status shows that the nagios process is running. Is the server functioning now and not displaying any errors or showing that any other services are not running?