Page 1 of 2
Database Error
Posted: Thu Feb 21, 2019 2:27 pm
by ksafa
Hello,
We keep getting database error message on Operations Center screen. We've tried running repair_database script several times but still no luck. Only getting message when accessing Operations Center view. Other views appear to work fine as well as detail views.
Thanks
Re: Database Error
Posted: Thu Feb 21, 2019 3:56 pm
by npolovenko
Hello,
@ksafa. Please try running the following query to repair the database:
mysqlcheck -r -f -uroot -pnagiosxi --all-databases --use_frm
If the issue still persists please take a screenshot of the page with errors in XI and share it with us in the ticket.
Re: Database Error
Posted: Thu Feb 21, 2019 4:09 pm
by ksafa
Below is output of the command. Screenshot attached.
Code: Select all
mysql.columns_priv OK
mysql.db
warning : Number of rows changed from 0 to 5
status : OK
mysql.event OK
mysql.func OK
mysql.help_category
warning : Number of rows changed from 0 to 39
status : OK
mysql.help_keyword
warning : Number of rows changed from 0 to 464
status : OK
mysql.help_relation
warning : Number of rows changed from 0 to 1028
status : OK
mysql.help_topic
warning : Number of rows changed from 0 to 508
status : OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.proxies_priv
warning : Number of rows changed from 0 to 2
status : OK
mysql.servers OK
mysql.tables_priv OK
mysql.time_zone OK
mysql.time_zone_leap_second OK
mysql.time_zone_name OK
mysql.time_zone_transition OK
mysql.time_zone_transition_type OK
mysql.user
warning : Number of rows changed from 0 to 9
status : OK
nagios.nagios_acknowledgements
warning : Number of rows changed from 0 to 390
status : OK
nagios.nagios_commands
warning : Number of rows changed from 0 to 144
status : OK
nagios.nagios_commenthistory
warning : Number of rows changed from 0 to 85376
status : OK
nagios.nagios_comments
warning : Number of rows changed from 0 to 551
status : OK
nagios.nagios_configfiles
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_configfilevariables
warning : Number of rows changed from 0 to 131
status : OK
nagios.nagios_conninfo
warning : Number of rows changed from 0 to 172
status : OK
nagios.nagios_contact_addresses OK
nagios.nagios_contact_notificationcommands
warning : Number of rows changed from 0 to 228
status : OK
nagios.nagios_contactgroup_members
warning : Number of rows changed from 0 to 19
status : OK
nagios.nagios_contactgroups
warning : Number of rows changed from 0 to 2
status : OK
nagios.nagios_contactnotificationmethods
warning : Number of rows changed from 0 to 99860
status : OK
nagios.nagios_contactnotifications
warning : Number of rows changed from 0 to 99849
status : OK
nagios.nagios_contacts
warning : Number of rows changed from 0 to 19
status : OK
nagios.nagios_contactstatus
warning : Number of rows changed from 0 to 19
status : OK
nagios.nagios_customvariables
warning : Number of rows changed from 0 to 3239
status : OK
nagios.nagios_customvariablestatus
warning : Number of rows changed from 0 to 3239
status : OK
nagios.nagios_dbversion
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_downtimehistory
warning : Number of rows changed from 0 to 7211
status : OK
nagios.nagios_eventhandlers OK
nagios.nagios_externalcommands
info : Delete link points outside datafile at 74500
info : Delete link points outside datafile at 889904
warning : Number of rows changed from 0 to 1247223
status : OK
nagios.nagios_flappinghistory
warning : Number of rows changed from 0 to 18286
status : OK
nagios.nagios_host_contactgroups
warning : Number of rows changed from 0 to 3
status : OK
nagios.nagios_host_contacts
warning : Number of rows changed from 0 to 237
status : OK
nagios.nagios_host_parenthosts OK
nagios.nagios_hostchecks OK
nagios.nagios_hostdependencies OK
nagios.nagios_hostescalation_contactgroups OK
nagios.nagios_hostescalation_contacts OK
nagios.nagios_hostescalations OK
nagios.nagios_hostgroup_members
warning : Number of rows changed from 0 to 126
status : OK
nagios.nagios_hostgroups
warning : Number of rows changed from 0 to 2
status : OK
nagios.nagios_hosts
warning : Number of rows changed from 0 to 268
status : OK
nagios.nagios_hoststatus
warning : Number of rows changed from 0 to 268
status : OK
nagios.nagios_instances
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_logentries
warning : Number of rows changed from 0 to 4357
status : OK
nagios.nagios_notifications
warning : Number of rows changed from 0 to 111004
status : OK
nagios.nagios_objects
warning : Number of rows changed from 0 to 5133
status : OK
nagios.nagios_processevents
warning : Number of rows changed from 0 to 734
status : OK
nagios.nagios_programstatus
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_runtimevariables
warning : Number of rows changed from 0 to 18
status : OK
nagios.nagios_scheduleddowntime
warning : Number of rows changed from 0 to 476
status : OK
nagios.nagios_service_contactgroups
warning : Number of rows changed from 0 to 170
status : OK
nagios.nagios_service_contacts
warning : Number of rows changed from 0 to 3617
status : OK
nagios.nagios_service_parentservices OK
nagios.nagios_servicechecks OK
nagios.nagios_servicedependencies OK
nagios.nagios_serviceescalation_contactgroups OK
nagios.nagios_serviceescalation_contacts OK
nagios.nagios_serviceescalations OK
nagios.nagios_servicegroup_members
warning : Number of rows changed from 0 to 933
status : OK
nagios.nagios_servicegroups
warning : Number of rows changed from 0 to 10
status : OK
nagios.nagios_services
warning : Number of rows changed from 0 to 3901
status : OK
nagios.nagios_servicestatus
warning : Number of rows changed from 0 to 3901
status : OK
nagios.nagios_statehistory
warning : Number of rows changed from 0 to 1325803
status : OK
nagios.nagios_systemcommands OK
nagios.nagios_timedeventqueue OK
nagios.nagios_timedevents OK
nagios.nagios_timeperiod_timeranges
warning : Number of rows changed from 0 to 152
status : OK
nagios.nagios_timeperiods
warning : Number of rows changed from 0 to 25
status : OK
nagiosql.tbl_command
warning : Number of rows changed from 0 to 147
status : OK
nagiosql.tbl_contact
warning : Number of rows changed from 0 to 18
status : OK
nagiosql.tbl_contactgroup
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_contacttemplate
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_domain
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_host
warning : Number of rows changed from 0 to 276
status : OK
nagiosql.tbl_hostdependency OK
nagiosql.tbl_hostescalation OK
nagiosql.tbl_hostextinfo OK
nagiosql.tbl_hostgroup
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_hosttemplate
warning : Number of rows changed from 0 to 50
status : OK
nagiosql.tbl_info
warning : Number of rows changed from 0 to 218
status : OK
nagiosql.tbl_lnkContactToCommandHost
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkContactToCommandService
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkContactToContactgroup OK
nagiosql.tbl_lnkContactToContacttemplate
warning : Number of rows changed from 0 to 18
status : OK
nagiosql.tbl_lnkContactToVariabledefinition OK
nagiosql.tbl_lnkContactgroupToContact
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkContactgroupToContactgroup OK
nagiosql.tbl_lnkContacttemplateToCommandHost
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_lnkContacttemplateToCommandService
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_lnkContacttemplateToContactgroup
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkContacttemplateToContacttemplate OK
nagiosql.tbl_lnkContacttemplateToVariabledefinition OK
nagiosql.tbl_lnkHostToContact
warning : Number of rows changed from 0 to 245
status : OK
nagiosql.tbl_lnkHostToContactgroup OK
nagiosql.tbl_lnkHostToHost OK
nagiosql.tbl_lnkHostToHostgroup OK
nagiosql.tbl_lnkHostToHosttemplate
warning : Number of rows changed from 0 to 278
status : OK
nagiosql.tbl_lnkHostToVariabledefinition
warning : Number of rows changed from 0 to 273
status : OK
nagiosql.tbl_lnkHostdependencyToHost_DH OK
nagiosql.tbl_lnkHostdependencyToHost_H OK
nagiosql.tbl_lnkHostdependencyToHostgroup_DH OK
nagiosql.tbl_lnkHostdependencyToHostgroup_H OK
nagiosql.tbl_lnkHostescalationToContact OK
nagiosql.tbl_lnkHostescalationToContactgroup OK
nagiosql.tbl_lnkHostescalationToHost OK
nagiosql.tbl_lnkHostescalationToHostgroup OK
nagiosql.tbl_lnkHostgroupToHost
warning : Number of rows changed from 0 to 126
status : OK
nagiosql.tbl_lnkHostgroupToHostgroup OK
nagiosql.tbl_lnkHosttemplateToContact OK
nagiosql.tbl_lnkHosttemplateToContactgroup
warning : Number of rows changed from 0 to 4
status : OK
nagiosql.tbl_lnkHosttemplateToHost OK
nagiosql.tbl_lnkHosttemplateToHostgroup
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkHosttemplateToHosttemplate
warning : Number of rows changed from 0 to 48
status : OK
nagiosql.tbl_lnkHosttemplateToVariabledefinition OK
nagiosql.tbl_lnkServiceToContact
warning : Number of rows changed from 0 to 2577
status : OK
nagiosql.tbl_lnkServiceToContactgroup OK
nagiosql.tbl_lnkServiceToHost
warning : Number of rows changed from 0 to 3293
status : OK
nagiosql.tbl_lnkServiceToHostgroup
warning : Number of rows changed from 0 to 10
status : OK
nagiosql.tbl_lnkServiceToServicegroup OK
nagiosql.tbl_lnkServiceToServicetemplate
warning : Number of rows changed from 0 to 2950
status : OK
nagiosql.tbl_lnkServiceToVariabledefinition
warning : Number of rows changed from 0 to 2834
status : OK
nagiosql.tbl_lnkServicedependencyToHost_DH OK
nagiosql.tbl_lnkServicedependencyToHost_H OK
nagiosql.tbl_lnkServicedependencyToHostgroup_DH OK
nagiosql.tbl_lnkServicedependencyToHostgroup_H OK
nagiosql.tbl_lnkServicedependencyToService_DS OK
nagiosql.tbl_lnkServicedependencyToService_S OK
nagiosql.tbl_lnkServiceescalationToContact OK
nagiosql.tbl_lnkServiceescalationToContactgroup OK
nagiosql.tbl_lnkServiceescalationToHost OK
nagiosql.tbl_lnkServiceescalationToHostgroup OK
nagiosql.tbl_lnkServiceescalationToService OK
nagiosql.tbl_lnkServicegroupToService
warning : Number of rows changed from 0 to 626
status : OK
nagiosql.tbl_lnkServicegroupToServicegroup
warning : Number of rows changed from 0 to 4
status : OK
nagiosql.tbl_lnkServicetemplateToContact OK
nagiosql.tbl_lnkServicetemplateToContactgroup
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkServicetemplateToHost OK
nagiosql.tbl_lnkServicetemplateToHostgroup OK
nagiosql.tbl_lnkServicetemplateToServicegroup OK
nagiosql.tbl_lnkServicetemplateToServicetemplate
warning : Number of rows changed from 0 to 81
status : OK
nagiosql.tbl_lnkServicetemplateToVariabledefinition OK
nagiosql.tbl_lnkTimeperiodToTimeperiod OK
nagiosql.tbl_logbook
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_mainmenu
warning : Number of rows changed from 0 to 7
status : OK
nagiosql.tbl_service
warning : Number of rows changed from 0 to 2890
status : OK
nagiosql.tbl_servicedependency OK
nagiosql.tbl_serviceescalation OK
nagiosql.tbl_serviceextinfo OK
nagiosql.tbl_servicegroup
warning : Number of rows changed from 0 to 10
status : OK
nagiosql.tbl_servicetemplate
warning : Number of rows changed from 0 to 83
status : OK
nagiosql.tbl_session OK
nagiosql.tbl_session_locks OK
nagiosql.tbl_settings
warning : Number of rows changed from 0 to 12
status : OK
nagiosql.tbl_submenu
warning : Number of rows changed from 0 to 29
status : OK
nagiosql.tbl_timedefinition
warning : Number of rows changed from 0 to 159
status : OK
nagiosql.tbl_timeperiod
warning : Number of rows changed from 0 to 25
status : OK
nagiosql.tbl_user
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_variabledefinition
warning : Number of rows changed from 0 to 3107
status : OK
nagiosxi.xi_auditlog
info : Delete link points outside datafile at 61400
warning : Number of rows changed from 0 to 9239
status : OK
nagiosxi.xi_commands
warning : Number of rows changed from 0 to 17
status : OK
nagiosxi.xi_eventqueue
info : Delete link points outside datafile at 0
info : Found block that points outside data file at 3096
status : OK
nagiosxi.xi_events
warning : Number of rows changed from 0 to 1134
status : OK
nagiosxi.xi_incidents
note : The storage engine for the table doesn't support repair
nagiosxi.xi_meta
info : Delete link points outside datafile at 150808
info : Delete link points outside datafile at 155624
info : Delete link points outside datafile at 506368
warning : Number of rows changed from 0 to 1140
status : OK
nagiosxi.xi_options
warning : Number of rows changed from 0 to 157
status : OK
nagiosxi.xi_sysstat
warning : Number of rows changed from 0 to 17
status : OK
nagiosxi.xi_usermeta
warning : Number of rows changed from 0 to 1467
status : OK
nagiosxi.xi_users
warning : Number of rows changed from 0 to 40
status : OK
Re: Database Error
Posted: Thu Feb 21, 2019 5:23 pm
by npolovenko
@ksafa, I see. Could you upload the DB log? Hopefully, it will show us some more details.
/var/log/mariadb/mariadb.log
/var/log/mysqld.log
Re: Database Error
Posted: Fri Feb 22, 2019 7:50 am
by ksafa
mariadb.log attached. Could not find mysqld.log.
Re: Database Error
Posted: Fri Feb 22, 2019 3:32 pm
by tgriep
If you look at the bottom of the log file, is shows that the tables are repaired.
But, if this is an older XI server or an older XI server that was migrated, it could be still using the Postgres database and that one is causing the problems.
If the server is using Postgres, there should be log files in this folder
Get the following file from the server and upload it to the post.
Typically, the Postgres database just needs to be vacuumed to remove the old transaction entries.
This KB article has instructions for doing that.
https://support.nagios.com/kb/article/n ... ce-25.html
If that still does not work, I would need you to run the follwoing as root and post the output here.
Code: Select all
df -h
df -i
chage -l nagios
ps -ef --cols=300
mysql -u root -pnagiosxi -e "show global status like '%used_connections%'; show variables like 'max_connections';"
Re: Database Error
Posted: Fri Feb 22, 2019 4:05 pm
by ksafa
There is nothing under /var/lib/pgsql/data/ folder.
Not seeing the database error anymore. I did notice at the time we were having the issue that the load was quite high on the server so i shutdown the monitoring engine just to see what would happen. Is it possible we ran into some sort of load/connection limit. I ran the commands you requested and will PM the output to you.
Thanks
Re: Database Error
Posted: Fri Feb 22, 2019 4:17 pm
by tgriep
It could be that there was a stuck Nagios process running and stopping them stopped the error.
The data in the PM does show that the nagios process is not running so it did not startup.
Run this to start it and check the status.
Code: Select all
service nagios start
service nagios status
Post the output.
Re: Database Error
Posted: Mon Feb 25, 2019 8:05 am
by ksafa
#service nagios status
nagios (pid 15006) is running...
Re: Database Error
Posted: Mon Feb 25, 2019 12:08 pm
by tgriep
The status shows that the nagios process is running. Is the server functioning now and not displaying any errors or showing that any other services are not running?