Databse Error - issue persists after repairing

[danil.kim]

As I have got the error:
Database Error
A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.
Run the following from the CLI as root to attempt to repair the DB:

/usr/local/nagiosxi/scripts/repair_databases.sh

I started repair_databases.sh script multiple times, but without success.

the result of repair_databases.sh is below:
-----------------
DATABASE: nagios
TABLE:
nagios.nagios_acknowledgements OK
nagios.nagios_commands OK
nagios.nagios_commenthistory OK
nagios.nagios_comments OK
nagios.nagios_configfiles OK
nagios.nagios_configfilevariables OK
nagios.nagios_conninfo OK
nagios.nagios_contact_addresses OK
nagios.nagios_contact_notificationcommands OK
nagios.nagios_contactgroup_members OK
nagios.nagios_contactgroups OK
nagios.nagios_contactnotificationmethods OK
nagios.nagios_contactnotifications OK
nagios.nagios_contacts OK
nagios.nagios_contactstatus OK
nagios.nagios_customvariables OK
nagios.nagios_customvariablestatus OK
nagios.nagios_dbversion OK
nagios.nagios_downtimehistory OK
nagios.nagios_eventhandlers OK
nagios.nagios_externalcommands OK
nagios.nagios_flappinghistory OK
nagios.nagios_host_contactgroups OK
nagios.nagios_host_contacts OK
nagios.nagios_host_parenthosts OK
nagios.nagios_hostchecks OK
nagios.nagios_hostdependencies OK
nagios.nagios_hostescalation_contactgroups OK
nagios.nagios_hostescalation_contacts OK
nagios.nagios_hostescalations OK
nagios.nagios_hostgroup_members OK
nagios.nagios_hostgroups OK
nagios.nagios_hosts OK
nagios.nagios_hoststatus OK
nagios.nagios_instances OK
nagios.nagios_logentries OK
nagios.nagios_notifications OK
nagios.nagios_objects OK
nagios.nagios_processevents OK
nagios.nagios_programstatus OK
nagios.nagios_runtimevariables OK
nagios.nagios_scheduleddowntime OK
nagios.nagios_service_contactgroups OK
nagios.nagios_service_contacts OK
nagios.nagios_service_parentservices OK
nagios.nagios_servicechecks OK
nagios.nagios_servicedependencies OK
nagios.nagios_serviceescalation_contactgroups OK
nagios.nagios_serviceescalation_contacts OK
nagios.nagios_serviceescalations OK
nagios.nagios_servicegroup_members OK
nagios.nagios_servicegroups OK
nagios.nagios_services OK
nagios.nagios_servicestatus OK
nagios.nagios_statehistory OK
nagios.nagios_systemcommands OK
nagios.nagios_timedeventqueue OK
nagios.nagios_timedevents OK
nagios.nagios_timeperiod_timeranges OK
nagios.nagios_timeperiods OK
Issued remote command 'mysqlcheck -f -r -u nagios -pmypassword -h 10.239.116.8 --port=3306 --databases nagios'
DATABASE: nagiosql
TABLE:
nagiosql.tbl_command OK
nagiosql.tbl_contact OK
nagiosql.tbl_contactgroup OK
nagiosql.tbl_contacttemplate OK
nagiosql.tbl_domain OK
nagiosql.tbl_host OK
nagiosql.tbl_hostdependency OK
nagiosql.tbl_hostescalation OK
nagiosql.tbl_hostextinfo OK
nagiosql.tbl_hostgroup OK
nagiosql.tbl_hosttemplate OK
nagiosql.tbl_info OK
nagiosql.tbl_lnkContactToCommandHost OK
nagiosql.tbl_lnkContactToCommandService OK
nagiosql.tbl_lnkContactToContactgroup OK
nagiosql.tbl_lnkContactToContacttemplate OK
nagiosql.tbl_lnkContactToVariabledefinition OK
nagiosql.tbl_lnkContactgroupToContact OK
nagiosql.tbl_lnkContactgroupToContactgroup OK
nagiosql.tbl_lnkContacttemplateToCommandHost OK
nagiosql.tbl_lnkContacttemplateToCommandService OK
nagiosql.tbl_lnkContacttemplateToContactgroup OK
nagiosql.tbl_lnkContacttemplateToContacttemplate OK
nagiosql.tbl_lnkContacttemplateToVariabledefinition OK
nagiosql.tbl_lnkHostToContact OK
nagiosql.tbl_lnkHostToContactgroup OK
nagiosql.tbl_lnkHostToHost OK
nagiosql.tbl_lnkHostToHostgroup OK
nagiosql.tbl_lnkHostToHosttemplate OK
nagiosql.tbl_lnkHostToVariabledefinition OK
nagiosql.tbl_lnkHostdependencyToHost_DH OK
nagiosql.tbl_lnkHostdependencyToHost_H OK
nagiosql.tbl_lnkHostdependencyToHostgroup_DH OK
nagiosql.tbl_lnkHostdependencyToHostgroup_H OK
nagiosql.tbl_lnkHostescalationToContact OK
nagiosql.tbl_lnkHostescalationToContactgroup OK
nagiosql.tbl_lnkHostescalationToHost OK
nagiosql.tbl_lnkHostescalationToHostgroup OK
nagiosql.tbl_lnkHostgroupToHost OK
nagiosql.tbl_lnkHostgroupToHostgroup OK
nagiosql.tbl_lnkHosttemplateToContact OK
nagiosql.tbl_lnkHosttemplateToContactgroup OK
nagiosql.tbl_lnkHosttemplateToHost OK
nagiosql.tbl_lnkHosttemplateToHostgroup OK
nagiosql.tbl_lnkHosttemplateToHosttemplate OK
nagiosql.tbl_lnkHosttemplateToVariabledefinition OK
nagiosql.tbl_lnkServiceToContact OK
nagiosql.tbl_lnkServiceToContactgroup OK
nagiosql.tbl_lnkServiceToHost OK
nagiosql.tbl_lnkServiceToHostgroup OK
nagiosql.tbl_lnkServiceToServicegroup OK
nagiosql.tbl_lnkServiceToServicetemplate OK
nagiosql.tbl_lnkServiceToVariabledefinition OK
nagiosql.tbl_lnkServicedependencyToHost_DH OK
nagiosql.tbl_lnkServicedependencyToHost_H OK
nagiosql.tbl_lnkServicedependencyToHostgroup_DH OK
nagiosql.tbl_lnkServicedependencyToHostgroup_H OK
nagiosql.tbl_lnkServicedependencyToService_DS OK
nagiosql.tbl_lnkServicedependencyToService_S OK
nagiosql.tbl_lnkServiceescalationToContact OK
nagiosql.tbl_lnkServiceescalationToContactgroup OK
nagiosql.tbl_lnkServiceescalationToHost OK
nagiosql.tbl_lnkServiceescalationToHostgroup OK
nagiosql.tbl_lnkServiceescalationToService OK
nagiosql.tbl_lnkServicegroupToService OK
nagiosql.tbl_lnkServicegroupToServicegroup OK
nagiosql.tbl_lnkServicetemplateToContact OK
nagiosql.tbl_lnkServicetemplateToContactgroup OK
nagiosql.tbl_lnkServicetemplateToHost OK
nagiosql.tbl_lnkServicetemplateToHostgroup OK
nagiosql.tbl_lnkServicetemplateToServicegroup OK
nagiosql.tbl_lnkServicetemplateToServicetemplate OK
nagiosql.tbl_lnkServicetemplateToVariabledefinition OK
nagiosql.tbl_lnkTimeperiodToTimeperiod OK
nagiosql.tbl_logbook OK
nagiosql.tbl_mainmenu OK
nagiosql.tbl_permission
note : The storage engine for the table doesn't support repair
nagiosql.tbl_permission_inactive
note : The storage engine for the table doesn't support repair
nagiosql.tbl_service OK
nagiosql.tbl_servicedependency OK
nagiosql.tbl_serviceescalation OK
nagiosql.tbl_serviceextinfo OK
nagiosql.tbl_servicegroup OK
nagiosql.tbl_servicetemplate OK
nagiosql.tbl_session OK
nagiosql.tbl_session_locks OK
nagiosql.tbl_settings OK
nagiosql.tbl_submenu OK
nagiosql.tbl_timedefinition OK
nagiosql.tbl_timeperiod OK
nagiosql.tbl_user OK
nagiosql.tbl_variabledefinition OK
Issued remote command 'mysqlcheck -f -r -u nagiosql -pmypassword -h 10.239.116.8 --port=3306 --databases nagiosql'
Stopping ndo2db: done.
Starting ndo2db: done.
Stopping nagios: .done.
Starting nagios: done.

=======================
nagios offloaded database repair succeeded
nagiosql offloaded database repair succeeded

[dchurch]

That error message turns out is just a guess as to what's going on with the database. All that's known when it emits the error "A database connection error has been detected" is that PHP tried to open a connection to the database and it failed. OR, depending on the config, it tried to re-use a persistent database connection which failed.

If you PM me a system profile I can diagnose further. Get one by going to Admin (top menu) => System Profile (in the left menu), then clicking the blue button.

If you're unable to generate the the profile through the web interface, please try generating it from the command line by running these commands as root:

Code: Select all

rm -rf /usr/local/nagiosxi/var/components/profile*
/usr/local/nagiosxi/scripts/components/getprofile.sh SUPPORT

Then send me the resulting /usr/local/nagiosxi/var/components/profile.zip file.
If the profile script fails, please include the ENTIRE output.

In the mean time, could you investigate whether your Nagios XI system has a firewall between it and the database server that would close long-lived TCP connections? Or, if the rebooting of some network hardware would cause this? Or, if the database server had been taken offline or rebooted?

[danil.kim]

Hi dchurch,

thank you for reply!

The error I get -comes from Web-Interface - I am not able to login into Web-Interface of the NagiosXI.

The folder /usr/local/nagiosxi/scripts/components/ does not exist in my system.

The filrewall is opened between Nagios-App server and the Database server - this is not an issue.

The initial error has been caused from the following: the /VAR partition was full and Postgresql has been stopped, we extended the /VAR partition and afterwards came this error.

The state now:

MySQL is working
Postgresql is working
Nagios services are working Apache as well.

The Nagios Web-Application gives this error out.

[ssax]

Try running this one to get the profile as we need it to review the settings:

Code: Select all

/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh

Run these commands and send the full output:
- Use the usernames/passwords from /usr/local/nagiosxi/html/config.inc.php
- The first one is checking the nagios DB
- The second one is checking the nagisql DB

Code: Select all

mysql -h 10.239.116.8 -uyourusername -p'yourpassword' nagios -e 'status;'
mysql -h 10.239.116.8 -uyourusername -p'yourpassword' nagiosql -e 'status;'

If your nagiosxi DB is stored in postgresql, make sure your /usr/local/nagiosxi/html/config.inc.php has these setting on the nagiosxi db_info:

Code: Select all

$cfg['db_info'] = array(
    "nagiosxi" => array(
        "dbtype" => 'pgsql',
        "dbserver" => '',
        "user" => 'nagiosxi',
        "pwd" => 'n@gweb',
        "db" => 'nagiosxi',

What is the output of this command (checking connectivity):

Code: Select all

echo 'select count(*) from xi_users;' | psql nagiosxi nagiosxi

[danil.kim]

Try running this one to get the profile as we need it to review the settings:

Code: Select all

/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh

Run these commands and send the full output:
- Use the usernames/passwords from /usr/local/nagiosxi/html/config.inc.php
- The first one is checking the nagios DB
- The second one is checking the nagisql DB

Code: Select all

mysql -h 10.239.116.8 -uyourusername -p'yourpassword' nagios -e 'status;'
mysql -h 10.239.116.8 -uyourusername -p'yourpassword' nagiosql -e 'status;'

If your nagiosxi DB is stored in postgresql, make sure your /usr/local/nagiosxi/html/config.inc.php has these setting on the nagiosxi db_info:

Code: Select all

$cfg['db_info'] = array(
    "nagiosxi" => array(
        "dbtype" => 'pgsql',
        "dbserver" => '',
        "user" => 'nagiosxi',
        "pwd" => 'n@gweb',
        "db" => 'nagiosxi',

What is the output of this command (checking connectivity):

Code: Select all

echo 'select count(*) from xi_users;' | psql nagiosxi nagiosxi

Hi ssax,

thank you for your help!

There were some damaged Postgresql files. We did restore them and now everything is working again!

Thanks to all for help!

[dchurch]

Glad to hear you resolved it! Locking thread.

If you have any additional issues, feel free to make a new thread.