Nagios DB error page after RHEL6 patching

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
mgallowaycac
Posts: 5
Joined: Wed Jun 20, 2018 11:29 am

Nagios DB error page after RHEL6 patching

Post by mgallowaycac »

Howdy friends!

We recently patched our RHEL 6 server from 6.9 to 6.10. After said patching (and rebooting of the host) nagios is presenting the following page

Code: Select all

Databse Error
A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.
Run the following from the CLI as root to attempt to repair the DB:

/usr/local/nagiosxi/scripts/repair_databases.sh
So, we did as told, and tried to run the repair_databases.sh script. That keeps yielding the following and even after it runs, we are still told there is a db connection error detected.

Code: Select all

root@pu-nagios-01:/var/log > /usr/local/nagiosxi/scripts/repair_databases.sh
mysqld (pid  24796) is running...
mysqld (pid  24796) is running...
DATABASE: nagios
TABLE: 
/proj/database/mysql/nagios /var/log
Stopping mysqld:                                           [  OK  ]
- recovering (with sort) MyISAM-table 'nagios_service_parentservices.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
Starting mysqld:                                           [  OK  ]
/var/log
 
===============
REPAIR COMPLETE
===============
mysqld (pid  16227) is running...
DATABASE: nagiosql
TABLE: 
/proj/database/mysql/nagiosql /var/log
Stopping mysqld:                                           [  OK  ]
- recovering (with sort) MyISAM-table 'tbl_command.MYI'
Data records: 281
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_contactgroup.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_contact.MYI'
Data records: 51
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_contacttemplate.MYI'
Data records: 2
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_domain.MYI'
Data records: 1
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_hostdependency.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_hostescalation.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_hostextinfo.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_hostgroup.MYI'
Data records: 22
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_host.MYI'
Data records: 1044
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_hosttemplate.MYI'
Data records: 52
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_info.MYI'
Data records: 218
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactgroupToContactgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactgroupToContact.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkContacttemplateToCommandHost.MYI'
Data records: 2
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkContacttemplateToCommandService.MYI'
Data records: 2
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkContacttemplateToContactgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkContacttemplateToContacttemplate.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkContacttemplateToVariabledefinition.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactToCommandHost.MYI'
Data records: 6
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactToCommandService.MYI'
Data records: 6
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactToContactgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactToContacttemplate.MYI'
Data records: 52
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkContactToVariabledefinition.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostdependencyToHost_DH.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostdependencyToHostgroup_DH.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostdependencyToHostgroup_H.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostdependencyToHost_H.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostescalationToContactgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostescalationToContact.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostescalationToHostgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostescalationToHost.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostgroupToHostgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostgroupToHost.MYI'
Data records: 229
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHosttemplateToContactgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHosttemplateToContact.MYI'
Data records: 1
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHosttemplateToHostgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHosttemplateToHost.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHosttemplateToHosttemplate.MYI'
Data records: 50
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHosttemplateToVariabledefinition.MYI'
Data records: 1
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostToContactgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostToContact.MYI'
Data records: 1019
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostToHostgroup.MYI'
Data records: 22
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostToHost.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostToHosttemplate.MYI'
Data records: 1037
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkHostToVariabledefinition.MYI'
Data records: 966
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicedependencyToHost_DH.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicedependencyToHostgroup_DH.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicedependencyToHostgroup_H.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicedependencyToHost_H.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicedependencyToService_DS.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicedependencyToService_S.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceescalationToContactgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceescalationToContact.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceescalationToHostgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceescalationToHost.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceescalationToService.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicegroupToServicegroup.MYI'
Data records: 19
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicegroupToService.MYI'
Data records: 696
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToContactgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToContact.MYI'
Data records: 5
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToHostgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToHost.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToServicegroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToServicetemplate.MYI'
Data records: 82
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServicetemplateToVariabledefinition.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToContactgroup.MYI'
Data records: 0
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToContact.MYI'
Data records: 12151
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToHostgroup.MYI'
Data records: 3
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToHost.MYI'
Data records: 12078
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToServicegroup.MYI'
Data records: 295
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToServicetemplate.MYI'
Data records: 9192
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkServiceToVariabledefinition.MYI'
Data records: 9625
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_lnkTimeperiodToTimeperiod.MYI'
Data records: 1
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_logbook.MYI'
Data records: 2
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_mainmenu.MYI'
Data records: 7
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_servicedependency.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_serviceescalation.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_serviceextinfo.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_servicegroup.MYI'
Data records: 213
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_service.MYI'
Data records: 12016
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_servicetemplate.MYI'
Data records: 88
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_session_locks.MYI'
Data records: 1
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_session.MYI'
Data records: 1
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_settings.MYI'
Data records: 13
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_submenu.MYI'
Data records: 29
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_timedefinition.MYI'
Data records: 608
- Fixing index 1
          
---------

- recovering (with sort) MyISAM-table 'tbl_timeperiod.MYI'
Data records: 92
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_user.MYI'
Data records: 2
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_variabledefinition.MYI'
Data records: 17949
- Fixing index 1
Starting mysqld:                                           [  OK  ]
/var/log
 
===============
REPAIR COMPLETE
===============
Stopping ndo2db: done.
Starting ndo2db: done.
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.

=======================
nagios database repair succeeded
nagiosql database repair succeeded
/var/log/msqld.log is empty

cleaner.log shows the following

Code: Select all

root@pu-nagios-01:/usr/local/nagiosxi/var > cat cleaner.log
PHP Notice:  Undefined index: nagiosxi in /usr/local/nagiosxi/html/includes/db.inc.php on line 488
PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nagiosim/nagiosim.inc.php on line 489
PHP Notice:  Undefined index: nagiosxi in /usr/local/nagiosxi/html/includes/db.inc.php on line 488
PHP Notice:  Undefined index: nagiosxi in /usr/local/nagiosxi/html/includes/db.inc.php on line 488
PHP Warning:  pg_pconnect(): Unable to connect to PostgreSQL server: FATAL:  Ident authentication failed for user "nagiosxi" in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 699
PHP Warning:  pg_pconnect(): Unable to connect to PostgreSQL server: FATAL:  Ident authentication failed for user "nagiosxi" in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 699
PHP Warning:  pg_pconnect(): Unable to connect to PostgreSQL server: FATAL:  Ident authentication failed for user "nagiosxi" in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 699
PHP Warning:  pg_pconnect(): Unable to connect to PostgreSQL server: FATAL:  Ident authentication failed for user "nagiosxi" in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 699
PHP Warning:  pg_pconnect(): Unable to connect to PostgreSQL server: FATAL:  Ident authentication failed for user "nagiosxi" in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 699
PHP Warning:  pg_pconnect(): Unable to connect to PostgreSQL server: FATAL:  Ident authentication failed for user "nagiosxi" in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 699
PHP Warning:  pg_pconnect(): Unable to connect to PostgreSQL server: FATAL:  Ident authentication failed for user "nagiosxi" in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 699
PHP Warning:  pg_pconnect(): Unable to connect to PostgreSQL server: FATAL:  Ident authentication failed for user "nagiosxi" in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 699
PHP Warning:  pg_pconnect(): Unable to connect to PostgreSQL server: FATAL:  Ident authentication failed for user "nagiosxi" in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 699
PHP Warning:  pg_pconnect(): Unable to connect to PostgreSQL server: FATAL:  Ident authentication failed for user "nagiosxi" in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 699
<h3>Databse Error</h3>A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.<p>Run the following from the CLI as root to attempt to repair the DB:<br><pre>/usr/local/nagiosxi/scripts/repair_databases.sh</pre></p>root@pu-nagios-01:/usr/local/nagiosxi/var > psql postgres postgres
psql: FATAL:  Ident authentication failed for user "postgres"
But I'm looking at the postgres db and I don't see anything nagios related....

This is my first deeper dive with Nagios so I'm a bit wayward on which way to take my troubleshooting.
Top
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Nagios DB error page after RHEL6 patching

Post by cdienger »

What version of XI is this?

Are you able to connect to the postgres database? Try running

psql nagiosxi postgres

and if allow you to connect, run:

\d

which should return a list of tables available:

List of relations
Schema | Name | Type | Owner
--------+-----------------------------+----------+----------
public | if_command_id_seq | sequence | nagiosxi
public | if_meta_id_seq | sequence | nagiosxi
public | if_option_id_seq | sequence | nagiosxi
public | if_sysstat_id_seq | sequence | nagiosxi
public | if_user_id_seq | sequence | nagiosxi
public | if_usermeta_id_seq | sequence | nagiosxi
public | xi_auditlog | table | nagiosxi
public | xi_auditlog_auditlog_id_seq | sequence | nagiosxi
public | xi_commands | table | nagiosxi
public | xi_commands_command_id_seq | sequence | nagiosxi
public | xi_events | table | nagiosxi
public | xi_events_event_id_seq | sequence | nagiosxi
public | xi_incidents | table | nagiosxi
public | xi_incidents_id_seq | sequence | nagiosxi
public | xi_meta | table | nagiosxi
public | xi_meta_meta_id_seq | sequence | nagiosxi
public | xi_options | table | nagiosxi
public | xi_options_option_id_seq | sequence | nagiosxi
public | xi_sysstat | table | nagiosxi
public | xi_sysstat_sysstat_id_seq | sequence | nagiosxi
public | xi_usermeta | table | nagiosxi
public | xi_usermeta_usermeta_id_seq | sequence | nagiosxi
public | xi_users | table | nagiosxi
public | xi_users_user_id_seq | sequence | nagiosx

anything of interest in the logs(/var/lib/pgsql/data/pg_log) ?
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Top
mgallowaycac
Posts: 5
Joined: Wed Jun 20, 2018 11:29 am

Re: Nagios DB error page after RHEL6 patching

Post by mgallowaycac »

So I gave connecting a try. I got the following

Code: Select all

root@pu-nagios-01:/var/lib/pgsql/data/pg_log > psql nagiosxi postgres
psql: FATAL:  Ident authentication failed for user "postgres"
I su'ed over to the postgres user and was able to get the following list of DB's

Code: Select all

postgres@pu-nagios-01:~ > psql
psql (8.4.20)
Type "help" for help.

postgres=# \list
                                  List of databases
   Name    |  Owner   | Encoding |  Collation  |    Ctype    |   Access privileges   
-----------+----------+----------+-------------+-------------+-----------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres
                                                             : postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres
                                                             : postgres=CTc/postgres
(3 rows)
Mysql shows the following

Code: Select all

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| cac_misc           |
| mysql              |
| nagios             |
| nagiosql           |
| nagiosxi           |
| reboots            |
| removed_status     |
+--------------------+
and

Code: Select all

mysql> use nagiosxi;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+--------------------+
| Tables_in_nagiosxi |
+--------------------+
| xi_auditlog        |
| xi_commands        |
| xi_events          |
| xi_incidents       |
+--------------------+
4 rows in set (0.00 sec)
And I did check the files in /var/lib/pgsql/data/pg_log and I see some logs from as far back 2017

Code: Select all

root@pu-nagios-01:/var/lib/pgsql/data/pg_log > ls -latr
total 2684
-rw-------  1 postgres postgres  105598 Mar 10  2017 postgresql-Fri.log
-rw-------  1 postgres postgres  386955 Jul  3  2018 postgresql-Tue.log
drwx------  2 postgres postgres    4096 Mar 25 15:42 .
drwx------ 12 postgres postgres    4096 Mar 25 17:20 ..
-rw-------  1 postgres postgres 2233965 Mar 25 20:23 postgresql-Mon.log
However, even the one from 2017 has the Ident errors in them so now I'm feeling like these auth errors I'm seeing might be a red herring.
Top
mgallowaycac
Posts: 5
Joined: Wed Jun 20, 2018 11:29 am

Re: Nagios DB error page after RHEL6 patching

Post by mgallowaycac »

This can be considered resolved. After talking with some company vets, it seems the OS based init script for Postgres was never updated to reflect the correct location of the database nor was it documented as such :|.

So yeah, Full admission to a self inflicted wound.
Top
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Nagios DB error page after RHEL6 patching

Post by cdienger »

No worries! Glad a solution was found!
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Top
Locked

Return to “Nagios XI”