database connection error

[QS1]

VM Centos 5.7 32 bit
nagiosxi 2014R2.3

We recently moved off the nagiosxi 32 bit OS and into the 64 bit OS. I tried accessing the non-production 32 bit OS to run a historical report and get the following
Message:
- A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.
Run the following from the CLI as root to attempt to repair the DB
/usr/local/nagiosxi/scripts/repair_databases.sh

- I ran the repair and restarted the server but I still cannot get any further than the NagiosXI Welcome page. I've tried to access the Nagios webpage and get a 403 forbidden error.

Please advise

[lmiltchev]

Do you have another machine running on the same IP address? Run:

Code: Select all

service httpd restart
getenforce
tail -50 /var/log/httpd/error_log
tail -50 /var/log/mysqld.log

and post the output in code wraps.

[QS1]

root@nagiosxi ~]# getenforce
Disabled
[root@nagiosxi ~]# tail -50 /var/log/httpd/error_log
- error logs look okay



[root@nagiosxi ~]# tail -50 /var/log/mysqld.log
150205 10:48:48 mysqld ended

150205 10:49:50 mysqld started
150205 10:49:51 InnoDB: Started; log sequence number 0 43675
150205 10:49:51 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150205 11:06:20 [Note] /usr/libexec/mysqld: Normal shutdown

150205 11:06:22 InnoDB: Starting shutdown...
150205 11:06:23 InnoDB: Shutdown completed; log sequence number 0 43675
150205 11:06:23 [Note] /usr/libexec/mysqld: Shutdown complete

150205 11:06:24 mysqld ended

150205 11:24:35 mysqld started
150205 11:24:36 InnoDB: Started; log sequence number 0 43675
150205 11:24:36 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150205 13:07:52 [Note] /usr/libexec/mysqld: Normal shutdown

150205 13:07:52 InnoDB: Starting shutdown...
150205 13:07:53 InnoDB: Shutdown completed; log sequence number 0 43675
150205 13:07:53 [Note] /usr/libexec/mysqld: Shutdown complete

150205 13:07:53 mysqld ended

150205 13:07:56 mysqld started
150205 13:07:57 InnoDB: Started; log sequence number 0 43675
150205 13:07:57 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150206 7:03:01 [ERROR] /usr/libexec/mysqld: Disk is full writing './nagios/nagios_logentries.MYD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
150206 7:03:01 [ERROR] /usr/libexec/mysqld: Retry in 60 secs. Message reprinted in 600 secs
150207 7:01:50 [ERROR] /usr/libexec/mysqld: Disk is full writing './nagios/nagios_notifications.MYD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
150207 7:01:50 [ERROR] /usr/libexec/mysqld: Retry in 60 secs. Message reprinted in 600 secs
150208 7:01:59 [ERROR] /usr/libexec/mysqld: Disk is full writing './nagios/nagios_notifications.MYD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
150208 7:01:59 [ERROR] /usr/libexec/mysqld: Retry in 60 secs. Message reprinted in 600 secs
150209 7:02:06 [ERROR] /usr/libexec/mysqld: Disk is full writing '/var/lib/mysql/nagios/nagios_logentries.MYI' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
150209 7:02:06 [ERROR] /usr/libexec/mysqld: Retry in 60 secs. Message reprinted in 600 secs
150210 7:02:28 [ERROR] /usr/libexec/mysqld: Disk is full writing './nagios/nagios_notifications.MYD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs de150210 10:55:48 [Note] /usr/libexec/mysqld: Normal shutdown

150210 10:55:48 InnoDB: Starting shutdown...
150210 10:55:49 InnoDB: Shutdown completed; log sequence number 0 43675
150210 10:55:49 [Note] /usr/libexec/mysqld: Shutdown complete

150210 10:55:49 mysqld ended

150210 10:57:51 mysqld started
150210 10:57:52 InnoDB: Started; log sequence number 0 43675
150210 10:57:52 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
[root@nagiosxi ~]#

[lmiltchev]

150209 7:02:06 [ERROR] /usr/libexec/mysqld: Disk is full writing '/var/lib/mysql/nagios/nagios_logentries.MYI' (Errcode: 28). Waiting for someone to free space...

Do you have enough HDD space left?

Code: Select all

df -h

Did you set up any mysql disk quotas? Are you seeing any filesystem errors in /var/log/messages?

[QS1]

It was a bit full (only 1.9 GB remaining) 91%

I deleted older backups and now have:

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00 108G 82G 20G 81% /
/dev/sda1 99M 25M 70M 27% /boot
tmpfs 1.5G 0 1.5G 0% /dev/shm


I re-ran the repair database script and still see the database connection error

I do not see any errors in var/log/messages

[scottwilkerson]

Lets also verify you aren't out of inodes

Code: Select all

df -i

Finally, if it still get the same messages, lets run the following:

Code: Select all

echo "REPAIR TABLE nagios_logentries;"|mysql -pnagiosxi nagios

[QS1]

[root@nagiosxi ~]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup00-LogVol00 28999680 2560214 26439466 9% /
/dev/sda1 26104 47 26057 1% /boot
tmpfs 219732 1 219731 1% /dev/shm

root@nagiosxi ~]# echo "REPAIR TABLE nagios_logentries;"|mysql -pnagiosxi nagios
Table Op Msg_type Msg_text
nagios.nagios_logentries repair warning Number of rows changed from 8853181 to 11058674
nagios.nagios_logentries repair status OK

I rebooted the VM and still get the database connection error

[scottwilkerson]

can you show us the results of the following commands

Code: Select all

tail -50 /var/log/httpd/error_log
tail -50 /var/log/mysqld.log

Thanks

[QS1]

[root@nagiosxi ~]# tail -50 /var/log/httpd/error_log

[Wed Feb 11 10:53:01 2015] [error] [client 127.0.0.1] PHP Warning: Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nagiosim/nagiosim.inc.php on line 395
[Wed Feb 11 10:53:01 2015] [error] [client 127.0.0.1] PHP Notice: Undefined index: ndoutils in /usr/local/nagiosxi/html/includes/db.inc.php on line 475
[Wed Feb 11 10:53:01 2015] [error] [client 127.0.0.1] PHP Warning: pg_pconnect() [<a href='function.pg-pconnect'>function.pg-pconnect</a>]: Unable to connect to PostgreSQL server: could not translate host name "localhost" to address: Temporary failure in name resolution in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 682
[Wed Feb 11 10:53:01 2015] [error] [client 127.0.0.1] PHP Notice: Undefined variable: result in /usr/local/nagiosxi/html/includes/db.inc.php on line 241
[Wed Feb 11 10:54:13 2015] [error] [client 127.0.0.1] Invalid URI in request GET 72.34.211.102/help HTTP/1.0
[Wed Feb 11 10:54:23 2015] [error] [client 127.0.0.1] PHP Warning: pg_pconnect() [<a href='function.pg-pconnect'>function.pg-pconnect</a>]: Unable to connect to PostgreSQL server: could not translate host name "localhost" to address: Temporary failure in name resolution in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 682
[Wed Feb 11 10:54:23 2015] [error] [client 127.0.0.1] PHP Notice: Undefined variable: result in /usr/local/nagiosxi/html/includes/db.inc.php on line 241
[Wed Feb 11 10:54:23 2015] [error] [client 127.0.0.1] PHP Warning: pg_pconnect() [<a href='function.pg-pconnect'>function.pg-pconnect</a>]: Unable to connect to PostgreSQL server: could not translate host name "localhost" to address: Temporary failure in name resolution in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 682
[Wed Feb 11 10:54:23 2015] [error] [client 127.0.0.1] PHP Notice: Undefined variable: result in /usr/local/nagiosxi/html/includes/db.inc.php on line 241
[Wed Feb 11 10:54:23 2015] [error] [client 127.0.0.1] PHP Warning: Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nagiosim/nagiosim.inc.php on line 395
[Wed Feb 11 10:54:24 2015] [error] [client 127.0.0.1] PHP Notice: Undefined index: ndoutils in /usr/local/nagiosxi/html/includes/db.inc.php on line 475
[Wed Feb 11 10:54:24 2015] [error] [client 127.0.0.1] PHP Warning: pg_pconnect() [<a href='function.pg-pconnect'>function.pg-pconnect</a>]: Unable to connect to PostgreSQL server: could not translate host name "localhost" to address: Temporary failure in name resolution in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 682
[Wed Feb 11 10:54:24 2015] [error] [client 127.0.0.1] PHP Notice: Undefined variable: result in /usr/local/nagiosxi/html/includes/db.inc.php on line 241
[Wed Feb 11 10:54:30 2015] [error] [client 127.0.0.1] PHP Warning: pg_pconnect() [<a href='function.pg-pconnect'>function.pg-pconnect</a>]: Unable to connect to PostgreSQL server: could not translate host name "localhost" to address: Temporary failure in name resolution in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 682
[Wed Feb 11 10:54:30 2015] [error] [client 127.0.0.1] PHP Notice: Undefined variable: result in /usr/local/nagiosxi/html/includes/db.inc.php on line 241
[Wed Feb 11 10:54:30 2015] [error] [client 127.0.0.1] PHP Warning: pg_pconnect() [<a href='function.pg-pconnect'>function.pg-pconnect</a>]: Unable to connect to PostgreSQL server: could not translate host name "localhost" to address: Temporary failure in name resolution in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 682
[Wed Feb 11 10:54:30 2015] [error] [client 127.0.0.1] PHP Notice: Undefined variable: result in /usr/local/nagiosxi/html/includes/db.inc.php on line 241
[Wed Feb 11 10:54:30 2015] [error] [client 127.0.0.1] PHP Warning: Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nagiosim/nagiosim.inc.php on line 395
[Wed Feb 11 10:54:30 2015] [error] [client 127.0.0.1] PHP Notice: Undefined index: ndoutils in /usr/local/nagiosxi/html/includes/db.inc.php on line 475
[Wed Feb 11 10:54:30 2015] [error] [client 127.0.0.1] PHP Warning: pg_pconnect() [<a href='function.pg-pconnect'>function.pg-pconnect</a>]: Unable to connect to PostgreSQL server: could not translate host name "localhost" to address: Temporary failure in name resolution in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 682
[Wed Feb 11 10:54:30 2015] [error] [client 127.0.0.1] PHP Notice: Undefined variable: result in /usr/local/nagiosxi/html/includes/db.inc.php on line 241
[Wed Feb 11 10:54:35 2015] [notice] caught SIGTERM, shutting down
[Wed Feb 11 10:54:35 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Feb 11 10:54:35 2015] [notice] SSL FIPS mode disabled
[Wed Feb 11 10:54:35 2015] [warn] RSA server certificate CommonName (CN) `www.qnagios.com' does NOT match server name!?
[Wed Feb 11 10:54:35 2015] [notice] Digest: generating secret for digest authentication ...
[Wed Feb 11 10:54:35 2015] [notice] Digest: done
[Wed Feb 11 10:54:35 2015] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Wed Feb 11 10:54:35 2015] [notice] SSL FIPS mode disabled
[Wed Feb 11 10:54:36 2015] [warn] RSA server certificate CommonName (CN) `www.qnagios.com' does NOT match server name!?
[Wed Feb 11 10:54:36 2015] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Wed Feb 11 11:09:13 2015] [error] [client 127.0.0.1] Invalid URI in request GET 72.34.211.102/help HTTP/1.0
[Wed Feb 11 11:24:13 2015] [error] [client 127.0.0.1] Invalid URI in request GET 72.34.211.102/help HTTP/1.0
[Wed Feb 11 11:39:13 2015] [error] [client 127.0.0.1] Invalid URI in request GET 72.34.211.102/help HTTP/1.0
[Wed Feb 11 11:54:13 2015] [error] [client 127.0.0.1] Invalid URI in request GET 72.34.211.102/help HTTP/1.0
[Wed Feb 11 12:09:13 2015] [error] [client 127.0.0.1] Invalid URI in request GET 72.34.211.102/help HTTP/1.0
[Wed Feb 11 12:24:13 2015] [error] [client 127.0.0.1] Invalid URI in request GET 72.34.211.102/help HTTP/1.0
[Wed Feb 11 12:39:13 2015] [error] [client 127.0.0.1] Invalid URI in request GET 72.34.211.102/help HTTP/1.0
[Wed Feb 11 12:54:13 2015] [error] [client 127.0.0.1] Invalid URI in request GET 72.34.211.102/help HTTP/1.0
[Wed Feb 11 13:09:14 2015] [error] [client 127.0.0.1] Invalid URI in request GET 72.34.211.102/help HTTP/1.0
[Wed Feb 11 13:09:36 2015] [notice] caught SIGTERM, shutting down
[Wed Feb 11 13:09:38 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Feb 11 13:09:38 2015] [notice] SSL FIPS mode disabled
[Wed Feb 11 13:09:38 2015] [warn] RSA server certificate CommonName (CN) `www.qnagios.com' does NOT match server name!?
[Wed Feb 11 13:09:38 2015] [notice] Digest: generating secret for digest authentication ...
[Wed Feb 11 13:09:38 2015] [notice] Digest: done
[Wed Feb 11 13:09:38 2015] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Wed Feb 11 13:09:38 2015] [notice] SSL FIPS mode disabled
[Wed Feb 11 13:09:38 2015] [warn] RSA server certificate CommonName (CN) `www.qnagios.com' does NOT match server name!?
[Wed Feb 11 13:09:38 2015] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[root@nagiosxi ~]#

root@nagiosxi ~]# tail -50 /var/log/mysqld.log
Version: '5.0.95' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150210 17:06:46 [Note] /usr/libexec/mysqld: Normal shutdown

150210 17:06:46 InnoDB: Starting shutdown...
150210 17:06:47 InnoDB: Shutdown completed; log sequence number 0 43675
150210 17:06:47 [Note] /usr/libexec/mysqld: Shutdown complete

150210 17:06:47 mysqld ended

150210 17:06:48 mysqld started
150210 17:06:48 InnoDB: Started; log sequence number 0 43675
150210 17:06:48 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150210 17:16:14 [Note] /usr/libexec/mysqld: Normal shutdown

150210 17:16:16 InnoDB: Starting shutdown...
150210 17:16:18 InnoDB: Shutdown completed; log sequence number 0 43675
150210 17:16:18 [Note] /usr/libexec/mysqld: Shutdown complete

150210 17:16:18 mysqld ended

150210 17:18:32 mysqld started
150210 17:18:33 InnoDB: Started; log sequence number 0 43675
150210 17:18:33 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150211 8:35:22 [Note] /usr/libexec/mysqld: Normal shutdown

150211 8:35:25 InnoDB: Starting shutdown...
150211 8:35:26 InnoDB: Shutdown completed; log sequence number 0 43675
150211 8:35:26 [Note] /usr/libexec/mysqld: Shutdown complete

150211 08:35:26 mysqld ended

150211 08:37:15 mysqld started
150211 8:37:15 InnoDB: Started; log sequence number 0 43675
150211 8:37:15 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150211 9:20:35 [Note] Found 11058674 of 8853181 rows when repairing './nagios/nagios_logentries'
150211 9:34:25 [Note] /usr/libexec/mysqld: Normal shutdown

150211 9:34:26 InnoDB: Starting shutdown...
150211 9:34:26 InnoDB: Shutdown completed; log sequence number 0 43675
150211 9:34:26 [Note] /usr/libexec/mysqld: Shutdown complete

150211 09:34:26 mysqld ended

150211 09:36:53 mysqld started
150211 9:36:54 InnoDB: Started; log sequence number 0 43675
150211 9:36:54 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
[root@nagiosxi ~]#

[abrist]

[Wed Feb 11 10:53:01 2015] [error] [client 127.0.0.1] PHP Warning: pg_pconnect() [<a href='function.pg-pconnect'>function.pg-pconnect</a>]: Unable to connect to PostgreSQL server: could not translate host name "localhost" to address: Temporary failure in name resolution in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 682

Can the nagios server ping "localhost?

Code: Select all

ping localhost