Increasingly frequent DB connection errors

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Increasingly frequent DB connection errors

Post by rkennedy »

Could you also run the second command I posted? It'll show us what all process is consuming the most CPU.

Code: Select all

ps -eo pcpu,args --sort=-%cpu
Former Nagios Employee
Top
FLCUISIT
Posts: 93
Joined: Mon Feb 01, 2010 12:09 pm

Re: Increasingly frequent DB connection errors

Post by FLCUISIT »

[root@nagios ~]# ps -eo pcpu,args --sort=-%cpu
%CPU COMMAND
7.1 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
4.7 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --
3.6 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
0.3 /usr/bin/ssh -l vi-admin 192.168.21.9 ~/box293_check_vmware.pl --timeout 90
0.2 /usr/bin/ssh -l vi-admin 192.168.21.9 ~/box293_check_vmware.pl --timeout 90
0.1 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
0.1 [kjournald]
0.1 /usr/bin/ssh -l vi-admin 192.168.21.9 ~/box293_check_vmware.pl --timeout 90
0.1 [pdflush]
0.1 /usr/bin/ssh -l vi-admin 192.168.21.9 ~/box293_check_vmware.pl --timeout 90
0.1 /usr/bin/ssh -l vi-admin 192.168.21.9 ~/box293_check_vmware.pl --timeout 90
0.0 /usr/bin/ssh -l vi-admin 192.168.21.9 ~/box293_check_vmware.pl --timeout 90
0.0 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0.0 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0.0 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0.0 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0.0 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0.0 /usr/bin/ssh -l vi-admin 192.168.21.9 ~/box293_check_vmware.pl --timeout 90
0.0 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0.0 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0.0 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0.0 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0.0 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0.0 /usr/bin/ssh -l vi-admin 192.168.21.9 ~/box293_check_vmware.pl --timeout 90
0.0 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
0.0 hald-addon-storage: polling /dev/hdc
0.0 init [3]
0.0 [migration/0]
0.0 /usr/sbin/httpd
0.0 /usr/sbin/httpd
0.0 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data
0.0 [migration/5]
0.0 [migration/4]
0.0 [migration/3]
0.0 /usr/sbin/httpd
0.0 [migration/1]
0.0 [migration/2]
0.0 [migration/6]
0.0 /usr/sbin/vmtoolsd
0.0 hald
0.0 /usr/sbin/httpd
0.0 syslogd -m 0
0.0 auditd
0.0 /usr/sbin/httpd
0.0 /sbin/udevd -d
0.0 pcscd
0.0 /sbin/audispd
0.0 postgres: logger process
0.0 /usr/bin/python -tt /usr/sbin/yum-updatesd
0.0 irqbalance
0.0 crond
0.0 sshd: root@pts/0
0.0 postgres: stats buffer process
0.0 -bash
0.0 python /usr/share/ajaxterm/ajaxterm.py --daemon --port=8022 --uid=ajaxterm
0.0 [kauditd]
0.0 [kblockd/5]
0.0 postgres: writer process
0.0 postgres: stats collector process
0.0 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mys
0.0 klogd -x
0.0 dbus-daemon --system
0.0 xinetd -stayalive -pidfile /var/run/xinetd.pid
0.0 [vmmemctl]
0.0 automount
0.0 /usr/sbin/sshd
0.0 cupsd
0.0 sendmail: accepting connections
0.0 xfs -droppriv -daemon
0.0 /usr/sbin/httpd
0.0 /usr/sbin/httpd
0.0 avahi-daemon: running [nagios.local]
0.0 /usr/libexec/gam_server
0.0 [events/4]
0.0 [kblockd/0]
0.0 [ksoftirqd/0]
0.0 [watchdog/0]
0.0 [ksoftirqd/1]
0.0 [watchdog/1]
0.0 [ksoftirqd/2]
0.0 [watchdog/2]
0.0 [ksoftirqd/3]
0.0 [watchdog/3]
0.0 [ksoftirqd/4]
0.0 [watchdog/4]
0.0 [ksoftirqd/5]
0.0 [watchdog/5]
0.0 [ksoftirqd/6]
0.0 [watchdog/6]
0.0 [events/0]
0.0 [events/1]
0.0 [events/2]
0.0 [events/3]
0.0 [events/5]
0.0 [events/6]
0.0 [khelper]
0.0 [kthread]
0.0 [kblockd/1]
0.0 [kblockd/2]
0.0 [kblockd/3]
0.0 [kblockd/4]
0.0 [kblockd/6]
0.0 [kacpid]
0.0 [cqueue/0]
0.0 [cqueue/1]
0.0 [cqueue/2]
0.0 [cqueue/3]
0.0 [cqueue/4]
0.0 [cqueue/5]
0.0 [cqueue/6]
0.0 [khubd]
0.0 [kseriod]
0.0 [pdflush]
0.0 [kswapd0]
0.0 [aio/0]
0.0 [aio/1]
0.0 [aio/2]
0.0 [aio/3]
0.0 [aio/4]
0.0 [aio/5]
0.0 [aio/6]
0.0 [kpsmoused]
0.0 [mpt_poll_0]
0.0 [scsi_eh_0]
0.0 [ata/0]
0.0 [ata/1]
0.0 [ata/2]
0.0 [ata/3]
0.0 [ata/4]
0.0 [ata/5]
0.0 [ata/6]
0.0 [ata_aux]
0.0 [kstriped]
0.0 [ksnapd]
0.0 [kmpathd/0]
0.0 [kmpathd/1]
0.0 [kmpathd/2]
0.0 [kmpathd/3]
0.0 [kmpathd/4]
0.0 [kmpathd/5]
0.0 [kmpathd/6]
0.0 [kmpath_handlerd]
0.0 [kjournald]
0.0 portmap
0.0 [rpciod/0]
0.0 [rpciod/1]
0.0 [rpciod/2]
0.0 [rpciod/3]
0.0 [rpciod/4]
0.0 [rpciod/5]
0.0 [rpciod/6]
0.0 rpc.statd
0.0 rpc.idmapd
0.0 /usr/sbin/hcid
0.0 /usr/sbin/sdpd
0.0 [krfcommd]
0.0 /usr/sbin/acpid
0.0 hald-runner
0.0 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
0.0 hald-addon-keyboard: listening on /dev/input/event0
0.0 /usr/bin/hidd --server
0.0 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
0.0 gpm -m /dev/input/mice -t exps2
0.0 /usr/sbin/atd
0.0 /usr/sbin/httpd
0.0 /usr/sbin/httpd
0.0 avahi-daemon: chroot helper
0.0 /usr/local/nagios/libexec/check_by_ssh -E 1 -t 90 -l vi-admin -H 192.168.21
0.0 /usr/sbin/smartd -q never
0.0 /sbin/mingetty tty1
0.0 /sbin/mingetty tty2
0.0 /sbin/mingetty tty3
0.0 /sbin/mingetty tty4
0.0 /sbin/mingetty tty5
0.0 /sbin/mingetty tty6
0.0 /usr/local/nagios/libexec/check_by_ssh -E 1 -t 90 -l vi-admin -H 192.168.21
0.0 /usr/local/nagios/libexec/check_by_ssh -E 1 -t 90 -l vi-admin -H 192.168.21
0.0 /usr/local/nagios/libexec/check_by_ssh -E 1 -t 90 -l vi-admin -H 192.168.21
0.0 /usr/local/nagios/libexec/check_by_ssh -E 1 -t 90 -l vi-admin -H 192.168.21
0.0 /usr/local/nagios/libexec/check_by_ssh -E 1 -t 90 -l vi-admin -H 192.168.21
0.0 /usr/local/nagios/libexec/check_by_ssh -E 1 -t 90 -l vi-admin -H 192.168.21
0.0 /usr/local/nagios/libexec/check_by_ssh -E 1 -t 90 -l vi-admin -H 192.168.21
0.0 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 192.168.3.
0.0 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/192.168.20
0.0 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/192.168.3.
0.0 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/192.168.45
0.0 ps -eo pcpu,args --sort=-%cpu
0.0 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/192.168.5.
0.0 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
0.0 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Increasingly frequent DB connection errors

Post by rkennedy »

It looks like the issue is definitely with the SQL load and not something else bugging it. Could you PM over a profile for us to look at? (Admin -> System Profile -> Download Profile)
Former Nagios Employee
Top
FLCUISIT
Posts: 93
Joined: Mon Feb 01, 2010 12:09 pm

Re: Increasingly frequent DB connection errors

Post by FLCUISIT »

I would...I cannot even log into the web site to get it. That is the biggest problem. The errors I have are showing on the login screen.

Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh
Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh
SQL: SQL Error [nagiosxi] : Database connection failed SQL: SQL Error [nagiosxi] : Database connection failed SQL: SQL Error [nagiosxi] : Database connection failedMessage: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Increasingly frequent DB connection errors

Post by rkennedy »

Please post the following files then -

Code: Select all

/var/log/httpd/ssl_error_log
/var/log/httpd/error_log
/var/log/mysql/mysqld.log
/usr/local/nagios/var/*.log (nagios.log, npcd.log, perfdata.log)
Hopefully one of these will point us in the right direction.
Former Nagios Employee
Top
FLCUISIT
Posts: 93
Joined: Mon Feb 01, 2010 12:09 pm

Re: Increasingly frequent DB connection errors

Post by FLCUISIT »

[root@nagios ~]# more /var/log/httpd/ssl_error_log
[Sun Sep 04 04:02:55 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Sun Sep 04 04:02:55 2016] [warn] RSA server certificate CommonName (CN) `localhost.localdomain' does NOT match server name!?
[Mon Sep 05 13:10:19 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Sep 05 13:10:19 2016] [warn] RSA server certificate CommonName (CN) `localhost.localdomain' does NOT match server name!?
[Mon Sep 05 13:10:20 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Sep 05 13:10:20 2016] [warn] RSA server certificate CommonName (CN) `localhost.localdomain' does NOT match server name!?
[Mon Sep 05 13:16:37 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Sep 05 13:16:37 2016] [warn] RSA server certificate CommonName (CN) `localhost.localdomain' does NOT match server name!?
[Mon Sep 05 13:16:38 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Sep 05 13:16:38 2016] [warn] RSA server certificate CommonName (CN) `localhost.localdomain' does NOT match server name!?
[Mon Sep 05 13:21:10 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Sep 05 13:21:10 2016] [warn] RSA server certificate CommonName (CN) `localhost.localdomain' does NOT match server name!?
[Mon Sep 05 13:21:12 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Sep 05 13:21:12 2016] [warn] RSA server certificate CommonName (CN) `localhost.localdomain' does NOT match server name!?
[Tue Sep 06 15:10:51 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Tue Sep 06 15:10:51 2016] [warn] RSA server certificate CommonName (CN) `localhost.localdomain' does NOT match server name!?
[Tue Sep 06 15:10:54 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Tue Sep 06 15:10:54 2016] [warn] RSA server certificate CommonName (CN) `localhost.localdomain' does NOT match server name!?
[Tue Sep 06 15:43:15 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Tue Sep 06 15:43:15 2016] [warn] RSA server certificate CommonName (CN) `localhost.localdomain' does NOT match server name!?
[Tue Sep 06 15:43:19 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Tue Sep 06 15:43:19 2016] [warn] RSA server certificate CommonName (CN) `localhost.localdomain' does NOT match server name!?

mysql.log



150406 11:29:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150406 11:29:02 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150406 11:29:02 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150406 11:29:02 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150406 11:29:02 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150507 14:14:51 [Note] /usr/libexec/mysqld: Normal shutdown

150507 14:14:51 InnoDB: Starting shutdown...
150507 14:14:52 InnoDB: Shutdown completed; log sequence number 0 43655
150507 14:14:52 [Note] /usr/libexec/mysqld: Shutdown complete

150507 14:14:52 mysqld ended

150507 14:19:26 mysqld started
150507 14:19:26 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
150507 14:19:26 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
150507 14:19:27 InnoDB: Started; log sequence number 0 43655
150507 14:19:27 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150923 08:24:09 mysqld started
150923 8:24:09 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
150923 8:24:09 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
150923 8:24:09 InnoDB: Started; log sequence number 0 43655
150923 8:24:09 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150923 15:26:57 [Note] /usr/libexec/mysqld: Normal shutdown

150923 15:26:59 InnoDB: Starting shutdown...
150923 15:27:00 InnoDB: Shutdown completed; log sequence number 0 43655
150923 15:27:00 [Note] /usr/libexec/mysqld: Shutdown complete

150923 15:27:00 mysqld ended

150923 15:31:15 mysqld started
150923 15:31:16 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
150923 15:31:16 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
150923 15:31:17 InnoDB: Started; log sequence number 0 43655
150923 15:31:18 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160425 10:10:21 [Note] /usr/libexec/mysqld: Normal shutdown

160425 10:10:21 InnoDB: Starting shutdown...
160425 10:10:22 InnoDB: Shutdown completed; log sequence number 0 43655
160425 10:10:22 [Note] /usr/libexec/mysqld: Shutdown complete

160425 10:10:22 mysqld ended

160425 10:16:18 mysqld started
160425 10:16:18 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160425 10:16:18 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160425 10:16:18 InnoDB: Started; log sequence number 0 43655
160425 10:16:19 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160827 17:48:37 [Note] /usr/libexec/mysqld: Normal shutdown

160827 17:48:37 InnoDB: Starting shutdown...
160827 17:48:38 InnoDB: Shutdown completed; log sequence number 0 43655
160827 17:48:38 [Note] /usr/libexec/mysqld: Shutdown complete

160827 17:48:38 mysqld ended

160828 00:16:24 mysqld started
160828 0:16:25 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 0:16:25 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 0:16:28 InnoDB: Started; log sequence number 0 43655
160828 0:16:29 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160828 20:09:09 [Note] /usr/libexec/mysqld: Normal shutdown

160828 20:09:10 InnoDB: Starting shutdown...
160828 20:09:12 InnoDB: Shutdown completed; log sequence number 0 43655
160828 20:09:12 [Note] /usr/libexec/mysqld: Shutdown complete

160828 20:09:12 mysqld ended

160828 20:07:04 mysqld started
160828 20:07:06 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 20:07:06 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 20:07:08 InnoDB: Started; log sequence number 0 43655
160828 20:07:10 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160828 20:14:00 [Note] /usr/libexec/mysqld: Normal shutdown

160828 20:14:02 InnoDB: Starting shutdown...
160828 20:14:04 InnoDB: Shutdown completed; log sequence number 0 43655
160828 20:14:04 [Note] /usr/libexec/mysqld: Shutdown complete

160828 20:14:04 mysqld ended

160828 20:28:32 mysqld started
160828 20:28:32 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 20:28:32 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 20:28:38 InnoDB: Started; log sequence number 0 43655
160828 20:28:38 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160828 21:30:53 [Note] /usr/libexec/mysqld: Normal shutdown

160828 21:30:55 InnoDB: Starting shutdown...
160828 21:30:59 InnoDB: Shutdown completed; log sequence number 0 43655
160828 21:30:59 [Note] /usr/libexec/mysqld: Shutdown complete

160828 21:30:59 mysqld ended

160828 21:52:17 mysqld started
160828 21:52:20 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 21:52:20 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 21:52:28 InnoDB: Started; log sequence number 0 43655
160828 21:52:38 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160828 21:57:59 [Note] /usr/libexec/mysqld: Normal shutdown

160828 21:58:01 InnoDB: Starting shutdown...
160828 21:58:05 InnoDB: Shutdown completed; log sequence number 0 43655
160828 21:58:05 [Note] /usr/libexec/mysqld: Shutdown complete

160828 21:58:05 mysqld ended

160828 22:12:26 mysqld started
160828 22:12:26 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 22:12:26 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 22:12:33 InnoDB: Started; log sequence number 0 43655
160828 22:12:33 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160828 22:12:34 [Note] /usr/libexec/mysqld: Normal shutdown

160828 22:12:34 InnoDB: Starting shutdown...
160828 22:12:38 InnoDB: Shutdown completed; log sequence number 0 43655
160828 22:12:38 [Note] /usr/libexec/mysqld: Shutdown complete

160828 22:12:38 mysqld ended

160828 22:13:00 mysqld started
160828 22:13:00 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 22:13:00 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 22:13:08 InnoDB: Started; log sequence number 0 43655
160828 22:13:08 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160828 22:15:21 [Note] /usr/libexec/mysqld: Normal shutdown

160828 22:15:23 InnoDB: Starting shutdown...
160828 22:15:29 InnoDB: Shutdown completed; log sequence number 0 43655
160828 22:15:29 [Note] /usr/libexec/mysqld: Shutdown complete

160828 22:15:29 mysqld ended

160828 22:16:13 mysqld started
160828 22:16:13 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 22:16:13 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 22:16:46 mysqld started
160828 22:16:46 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 22:16:46 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
160828 22:16:47 InnoDB: Retrying to lock the first data file
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
160828 22:18:27 InnoDB: Unable to open the first data file
InnoDB: Error in opening ./ibdata1
160828 22:18:27 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/ ... codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!
160828 22:18:27 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160828 22:20:43 InnoDB: Started; log sequence number 0 43655
160828 22:20:43 [ERROR] Can't start server: Bind on TCP/IP port: Address already in use
160828 22:20:43 [ERROR] Do you already have another mysqld server running on port: 3306 ?
160828 22:20:43 [ERROR] Aborting

160828 22:20:43 InnoDB: Starting shutdown...
160828 22:20:46 InnoDB: Shutdown completed; log sequence number 0 43655
160828 22:20:46 [Note] /usr/libexec/mysqld: Shutdown complete

160828 22:20:46 mysqld ended

160828 22:23:14 [Note] /usr/libexec/mysqld: Normal shutdown

160828 22:23:17 [Note] /usr/libexec/mysqld: Shutdown complete

160828 22:23:17 mysqld ended

160828 22:26:25 mysqld started
160828 22:26:27 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 22:26:27 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160828 22:26:30 InnoDB: Started; log sequence number 0 43655
160828 22:26:32 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160905 13:06:53 [Note] /usr/libexec/mysqld: Normal shutdown

160905 13:06:53 InnoDB: Starting shutdown...
160905 13:06:54 InnoDB: Shutdown completed; log sequence number 0 43655
160905 13:06:54 [Note] /usr/libexec/mysqld: Shutdown complete

160905 13:06:54 mysqld ended

160905 13:07:24 mysqld started
160905 13:07:25 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:07:25 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:07:25 InnoDB: Started; log sequence number 0 43655
160905 13:07:25 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160905 13:07:27 [Note] /usr/libexec/mysqld: Normal shutdown

160905 13:07:27 InnoDB: Starting shutdown...
160905 13:07:27 InnoDB: Shutdown completed; log sequence number 0 43655
160905 13:07:27 [Note] /usr/libexec/mysqld: Shutdown complete

160905 13:07:27 mysqld ended

160905 13:07:28 mysqld started
160905 13:07:28 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:07:28 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:07:28 InnoDB: Started; log sequence number 0 43655
160905 13:07:28 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160905 13:08:56 [Note] /usr/libexec/mysqld: Normal shutdown

160905 13:08:58 InnoDB: Starting shutdown...
160905 13:08:59 InnoDB: Shutdown completed; log sequence number 0 43655
160905 13:08:59 [Note] /usr/libexec/mysqld: Shutdown complete

160905 13:08:59 mysqld ended

160905 13:10:14 mysqld started
160905 13:10:14 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:10:14 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:10:14 InnoDB: Started; log sequence number 0 43655
160905 13:10:15 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160905 13:13:04 [Note] /usr/libexec/mysqld: Normal shutdown

160905 13:13:04 InnoDB: Starting shutdown...
160905 13:13:05 InnoDB: Shutdown completed; log sequence number 0 43655
160905 13:13:05 [Note] /usr/libexec/mysqld: Shutdown complete

160905 13:13:07 mysqld ended

160905 13:14:39 mysqld started
160905 13:14:39 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:14:39 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:14:39 InnoDB: Started; log sequence number 0 43655
160905 13:14:39 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160905 13:14:40 [Note] /usr/libexec/mysqld: Normal shutdown

160905 13:14:42 InnoDB: Starting shutdown...
160905 13:14:43 InnoDB: Shutdown completed; log sequence number 0 43655
160905 13:14:43 [Note] /usr/libexec/mysqld: Shutdown complete

160905 13:14:43 mysqld ended

160905 13:14:45 mysqld started
160905 13:14:45 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:14:45 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:14:45 InnoDB: Started; log sequence number 0 43655
160905 13:14:45 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160905 13:17:56 [Note] /usr/libexec/mysqld: Normal shutdown

160905 13:17:56 InnoDB: Starting shutdown...
160905 13:17:57 InnoDB: Shutdown completed; log sequence number 0 43655
160905 13:17:57 [Note] /usr/libexec/mysqld: Shutdown complete

160905 13:17:57 mysqld ended

160905 13:18:28 mysqld started
160905 13:18:28 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:18:28 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:18:29 InnoDB: Started; log sequence number 0 43655
160905 13:18:29 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160905 13:18:30 [Note] /usr/libexec/mysqld: Normal shutdown

160905 13:18:30 InnoDB: Starting shutdown...
160905 13:18:30 InnoDB: Shutdown completed; log sequence number 0 43655
160905 13:18:30 [Note] /usr/libexec/mysqld: Shutdown complete

160905 13:18:30 mysqld ended

160905 13:18:31 mysqld started
160905 13:18:31 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:18:31 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:18:31 InnoDB: Started; log sequence number 0 43655
160905 13:18:31 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160905 13:19:22 [Note] /usr/libexec/mysqld: Normal shutdown

160905 13:19:24 InnoDB: Starting shutdown...
160905 13:19:25 InnoDB: Shutdown completed; log sequence number 0 43655
160905 13:19:25 [Note] /usr/libexec/mysqld: Shutdown complete

160905 13:19:25 mysqld ended

160905 13:21:05 mysqld started
160905 13:21:05 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:21:05 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:21:06 InnoDB: Started; log sequence number 0 43655
160905 13:21:06 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160905 13:40:31 [Note] /usr/libexec/mysqld: Normal shutdown

160905 13:40:33 InnoDB: Starting shutdown...
160905 13:40:33 InnoDB: Shutdown completed; log sequence number 0 43655
160905 13:40:33 [Note] /usr/libexec/mysqld: Shutdown complete

160905 13:40:33 mysqld ended

160905 13:42:20 mysqld started
160905 13:42:20 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:42:20 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:42:20 InnoDB: Started; log sequence number 0 43655
160905 13:42:20 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160905 13:42:21 [Note] /usr/libexec/mysqld: Normal shutdown

160905 13:42:21 InnoDB: Starting shutdown...
160905 13:42:22 InnoDB: Shutdown completed; log sequence number 0 43655
160905 13:42:22 [Note] /usr/libexec/mysqld: Shutdown complete

160905 13:42:22 mysqld ended

160905 13:42:24 mysqld started
160905 13:42:24 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:42:24 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160905 13:42:24 InnoDB: Started; log sequence number 0 43655
160905 13:42:24 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160906 15:05:17 [Note] /usr/libexec/mysqld: Normal shutdown

160906 15:05:19 InnoDB: Starting shutdown...
160906 15:05:21 InnoDB: Shutdown completed; log sequence number 0 43655
160906 15:05:21 [Note] /usr/libexec/mysqld: Shutdown complete

160906 15:05:21 mysqld ended

160906 15:06:06 mysqld started
160906 15:06:06 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 15:06:06 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 15:06:07 InnoDB: Started; log sequence number 0 43655
160906 15:06:07 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
A mysqld process already exists at Tue Sep 6 15:06:36 EDT 2016
160906 15:07:04 [Note] /usr/libexec/mysqld: Normal shutdown

160906 15:07:06 InnoDB: Starting shutdown...
160906 15:07:06 InnoDB: Shutdown completed; log sequence number 0 43655
160906 15:07:06 [Note] /usr/libexec/mysqld: Shutdown complete

160906 15:07:06 mysqld ended

160906 15:10:45 mysqld started
160906 15:10:45 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 15:10:45 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 15:10:46 InnoDB: Started; log sequence number 0 43655
160906 15:10:47 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160906 15:18:14 [Note] /usr/libexec/mysqld: Normal shutdown

160906 15:18:16 InnoDB: Starting shutdown...
160906 15:18:16 InnoDB: Shutdown completed; log sequence number 0 43655
160906 15:18:16 [Note] /usr/libexec/mysqld: Shutdown complete

160906 15:18:16 mysqld ended

160906 15:23:04 mysqld started
160906 15:23:04 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 15:23:04 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 15:23:04 InnoDB: Started; log sequence number 0 43655
160906 15:23:04 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160906 15:36:55 [Note] /usr/libexec/mysqld: Normal shutdown

160906 15:36:57 InnoDB: Starting shutdown...
160906 15:36:57 InnoDB: Shutdown completed; log sequence number 0 43655
160906 15:36:57 [Note] /usr/libexec/mysqld: Shutdown complete

160906 15:36:57 mysqld ended

160906 15:43:08 mysqld started
160906 15:43:08 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 15:43:08 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 15:43:09 InnoDB: Started; log sequence number 0 43655
160906 15:43:09 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160906 16:43:05 [Note] /usr/libexec/mysqld: Normal shutdown

160906 16:43:07 InnoDB: Starting shutdown...
160906 16:43:09 InnoDB: Shutdown completed; log sequence number 0 43655
160906 16:43:09 [Note] /usr/libexec/mysqld: Shutdown complete

160906 16:43:09 mysqld ended

160906 16:43:11 mysqld started
160906 16:43:11 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 16:43:11 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 16:43:11 InnoDB: Started; log sequence number 0 43655
160906 16:43:11 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
A mysqld process already exists at Tue Sep 6 16:43:24 EDT 2016
160906 16:43:51 [Note] /usr/libexec/mysqld: Normal shutdown

160906 16:43:53 InnoDB: Starting shutdown...
160906 16:43:54 InnoDB: Shutdown completed; log sequence number 0 43655
160906 16:43:54 [Note] /usr/libexec/mysqld: Shutdown complete

160906 16:43:54 mysqld ended

160906 16:45:56 mysqld started
160906 16:45:56 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 16:45:56 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 16:45:56 InnoDB: Started; log sequence number 0 43655
160906 16:45:56 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160906 16:45:57 [Note] /usr/libexec/mysqld: Normal shutdown

160906 16:45:59 InnoDB: Starting shutdown...
160906 16:45:59 InnoDB: Shutdown completed; log sequence number 0 43655
160906 16:45:59 [Note] /usr/libexec/mysqld: Shutdown complete

160906 16:45:59 mysqld ended

160906 16:46:00 mysqld started
160906 16:46:00 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 16:46:00 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
160906 16:46:00 InnoDB: Started; log sequence number 0 43655
160906 16:46:01 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

part of error log:


[Sun Sep 04 04:02:55 2016] [notice] Digest: generating secret for digest authentication ...
[Sun Sep 04 04:02:55 2016] [notice] Digest: done
[Sun Sep 04 04:02:55 2016] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Sun Sep 04 04:02:55 2016] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Sep 04 04:03:12 2016] [error] [client 192.168.21.46] PHP Warning: pg_pconnect() [<a href='function.pg-pconnect'>function.pg-pconnect</a>]: Unable to connect to PostgreSQL server: FATAL: database is not accepting commands to avoid wraparound data loss in database "postgres"\nHINT: Stop the postmaster and use a standalone backend to vacuum database "postgres". in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 682, referer: http://192.168.21.39/nagiosxi/
Top
FLCUISIT
Posts: 93
Joined: Mon Feb 01, 2010 12:09 pm

Re: Increasingly frequent DB connection errors

Post by FLCUISIT »

We were able to get the server back to operation by the use of the information within the following link:

https://support.nagios.com/kb/print.php?id=25

The issue was within the postgres database connection.
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Increasingly frequent DB connection errors

Post by rkennedy »

Awesome! We were out of the office when your initial responses came in, but glad to see it resolved.

Are we good to close this thread out?
Former Nagios Employee
Top
Locked

Return to “Nagios XI”