[HELP] Nagios DATABASE Backend connection error
Posted: Fri Jun 06, 2014 1:19 am
Hi, we are running Nagios XI 2012 R2.9 VMWare downloaded from Nagios!
We use offloading MySQL, but we have some problems at ~ 2:00. The Nagios XI stop working (blank screeen). Nagios lost connection with the DATABASE backend. I need to restart mysqld to nagios recovery to restore work. Here are some output commands before restart mysqld
Here is the /var/log/httpd/error_log (nagios machine)
Here is the /usr/local/nagios/var/nagios.log
Here is the VM-NGS04 (Mysql backend machine) mysql log
I just restart mysqld and the nagios xi interface start to work
Thanks for help
We use offloading MySQL, but we have some problems at ~ 2:00. The Nagios XI stop working (blank screeen). Nagios lost connection with the DATABASE backend. I need to restart mysqld to nagios recovery to restore work. Here are some output commands before restart mysqld
Code: Select all
[root@VM-NGS01 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
136G 22G 108G 17% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 485M 50M 410M 11% /boot
tmpfs 256M 24M 233M 10% /var/nagiosramdisk
//XX.XXX.XX.XX/backup
20G 9.3G 11G 47% /mnt/backup
Code: Select all
[root@VM-NGS01 ~]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
8931312 123111 8808201 2% /
tmpfs 1006755 1 1006754 1% /dev/shm
/dev/sda1 128016 44 127972 1% /boot
tmpfs 1006755 1393 1005362 1% /var/nagiosramdisk
//XX.XXX.XX.XX/backup
0 0 0 - /mnt/backup
Code: Select all
[root@VM-NGS01 ~]# service postgresql status
postmaster (pid 1491) is running...
Code: Select all
[Fri Jun 06 02:03:48 2014] [error] [client 10.246.35.16] PHP Warning: mysql_pconnect(): Can't connect to MySQL server on 'XX.XXX.XX.XX' (4) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383
[Fri Jun 06 02:03:49 2014] [error] [client 10.241.6.241] PHP Warning: mysql_pconnect(): Can't connect to MySQL server on 'XX.XXX.XX.XX' (4) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383, referer: http://nagios.XXX.com.br/nagiosxi/index.php
[Fri Jun 06 02:03:50 2014] [error] [client 10.241.6.241] PHP Warning: mysql_pconnect(): Can't connect to MySQL server on 'XX.XXX.XX.XX' (4) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383, referer: http://nagios.XXX.com.br/nagiosxi/index.php
[Fri Jun 06 02:03:52 2014] [error] [client 10.241.6.241] PHP Warning: mysql_pconnect(): Can't connect to MySQL server on 'XX.XXX.XX.XX' (4) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383, referer: http://nagios.teleperformance.com.br/nagiosxi/includes/components/xicore/status.php?show=hosts&search=DB24&noservices=1
[Fri Jun 06 02:03:53 2014] [error] [client 10.246.17.7] PHP Warning: mysql_pconnect(): Can't connect to MySQL server on 'XX.XXX.XX.XX' (4) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383, referer: http://nagios.teleperformance.com.br/nagiosxi/includes/components/xicore/status.php?show=services
Code: Select all
[1402031449] SERVICE ALERT: VM-NGS04;MySQL Uptime;CRITICAL;SOFT;3;CRITICAL - cannot connect to information_schema. Host 'vm-ngs01' is blocked because of many connection errors: unblock with 'mysqladmin flush-hosts'
[1402031449] SERVICE ALERT: VM-NGS04;MySQL InnoDB Buffer Pool Hit Rate;CRITICAL;SOFT;3;CRITICAL - cannot connect to information_schema. Host 'vm-ngs01' is blocked because of many connection errors: unblock with 'mysqladmin flush-hosts'
[1402031449] SERVICE ALERT: VM-APP16;Memory Usage;OK;HARD;1;Memory usage: total:7168.06 Mb - used: 3072.03 Mb (43%) - free: 4096.04 Mb (57%)
[1402031449] SERVICE ALERT: VM-NGS04;MySQL Table Cache Hit Rate;CRITICAL;SOFT;3;CRITICAL - cannot connect to information_schema. Host 'vm-ngs01' is blocked because of many connection errors: unblock with 'mysqladmin flush-hosts'
[1402031454] SERVICE ALERT: VM-TRN14;CPU Usage;CRITICAL;SOFT;3;CPU Load 100% (5 min average)
[1402031454] SERVICE ALERT: VM-NGS04;MySQL Open Connections;CRITICAL;SOFT;3;CRITICAL - cannot connect to information_schema. Host 'vm-ngs01' is blocked because of many connection errors: unblock with 'mysqladmin flush-hosts'
[1402031454] SERVICE ALERT: VM-NGS04;MySQL Thread Cache Hit Rate;CRITICAL;SOFT;3;CRITICAL - cannot connect to information_schema. Host 'vm-ngs01' is blocked because of many connection errors: unblock with 'mysqladmin flush-hosts'
[1402031454] SERVICE ALERT: VM-NGS04;MySQL Connection Time;CRITICAL;SOFT;3;CRITICAL - cannot connect to information_schema. Host 'vm-ngs01' is blocked because of many connection errors: unblock with 'mysqladmin flush-hosts'
Code: Select all
Version: '5.1.66' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
140509 1:59:57 [Note] /usr/libexec/mysqld: Normal shutdown
140509 2:00:26 [ERROR] /usr/libexec/mysqld: Sort aborted
140509 2:00:26 [ERROR] /usr/libexec/mysqld: Sort aborted
140509 2:00:26 [ERROR] /usr/libexec/mysqld: Sort aborted
140509 2:00:26 [ERROR] /usr/libexec/mysqld: Sort aborted
140509 2:00:26 [ERROR] /usr/libexec/mysqld: Sort aborted
140509 2:00:26 [ERROR] /usr/libexec/mysqld: Sort aborted
140509 2:00:26 [Note] Event Scheduler: Purging the queue. 0 events
140509 2:00:47 [Warning] /usr/libexec/mysqld: Forcing close of thread 2227927 user: 'nagiosql'
140509 2:01:27 InnoDB: Starting shutdown...
140509 2:02:03 InnoDB: Shutdown completed; log sequence number 0 44243
140509 2:02:06 [Note] /usr/libexec/mysqld: Shutdown complete
140509 02:02:26 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140509 02:11:54 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140509 2:11:54 InnoDB: Initializing buffer pool, size = 8.0M
140509 2:11:54 InnoDB: Completed initialization of buffer pool
140509 2:11:55 InnoDB: Started; log sequence number 0 44243
140509 2:11:56 [Note] Event Scheduler: Loaded 0 events
140509 2:11:56 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
140509 2:24:53 [Note] /usr/libexec/mysqld: Normal shutdown
140509 2:24:53 [Note] Event Scheduler: Purging the queue. 0 events
140509 2:24:55 InnoDB: Starting shutdown...
140509 2:24:56 InnoDB: Shutdown completed; log sequence number 0 44243
140509 2:24:56 [Note] /usr/libexec/mysqld: Shutdown complete
140509 02:24:56 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140509 02:25:59 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140509 2:25:59 InnoDB: Initializing buffer pool, size = 8.0M
140509 2:25:59 InnoDB: Completed initialization of buffer pool
140509 2:25:59 InnoDB: Started; log sequence number 0 44243
140509 2:25:59 [Note] Event Scheduler: Loaded 0 events
140509 2:25:59 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
140509 2:26:14 [Note] /usr/libexec/mysqld: Normal shutdown
140509 2:26:14 [Note] Event Scheduler: Purging the queue. 0 events
140509 2:26:16 InnoDB: Starting shutdown...
140509 2:26:19 InnoDB: Shutdown completed; log sequence number 0 44243
140509 2:26:19 [Note] /usr/libexec/mysqld: Shutdown complete
140509 02:26:19 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140509 02:26:20 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140509 2:26:20 InnoDB: Initializing buffer pool, size = 8.0M
140509 2:26:20 InnoDB: Completed initialization of buffer pool
140509 2:26:20 InnoDB: Started; log sequence number 0 44243
140509 2:26:20 [Note] Event Scheduler: Loaded 0 events
140509 2:26:20 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
140514 2:21:45 [Note] /usr/libexec/mysqld: Normal shutdown
140514 2:21:45 [Note] Event Scheduler: Purging the queue. 0 events
140514 2:21:45 InnoDB: Starting shutdown...
140514 2:21:51 InnoDB: Shutdown completed; log sequence number 0 44243
140514 2:21:51 [Note] /usr/libexec/mysqld: Shutdown complete
140514 02:21:52 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140514 02:21:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140514 2:21:56 InnoDB: Initializing buffer pool, size = 8.0M
140514 2:21:56 InnoDB: Completed initialization of buffer pool
140514 2:21:57 InnoDB: Started; log sequence number 0 44243
140514 2:21:57 [Note] Event Scheduler: Loaded 0 events
140514 2:21:57 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
140525 1:56:42 [Note] /usr/libexec/mysqld: Normal shutdown
140525 1:56:47 [Note] Event Scheduler: Purging the queue. 0 events
140525 1:58:31 InnoDB: Starting shutdown...
140525 1:59:48 InnoDB: Shutdown completed; log sequence number 0 44243
140525 1:59:48 [Note] /usr/libexec/mysqld: Shutdown complete
140525 01:59:49 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140525 02:46:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140525 2:46:30 InnoDB: Initializing buffer pool, size = 8.0M
140525 2:46:30 InnoDB: Completed initialization of buffer pool
140525 2:46:30 InnoDB: Started; log sequence number 0 44243
140525 2:46:30 [Note] Event Scheduler: Loaded 0 events
140525 2:46:30 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
140530 2:01:53 [Note] /usr/libexec/mysqld: Normal shutdown
140530 2:01:53 [Note] Event Scheduler: Purging the queue. 0 events
140530 2:01:55 [Warning] /usr/libexec/mysqld: Forcing close of thread 283977 user: 'nagios'
140530 2:01:55 [Warning] /usr/libexec/mysqld: Forcing close of thread 283852 user: 'nagios'
140530 2:01:55 [Warning] /usr/libexec/mysqld: Forcing close of thread 283842 user: 'nagios'
140530 2:01:55 [Warning] /usr/libexec/mysqld: Forcing close of thread 283818 user: 'nagiosql'
140530 2:01:55 [Warning] /usr/libexec/mysqld: Forcing close of thread 283797 user: 'nagios'
140530 2:01:55 [Warning] /usr/libexec/mysqld: Forcing close of thread 283620 user: 'nagiosql'
140530 2:02:00 InnoDB: Starting shutdown...
140530 2:02:06 InnoDB: Shutdown completed; log sequence number 0 44243
140530 2:02:06 [Note] /usr/libexec/mysqld: Shutdown complete
140530 02:02:07 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140530 02:05:04 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140530 2:05:04 InnoDB: Initializing buffer pool, size = 8.0M
140530 2:05:04 InnoDB: Completed initialization of buffer pool
140530 2:05:05 InnoDB: Started; log sequence number 0 44243
140530 2:05:05 [Note] Event Scheduler: Loaded 0 events
140530 2:05:05 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
140606 2:26:40 [Note] /usr/libexec/mysqld: Normal shutdown
140606 2:26:40 [Note] Event Scheduler: Purging the queue. 0 events
140606 2:26:40 InnoDB: Starting shutdown...
140606 2:26:43 InnoDB: Shutdown completed; log sequence number 0 44243
140606 2:26:43 [Note] /usr/libexec/mysqld: Shutdown complete
140606 02:26:43 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140606 02:26:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140606 2:26:45 InnoDB: Initializing buffer pool, size = 8.0M
140606 2:26:45 InnoDB: Completed initialization of buffer pool
140606 2:26:46 InnoDB: Started; log sequence number 0 44243
140606 2:26:46 [Note] Event Scheduler: Loaded 0 events
140606 2:26:46 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
Thanks for help