[HELP] Nagios DATABASE Backend connection error

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
gfesj69
Posts: 59
Joined: Fri Jan 03, 2014 10:04 am
Location: Brazil

[HELP] Nagios DATABASE Backend connection error

Post by gfesj69 »

Hi, we are running Nagios XI 2012 R2.9 VMWare downloaded from Nagios!

We use offloading MySQL, but we have some problems at ~ 2:00. The Nagios XI stop working (blank screeen). Nagios lost connection with the DATABASE backend. I need to restart mysqld to nagios recovery to restore work. Here are some output commands before restart mysqld

Code: Select all

[root@VM-NGS01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      136G   22G  108G  17% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/sda1             485M   50M  410M  11% /boot
tmpfs                 256M   24M  233M  10% /var/nagiosramdisk
//XX.XXX.XX.XX/backup
                       20G  9.3G   11G  47% /mnt/backup

Code: Select all

[root@VM-NGS01 ~]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
                     8931312  123111 8808201    2% /
tmpfs                1006755       1 1006754    1% /dev/shm
/dev/sda1             128016      44  127972    1% /boot
tmpfs                1006755    1393 1005362    1% /var/nagiosramdisk
//XX.XXX.XX.XX/backup
                           0       0       0    -  /mnt/backup

Code: Select all

[root@VM-NGS01 ~]# service postgresql status
postmaster (pid  1491) is running...
Here is the /var/log/httpd/error_log (nagios machine)

Code: Select all

[Fri Jun 06 02:03:48 2014] [error] [client 10.246.35.16] PHP Warning:  mysql_pconnect(): Can't connect to MySQL server on 'XX.XXX.XX.XX' (4) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383
[Fri Jun 06 02:03:49 2014] [error] [client 10.241.6.241] PHP Warning:  mysql_pconnect(): Can't connect to MySQL server on 'XX.XXX.XX.XX' (4) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383, referer: http://nagios.XXX.com.br/nagiosxi/index.php
[Fri Jun 06 02:03:50 2014] [error] [client 10.241.6.241] PHP Warning:  mysql_pconnect(): Can't connect to MySQL server on 'XX.XXX.XX.XX' (4) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383, referer: http://nagios.XXX.com.br/nagiosxi/index.php
[Fri Jun 06 02:03:52 2014] [error] [client 10.241.6.241] PHP Warning:  mysql_pconnect(): Can't connect to MySQL server on 'XX.XXX.XX.XX' (4) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383, referer: http://nagios.teleperformance.com.br/nagiosxi/includes/components/xicore/status.php?show=hosts&search=DB24&noservices=1
[Fri Jun 06 02:03:53 2014] [error] [client 10.246.17.7] PHP Warning:  mysql_pconnect(): Can't connect to MySQL server on 'XX.XXX.XX.XX' (4) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383, referer: http://nagios.teleperformance.com.br/nagiosxi/includes/components/xicore/status.php?show=services
Here is the /usr/local/nagios/var/nagios.log

Code: Select all

[1402031449] SERVICE ALERT: VM-NGS04;MySQL Uptime;CRITICAL;SOFT;3;CRITICAL - cannot connect to information_schema. Host 'vm-ngs01' is blocked because of many connection errors: unblock with 'mysqladmin flush-hosts'
[1402031449] SERVICE ALERT: VM-NGS04;MySQL InnoDB Buffer Pool Hit Rate;CRITICAL;SOFT;3;CRITICAL - cannot connect to information_schema. Host 'vm-ngs01' is blocked because of many connection errors: unblock with 'mysqladmin flush-hosts'
[1402031449] SERVICE ALERT: VM-APP16;Memory Usage;OK;HARD;1;Memory usage: total:7168.06 Mb - used: 3072.03 Mb (43%) - free: 4096.04 Mb (57%)
[1402031449] SERVICE ALERT: VM-NGS04;MySQL Table Cache Hit Rate;CRITICAL;SOFT;3;CRITICAL - cannot connect to information_schema. Host 'vm-ngs01' is blocked because of many connection errors: unblock with 'mysqladmin flush-hosts'
[1402031454] SERVICE ALERT: VM-TRN14;CPU Usage;CRITICAL;SOFT;3;CPU Load 100% (5 min average)
[1402031454] SERVICE ALERT: VM-NGS04;MySQL Open Connections;CRITICAL;SOFT;3;CRITICAL - cannot connect to information_schema. Host 'vm-ngs01' is blocked because of many connection errors: unblock with 'mysqladmin flush-hosts'
[1402031454] SERVICE ALERT: VM-NGS04;MySQL Thread Cache Hit Rate;CRITICAL;SOFT;3;CRITICAL - cannot connect to information_schema. Host 'vm-ngs01' is blocked because of many connection errors: unblock with 'mysqladmin flush-hosts'
[1402031454] SERVICE ALERT: VM-NGS04;MySQL Connection Time;CRITICAL;SOFT;3;CRITICAL - cannot connect to information_schema. Host 'vm-ngs01' is blocked because of many connection errors: unblock with 'mysqladmin flush-hosts'
Here is the VM-NGS04 (Mysql backend machine) mysql log

Code: Select all

Version: '5.1.66'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140509  1:59:57 [Note] /usr/libexec/mysqld: Normal shutdown

140509  2:00:26 [ERROR] /usr/libexec/mysqld: Sort aborted
140509  2:00:26 [ERROR] /usr/libexec/mysqld: Sort aborted
140509  2:00:26 [ERROR] /usr/libexec/mysqld: Sort aborted
140509  2:00:26 [ERROR] /usr/libexec/mysqld: Sort aborted
140509  2:00:26 [ERROR] /usr/libexec/mysqld: Sort aborted
140509  2:00:26 [ERROR] /usr/libexec/mysqld: Sort aborted
140509  2:00:26 [Note] Event Scheduler: Purging the queue. 0 events
140509  2:00:47 [Warning] /usr/libexec/mysqld: Forcing close of thread 2227927  user: 'nagiosql'

140509  2:01:27  InnoDB: Starting shutdown...
140509  2:02:03  InnoDB: Shutdown completed; log sequence number 0 44243
140509  2:02:06 [Note] /usr/libexec/mysqld: Shutdown complete

140509 02:02:26 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140509 02:11:54 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140509  2:11:54  InnoDB: Initializing buffer pool, size = 8.0M
140509  2:11:54  InnoDB: Completed initialization of buffer pool
140509  2:11:55  InnoDB: Started; log sequence number 0 44243
140509  2:11:56 [Note] Event Scheduler: Loaded 0 events
140509  2:11:56 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140509  2:24:53 [Note] /usr/libexec/mysqld: Normal shutdown

140509  2:24:53 [Note] Event Scheduler: Purging the queue. 0 events
140509  2:24:55  InnoDB: Starting shutdown...
140509  2:24:56  InnoDB: Shutdown completed; log sequence number 0 44243
140509  2:24:56 [Note] /usr/libexec/mysqld: Shutdown complete

140509 02:24:56 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140509 02:25:59 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140509  2:25:59  InnoDB: Initializing buffer pool, size = 8.0M
140509  2:25:59  InnoDB: Completed initialization of buffer pool
140509  2:25:59  InnoDB: Started; log sequence number 0 44243
140509  2:25:59 [Note] Event Scheduler: Loaded 0 events
140509  2:25:59 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140509  2:26:14 [Note] /usr/libexec/mysqld: Normal shutdown

140509  2:26:14 [Note] Event Scheduler: Purging the queue. 0 events
140509  2:26:16  InnoDB: Starting shutdown...
140509  2:26:19  InnoDB: Shutdown completed; log sequence number 0 44243
140509  2:26:19 [Note] /usr/libexec/mysqld: Shutdown complete

140509 02:26:19 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140509 02:26:20 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140509  2:26:20  InnoDB: Initializing buffer pool, size = 8.0M
140509  2:26:20  InnoDB: Completed initialization of buffer pool
140509  2:26:20  InnoDB: Started; log sequence number 0 44243
140509  2:26:20 [Note] Event Scheduler: Loaded 0 events
140509  2:26:20 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140514  2:21:45 [Note] /usr/libexec/mysqld: Normal shutdown

140514  2:21:45 [Note] Event Scheduler: Purging the queue. 0 events
140514  2:21:45  InnoDB: Starting shutdown...
140514  2:21:51  InnoDB: Shutdown completed; log sequence number 0 44243
140514  2:21:51 [Note] /usr/libexec/mysqld: Shutdown complete

140514 02:21:52 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140514 02:21:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140514  2:21:56  InnoDB: Initializing buffer pool, size = 8.0M
140514  2:21:56  InnoDB: Completed initialization of buffer pool
140514  2:21:57  InnoDB: Started; log sequence number 0 44243
140514  2:21:57 [Note] Event Scheduler: Loaded 0 events
140514  2:21:57 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140525  1:56:42 [Note] /usr/libexec/mysqld: Normal shutdown

140525  1:56:47 [Note] Event Scheduler: Purging the queue. 0 events
140525  1:58:31  InnoDB: Starting shutdown...
140525  1:59:48  InnoDB: Shutdown completed; log sequence number 0 44243
140525  1:59:48 [Note] /usr/libexec/mysqld: Shutdown complete

140525 01:59:49 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140525 02:46:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140525  2:46:30  InnoDB: Initializing buffer pool, size = 8.0M
140525  2:46:30  InnoDB: Completed initialization of buffer pool
140525  2:46:30  InnoDB: Started; log sequence number 0 44243
140525  2:46:30 [Note] Event Scheduler: Loaded 0 events
140525  2:46:30 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140530  2:01:53 [Note] /usr/libexec/mysqld: Normal shutdown

140530  2:01:53 [Note] Event Scheduler: Purging the queue. 0 events
140530  2:01:55 [Warning] /usr/libexec/mysqld: Forcing close of thread 283977  user: 'nagios'

140530  2:01:55 [Warning] /usr/libexec/mysqld: Forcing close of thread 283852  user: 'nagios'

140530  2:01:55 [Warning] /usr/libexec/mysqld: Forcing close of thread 283842  user: 'nagios'

140530  2:01:55 [Warning] /usr/libexec/mysqld: Forcing close of thread 283818  user: 'nagiosql'

140530  2:01:55 [Warning] /usr/libexec/mysqld: Forcing close of thread 283797  user: 'nagios'

140530  2:01:55 [Warning] /usr/libexec/mysqld: Forcing close of thread 283620  user: 'nagiosql'

140530  2:02:00  InnoDB: Starting shutdown...
140530  2:02:06  InnoDB: Shutdown completed; log sequence number 0 44243
140530  2:02:06 [Note] /usr/libexec/mysqld: Shutdown complete

140530 02:02:07 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140530 02:05:04 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140530  2:05:04  InnoDB: Initializing buffer pool, size = 8.0M
140530  2:05:04  InnoDB: Completed initialization of buffer pool
140530  2:05:05  InnoDB: Started; log sequence number 0 44243
140530  2:05:05 [Note] Event Scheduler: Loaded 0 events
140530  2:05:05 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140606  2:26:40 [Note] /usr/libexec/mysqld: Normal shutdown

140606  2:26:40 [Note] Event Scheduler: Purging the queue. 0 events
140606  2:26:40  InnoDB: Starting shutdown...
140606  2:26:43  InnoDB: Shutdown completed; log sequence number 0 44243
140606  2:26:43 [Note] /usr/libexec/mysqld: Shutdown complete

140606 02:26:43 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140606 02:26:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140606  2:26:45  InnoDB: Initializing buffer pool, size = 8.0M
140606  2:26:45  InnoDB: Completed initialization of buffer pool
140606  2:26:46  InnoDB: Started; log sequence number 0 44243
140606  2:26:46 [Note] Event Scheduler: Loaded 0 events
140606  2:26:46 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.66'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
I just restart mysqld and the nagios xi interface start to work

Thanks for help
Top
Locked

Return to “Nagios XI”