Frequent Database Errors

[pfsweb]

We are frequently receiving Database Errors in Nagiosxi. The error message suggest to run the database repair which we have but we continue to see these error messages. In addition to seeing these error messages Nagios has also started to get really slow loading pages and applying the configuration. Can anyone point me in the right direction for trouble shooting this?

[hsmith]

Can we get some information about your environment?

1. What version of XI are you running?
2. How much memory do you have?
3. How many cpu cores are you using?
4. How many host / service checks are configured?
5. Do you have a RAMDisk in place?
6. Is your Database offloaded?

That may seem like a lot of questions, but there can be a lot of things that cause Nagios to run slow.

[pfsweb]

Sure that's not a problem,

1. Nagios XI Version : 2014R2.7
2. 12GB of Ram
3. 6 CPU
4. 1196 Hosts and 8614 Services
5. We do have a RAMDisk in place
6. Our Database is offloaded

[hsmith]

Are the database errors currently on the page? If so, could we see what they are?

Can we see some resource usage information?

Code: Select all

top | head -n5
free -m
df -h
df -ih

Is there any extreme latency between your DB and XI server?

I've seen adding more RAM help out with UI responsiveness, but I'm more concerned with the database errors at this point.

[pfsweb]

We are not currently seeing any database errors but if it helps sometimes its seems to be the result of applying the configuration. But that could just be a coincidence.


Command: top | head -n5
Results:
top - 14:20:16 up 4 days, 21:54, 1 user, load average: 1.18, 3.95, 7.25
Tasks: 549 total, 1 running, 548 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.7%us, 7.1%sy, 0.0%ni, 67.4%id, 3.5%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 12198052k total, 9698828k used, 2499224k free, 234264k buffers
Swap: 5095420k total, 0k used, 5095420k free, 6455268k cached

Command: free -m
Results:
total used free shared buffers cached
Mem: 11912 9560 2351 0 228 6304
-/+ buffers/cache: 3027 8884
Swap: 4975 0 4975

Command: df -h
Results:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_nagiosxi-lv_root 69G 27G 39G 41% /
tmpfs 5.9G 12K 5.9G 1% /dev/shm
/dev/sda1 485M 95M 365M 21% /boot
/dev/mapper/vg_nagiosxi-lv_home 143G 197M 135G 1% /home


Command: df -ih
Results:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg_nagiosxi-lv_root 4.4M 163K 4.3M 4% /
tmpfs 1.5M 4 1.5M 1% /dev/shm
/dev/sda1 126K 53 125K 1% /boot
/dev/mapper/vg_nagiosxi-lv_home 9.1M 30 9.1M 1% /home

[pfsweb]

Here's a screen shot of what we're seeing..

[lmiltchev]

Do you see any errors in the mysqld.log on the remote server?

Code: Select all

tail -100 /var/log/mysqld.log

[pfsweb]

No not really here are the contents of the log file:

Code: Select all

150918  0:21:26 [Note] /usr/libexec/mysqld: Shutdown complete

150918 00:21:26 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150921 10:07:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150921 10:07:58  InnoDB: Initializing buffer pool, size = 8.0M
150921 10:07:58  InnoDB: Completed initialization of buffer pool
150921 10:07:58  InnoDB: Started; log sequence number 0 44233
150921 10:07:58 [Note] Event Scheduler: Loaded 0 events
150921 10:07:58 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
151002 23:04:51 [Note] /usr/libexec/mysqld: Normal shutdown

151002 23:04:51 [Note] Event Scheduler: Purging the queue. 0 events
151002 23:04:51  InnoDB: Starting shutdown...
151002 23:04:56  InnoDB: Shutdown completed; log sequence number 0 44233
151002 23:04:56 [Note] /usr/libexec/mysqld: Shutdown complete

151002 23:04:56 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
151007 15:38:41 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
151007 15:38:42  InnoDB: Initializing buffer pool, size = 8.0M
151007 15:38:42  InnoDB: Completed initialization of buffer pool
151007 15:38:42  InnoDB: Started; log sequence number 0 44233
151007 15:38:42 [Note] Event Scheduler: Loaded 0 events
151007 15:38:42 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
151007 15:38:42 [Note] /usr/libexec/mysqld: Normal shutdown

151007 15:38:42 [Note] Event Scheduler: Purging the queue. 0 events
151007 15:38:42  InnoDB: Starting shutdown...
151007 15:38:47  InnoDB: Shutdown completed; log sequence number 0 44233
151007 15:38:47 [Note] /usr/libexec/mysqld: Shutdown complete

151007 15:38:47 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
151007 15:38:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
151007 15:38:58  InnoDB: Initializing buffer pool, size = 8.0M
151007 15:38:58  InnoDB: Completed initialization of buffer pool
151007 15:38:58  InnoDB: Started; log sequence number 0 44233
151007 15:38:58 [Note] Event Scheduler: Loaded 0 events
151007 15:38:58 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
151007 15:38:59 [Note] /usr/libexec/mysqld: Normal shutdown

151007 15:38:59 [Note] Event Scheduler: Purging the queue. 0 events
151007 15:38:59  InnoDB: Starting shutdown...
151007 15:39:04  InnoDB: Shutdown completed; log sequence number 0 44233
151007 15:39:04 [Note] /usr/libexec/mysqld: Shutdown complete

151007 15:39:04 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
151007 15:39:06 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
151007 15:39:06  InnoDB: Initializing buffer pool, size = 8.0M
151007 15:39:06  InnoDB: Completed initialization of buffer pool
151007 15:39:06  InnoDB: Started; log sequence number 0 44233
151007 15:39:06 [Note] Event Scheduler: Loaded 0 events
151007 15:39:06 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
151008 16:24:34 [Note] /usr/libexec/mysqld: Normal shutdown

151008 16:24:34 [Note] Event Scheduler: Purging the queue. 0 events
151008 16:24:34  InnoDB: Starting shutdown...
151008 16:24:39  InnoDB: Shutdown completed; log sequence number 0 44233
151008 16:24:39 [Note] /usr/libexec/mysqld: Shutdown complete

151008 16:24:39 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
151013 16:41:31 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
151013 16:41:32  InnoDB: Initializing buffer pool, size = 8.0M
151013 16:41:32  InnoDB: Completed initialization of buffer pool
151013 16:41:32  InnoDB: Started; log sequence number 0 44233
151013 16:41:32 [Note] Event Scheduler: Loaded 0 events
151013 16:41:32 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
151013 16:41:33 [Note] /usr/libexec/mysqld: Normal shutdown

151013 16:41:33 [Note] Event Scheduler: Purging the queue. 0 events
151013 16:41:33  InnoDB: Starting shutdown...
151013 16:41:37  InnoDB: Shutdown completed; log sequence number 0 44233
151013 16:41:37 [Note] /usr/libexec/mysqld: Shutdown complete

151013 16:41:37 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
151013 16:41:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
151013 16:41:53  InnoDB: Initializing buffer pool, size = 8.0M
151013 16:41:53  InnoDB: Completed initialization of buffer pool
151013 16:41:53  InnoDB: Started; log sequence number 0 44233
151013 16:41:53 [Note] Event Scheduler: Loaded 0 events
151013 16:41:53 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
151013 16:41:54 [Note] /usr/libexec/mysqld: Normal shutdown

151013 16:41:54 [Note] Event Scheduler: Purging the queue. 0 events
151013 16:41:54  InnoDB: Starting shutdown...
151013 16:41:59  InnoDB: Shutdown completed; log sequence number 0 44233
151013 16:41:59 [Note] /usr/libexec/mysqld: Shutdown complete

151013 16:41:59 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
151013 16:42:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
151013 16:42:00  InnoDB: Initializing buffer pool, size = 8.0M
151013 16:42:00  InnoDB: Completed initialization of buffer pool
151013 16:42:00  InnoDB: Started; log sequence number 0 44233
151013 16:42:00 [Note] Event Scheduler: Loaded 0 events
151013 16:42:00 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

[hsmith]

How is your performance on the DB server? Are you running out of space?

Code: Select all

df -h
df -ih
top | head -n5

are what I am especially interested in. How are you doing on latency between these servers?

[pfsweb]

Here are the results. I didn't see issues with space or latency.
#df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_prodmysql02-lv_root
45G 13G 31G 29% /
tmpfs 1.9G 12K 1.9G 1% /dev/shm
/dev/sda1 485M 54M 407M 12% /boot


# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg_prodmysql02-lv_root
2.9M 36K 2.9M 2% /
tmpfs 480K 4 480K 1% /dev/shm
/dev/sda1 126K 44 125K 1% /boot


# top | head -n5
top - 14:49:12 up 94 days, 1:24, 1 user, load average: 1.39, 1.35, 1.32
Tasks: 121 total, 1 running, 120 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.5%us, 1.5%sy, 0.0%ni, 82.7%id, 13.2%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 3924436k total, 3606936k used, 317500k free, 133696k buffers
Swap: 4063224k total, 435008k used, 3628216k free, 2951640k cached