Nagios 2014R2.3 on VM HIGH LOAD SPIKES

[carlos.atos]

Hello Nagios Support,

We are very close to purchase a couple of Nagios licenses for our projects and we had decided to deploy your nagios VM machine (version 2014r2.0) in Our VM ESXI host 5.5.0, I had installed VMtools on the VM, Upgraded RAM to 4GB and configured 8 host for ping services and 2 SNMP routers for network traffic, and the load spikes occurrs every 5~7 hours, these up to 70 and over.
-All the host has been added using the monitoring wizards.

After Reading all these posts I had tried everything here,as a final hope I have upgraded my nagios XI to Verson 2014R2.3, at the begining it seems to be working but then the spikes appeared as shows,

I had just changed in this VM ;
According the document http://assets.nagios.com/downloads/nagi ... ios-XI.pdf
The reaper timers in: usr/local/nagios/etc/nagios.cfg
check_result_reaper_frequency=3
max_check_result_reaper_time=10
and according to http://nagios.sourceforge.net/docs/nagi ... ule_checks
use_aggressive_host_checking=0
passive_host_checks_are_soft=1

and this is what i Got:

Top Nagios JAN-08.1.PNG

I have The CentOS 6.6(final)
Kernel 2.6.32-504.1.3.e16.x86_64 on an x86_64
Rightnow, I dont have any other VM running in the ESXI

My question is, How can I avoid this ?
why this is happening if I had dowloaded your VM image and I haven't change anything else on the VM?

I would appreciate your response
Regards,

[lmiltchev]

Do you see any errors in the mysqld.log? Crashed tables can cause this behavior.

Code: Select all

tail -50 /var/log/mysqld.log

[carlos.atos]

Hello, thanks for your response.

This is what I have from: tail -50 /var/log/mysqld.log

Code: Select all

141223 11:34:13  InnoDB: Completed initialization of buffer pool
141223 11:34:13  InnoDB: Started; log sequence number 0 44233
141223 11:34:13 [Note] Event Scheduler: Loaded 0 events
141223 11:34:13 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
141223 13:20:11 [Note] /usr/libexec/mysqld: Normal shutdown

141223 13:20:11 [Note] Event Scheduler: Purging the queue. 0 events
141223 13:20:11  InnoDB: Starting shutdown...
141223 13:20:15  InnoDB: Shutdown completed; log sequence number 0 44233
141223 13:20:15 [Note] /usr/libexec/mysqld: Shutdown complete


141223 13:20:15 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
141223 13:21:17 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
141223 13:21:17  InnoDB: Initializing buffer pool, size = 8.0M
141223 13:21:17  InnoDB: Completed initialization of buffer pool
141223 13:21:18  InnoDB: Started; log sequence number 0 44233
141223 13:21:18 [Note] Event Scheduler: Loaded 0 events
141223 13:21:18 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
141229 11:36:55 [Note] /usr/libexec/mysqld: Normal shutdown

141229 11:36:55 [Note] Event Scheduler: Purging the queue. 0 events
141229 11:36:57  InnoDB: Starting shutdown...
141229 11:37:00  InnoDB: Shutdown completed; log sequence number 0 44233
141229 11:37:00 [Note] /usr/libexec/mysqld: Shutdown complete

141229 11:37:00 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150107 15:17:55 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150107 15:17:55  InnoDB: Initializing buffer pool, size = 8.0M
150107 15:17:55  InnoDB: Completed initialization of buffer pool
150107 15:17:55  InnoDB: Started; log sequence number 0 44233
150107 15:17:55 [Note] Event Scheduler: Loaded 0 events
150107 15:17:55 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
150107 15:34:50 [Note] /usr/libexec/mysqld: Normal shutdown

150107 15:34:50 [Note] Event Scheduler: Purging the queue. 0 events
150107 15:34:50  InnoDB: Starting shutdown...
150107 15:34:52  InnoDB: Shutdown completed; log sequence number 0 44233
150107 15:34:52 [Note] /usr/libexec/mysqld: Shutdown complete

150107 15:34:52 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150107 15:35:34 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150107 15:35:34  InnoDB: Initializing buffer pool, size = 8.0M
150107 15:35:34  InnoDB: Completed initialization of buffer pool
150107 15:35:35  InnoDB: Started; log sequence number 0 44233
150107 15:35:35 [Note] Event Scheduler: Loaded 0 events
150107 15:35:35 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

just in case I had ran the repair scripts without errors
/usr/local/nagiosxi/scripts/repairmysql.sh nagios
/usr/local/nagiosxi/scripts/repairmysql.sh nagiosql

By the way, from yesterday to now, I have had this behaviour, a really big peak (150) at 5pm and then smaller peaks (below 50) :

localhost-current_load Jan09.jpg

hope this helps you to find a root cause for this.
Carlos

[tmcdonald]

What Core version are you running?

Code: Select all

/usr/local/nagios/bin/nagios | head -n 2

[carlos.atos]

I have :

Code: Select all

[root@localhost ~]# /usr/local/nagios/bin/nagios |head -n 2

Nagios Core 4.0.8

Cheers,

[carlos.atos]

Hello Guys,

this morning my Nagios is totally crashed.

This errors notification have appeared on the console

Error -12-01-15.PNG

The Nagios is totally stalted. no web interface, no response via console. only these messages.
According to my ESXI, the VM is consuming all te RAM(4GB) and processor (1595Mhz) available.

Any Idea?
Please,

[carlos.atos]

and the Load is quite high also:

TOP 343 -12-01-15.PNG

[lgroschen]

After doing some research this is a common problem in CentOS 6.6 and usually is fixed by a firmware update. Can you try updating your hardwares firmware?

https://bugs.centos.org/view.php?id=7915
https://www.centos.org/forums/viewtopic.php?t=2024

[carlos.atos]

Hi,

What do you mean with hardware firmware upgrade? If we are using a VM ( the virtual machine available to download from your website http://library.nagios.com/library/produ ... loads/main 64bits) i think this VM should have the latest upgrades installed, we have already installed VM tools (that are supossed to be the Hardware drivers for the VM) . there wont be any hardware problems right?

please clarify this

At the moment the machine is totally unresposive, we are going to upgrade all the packages for this centos Version as soon we can get control of the VM.

As told you before we are using ESXI 5.5.0 in a physical Fujitsu Server model rx300 S3

Regards,

[lgroschen]

I just researched the errors that you posted and the trend was people running versions of CentOS on a machine with outdated firmware got the exact same error. Not saying this is the source of the problem since you are running on a VM, but there is something with the VM that may be corrupted giving you this error.