Nagios Core - Memory leak

[bengal]

Hi,

We use Nagios Core to monitor an infrastructure, but we encountered issue with a memory leak phenomenon.
We need to restart it every two day.


Let see Infra.

Nagios Server :
OS : Redhat 6.2
RAM : 64Go
CPU : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz with 16 CORES

Nagios Version packages :
Nagios COre 4.11.1
gearmand-devel.x86_64 1.1.12-1 @nagios-rh6
gearmand-server.x86_64 1.1.12-1 @nagios-rh6
libevent-devel.x86_64 1.4.13-1.el6 @nagios-rh6
libthruk.x86_64 2.02-1.el6 @nagios-rh6
libtool-ltdl-devel.x86_64 2.2.6-15.5.el6 @nagios-rh6
mod_gearman.x86_64 1.4_nagios4-1.el6 @nagios-rh6
nagios.x86_64 4.1.1-2.el6 @nagios-rh6
nagios-plugins.x86_64 2.1.1-1 @/nagios-plugins-2.1.1-1.x86_64
nagios-plugins-disk.x86_64 1.4.15-2.el6 @nagios-rh6
nagios-plugins-load.x86_64 1.4.15-2.el6 @nagios-rh6
nagios-plugins-nrpe.x86_64 2.15-7.el6 @nagios-rh6
nagios-plugins-procs.x86_64 1.4.15-2.el6 @nagios-rh6
nrpe.x86_64 2.12-16.el6 @nagios-rh6
pnp4nagios.x86_64 0.6.25-1 @/pnp4nagios-0.6.25-1.x86_64
thruk.x86_64 2.02-1 @nagios-rh6
thruk-base.x86_64 2.02-1 @nagios-rh6
thruk-plugin-reporting.x86_64 2.02-1 @nagios-rh6

Code: Select all

nagios -v /etc/nagios/nagios.cfg

Nagios Core 4.1.1
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-19-2015
License: GPL

Website: https://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
	Checked 125847 services.
	Checked 14010 hosts.
	Checked 295 host groups.
	Checked 1 service groups.
	Checked 12 contacts.
	Checked 4 contact groups.
	Checked 63 commands.
	Checked 8 time periods.
	Checked 0 host escalations.
	Checked 0 service escalations.
Checking for circular paths...
	Checked 14010 hosts
	Checked 0 service dependencies
	Checked 0 host dependencies
	Checked 8 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0
Things look okay - No serious problems were detected during the pre-flight check

Code: Select all

nagiostats 

Nagios Stats 4.1.1
Copyright (c) 2003-2008 Ethan Galstad (www.nagios.org)
Last Modified: 08-19-2015
License: GPL

CURRENT STATUS DATA
------------------------------------------------------
Status File:                            /var/spool/nagios/status.dat
Status File Age:                        0d 0h 0m 14s
Status File Version:                    4.1.1

Program Running Time:                   2d 2h 19m 33s
Nagios PID:                             10315

Total Services:                         125847
Services Checked:                       125847
Services Scheduled:                     125847
Services Actively Checked:              125847
Services Passively Checked:             0
Total Service State Change:             0.000 / 44.470 / 0.011 %
Active Service Latency:                 0.000 / 5.866 / 1.874 sec
Active Service Execution Time:          0.000 / 20.235 / 0.084 sec
Active Service State Change:            0.000 / 44.470 / 0.011 %
Active Services Last 1/5/15/60 min:     7206 / 57684 / 125816 / 125847
Passive Service Latency:                0.000 / 0.000 / 0.000 sec
Passive Service State Change:           0.000 / 0.000 / 0.000 %
Passive Services Last 1/5/15/60 min:    0 / 0 / 0 / 0
Services Ok/Warn/Unk/Crit:              125401 / 17 / 8 / 421
Services Flapping:                      54
Services In Downtime:                   265

Total Hosts:                            14010
Hosts Checked:                          14010
Hosts Scheduled:                        13946
Hosts Actively Checked:                 14010
Host Passively Checked:                 0
Total Host State Change:                0.000 / 21.840 / 0.012 %
Active Host Latency:                    0.000 / 6.107 / 1.632 sec
Active Host Execution Time:             0.000 / 10.003 / 0.015 sec
Active Host State Change:               0.000 / 21.840 / 0.012 %
Active Hosts Last 1/5/15/60 min:        17 / 10863 / 13919 / 13919
Passive Host Latency:                   0.000 / 0.000 / 0.000 sec
Passive Host State Change:              0.000 / 0.000 / 0.000 %
Passive Hosts Last 1/5/15/60 min:       0 / 0 / 0 / 0
Hosts Up/Down/Unreach:                  13982 / 4 / 24
Hosts Flapping:                         6
Hosts In Downtime:                      8

Active Host Checks Last 1/5/15 min:     0 / 0 / 0
   Scheduled:                           0 / 0 / 0
   On-demand:                           0 / 0 / 0
   Parallel:                            0 / 0 / 0
   Serial:                              0 / 0 / 0
   Cached:                              0 / 0 / 0
Passive Host Checks Last 1/5/15 min:    0 / 0 / 0
Active Service Checks Last 1/5/15 min:  0 / 0 / 0
   Scheduled:                           0 / 0 / 0
   On-demand:                           0 / 0 / 0
   Cached:                              0 / 0 / 0
Passive Service Checks Last 1/5/15 min: 0 / 0 / 0

External Commands Last 1/5/15 min:      0 / 0 / 4

[tmcdonald]

This was identified as a bug in mod_gearman, so the only fix is either to disable MG or use an older version. The developer who discovered this is not in the office today, so I will ask him for specifics and we'll update this when he returns.

[bengal]

ps : it's not an april fish

[bengal]

Ok thx tmcdonald.

[tmcdonald]

It looks like we have an install script that should work to install the correct version, but I'd need to get details from our dev first. Before we dive too deep into that though, and with the understanding that I realize this might not be an easy transition, is there any way you could disable mod_gearman and watch the memory usage?