Page 4 of 5
Re: Nagios server memory use
Posted: Wed Jan 20, 2016 5:06 pm
by rkennedy
Got it.
Are you able to push for a mod_gearman update then (with time)?
Let us know the result, and if it helps with the changes you made.
Re: Nagios server memory use
Posted: Wed Jan 20, 2016 5:58 pm
by gormank
I can do that, but the thing that makes me nervous is that I see gearman running, but I don't see it installed.
# rpm -qa | grep -i gear
# yum list | grep -i gearman
# yum list installed | grep -i gearman
# yum list | wc -l
12036
I created a test VM w/ the same RHEL version as my live Nagios and and installed Nagios. It also has no output when running rpm -qa and yum list so I guess its normal and I'm worrying about nothing...
I'll do the german update on the test system to verify.
The servers were restarted about 36 hours ago. Right now memory use reported by htop is 9G on one live system and 7G on the other. 1G on the failover servers that aren't running any checks.
Nagios and top report 11G and 10G used on the primary servers.
Re: Nagios server memory use
Posted: Thu Jan 21, 2016 2:17 pm
by rkennedy
It seems to be leaking, but at this rate you should be good for a month with the leak!
I'll ask the developers about the package issue right now. Let us know how it works on your test system.
Re: Nagios server memory use
Posted: Thu Jan 21, 2016 2:56 pm
by rkennedy
I spoke with the developers, and here's the response.
Since it's not listed in yum or rpm then make sure to turn the old gearman explicitly off, and then install the new version.
Re: Nagios server memory use
Posted: Thu Jan 21, 2016 5:28 pm
by gormank
So turn off means stop them as shown?
service gearmand stop
service mod_gearman_worker stop
These are the only gearman services listed by chkconfig...
Re: Nagios server memory use
Posted: Fri Jan 22, 2016 2:31 pm
by rkennedy
That is part of it, yes, turn them off, and then the second part is to disable them from startup. Once that is done you should be fine to proceed with
@bheden's instructions.
Re: Nagios server memory use
Posted: Thu Jan 28, 2016 12:59 pm
by gormank
Just FYI...
Memory use is at 35G in one server and 29G in the other. Adding RAM isn't helping.
Re: Nagios server memory use
Posted: Thu Jan 28, 2016 3:21 pm
by bheden
I rewrote the ModGearman install script to install/upgrade automatically and copy the necessary configuration files over.
This is still in testing, but has passed all of our internal tests so far.
If you'd like to give it a go (with the caveat of supplying feedback, of course), the URL is:
http://assets.nagios.com/downloads/nagi ... Install.sh
On your server:
Code: Select all
wget http://assets.nagios.com/downloads/nagiosxi/scripts/ModGearmanInstall.sh
chmod +x ModGearmanInstall.sh
./ModGearmanInstall.sh --server --upgrade
And then follow the prompts. This *should* remove (or ignore) all gearman installs and install the new ones and set them up to be used by Nagios.
It will prompt you for configuration file locations - make sure they are correct before continuing.
Having a good known backup before hand is a good idea, too.
Re: Nagios server memory use
Posted: Mon Feb 01, 2016 2:23 pm
by gormank
Can I disable Nagios use of gearman by commenting the broker_module line in nagios.cfg and stopping gearmand, and mod_gearman_worker?
My systems aren't that big with ~160 hosts and ~1000 services.
Re: Nagios server memory use
Posted: Mon Feb 01, 2016 2:50 pm
by bheden
Disabling the broker_module line in nagios.cfg and restarting the nagios should do it.
Stopping gearmand or mod_gearman_worker isn't necessary, but wouldn't hurt either.