Memory Problem

[EnvBroker1]

We are having a memory problem, it just slowly drains away. Since Jan, our check memory service has been in critical or warning 22% of the time. A few days ago it was at 3% free, we rebooted and it went back to 89% free but after about 2 days it was back down to 50% free. Our memory was increased to 16GB and after the reboot it was over 80% free. Then this morning it was already back down to only 60% free. Rebooted it again and it was back up to 86% memory free.

I attached the zipped up system profile and also the state history for that service, showing how fast it's changing states. If we just watch and don't intervene, I'm sure it will go all the way to 3% free again.

Any ideas on our leaky memory problem?

System Status.JPG

Nagios Memory.JPG

[rkennedy]

Are you using mod_gearman at all? We recently had an issue where a memory leak was causing something similar to happen.

[EnvBroker1]

Are you using mod_gearman at all?

If it's setup as a default or something maybe. But I didn't do anything directly concerning mod_gearman.

How do I check to see if it's being used?

[Box293]

There will be a broker_module= line in /usr/local/nagios/etc/nagios.cfg. This should return output if you are using it:

Code: Select all

grep mod_gearman /usr/local/nagios/etc/nagios.cfg

[EnvBroker1]

We are not using mod_gearman.

[hsmith]

Looking at your profile, there are a few things that are concerning:

You have gnome installed. This is unsupported by us, we only support RHEL/CentOS minimal installations.
You're only using 2GB of your RAM in the profile you sent us. I would like a profile of when you're experiencing the issue.
You're getting heavily spammed by the following message(/var/log/httpd/error_log):

Code: Select all

[Thu Feb 04 11:33:16 2016] [error] [client 127.0.0.1] PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PST/-8.0/no DST' instead in /usr/local/nagiosxi/html/includes/pageparts.inc.php on line 345

That may not be the issue, but having something constantly logging like that could be an issue if that log is in memory and getting bigger over time.