Apache Running High on Memory Usage

[J.A.K]

I've noticed on my XI box Apache slowly increases on memory usage eventually using about 15GB of the 16GB available.

Capture2.PNG

Restarting PHP-FPM drops it back down, but over the next few days it slowly climbs again. Any advice on what I can check to see why it's using so much memory?

[dchurch]

Apparently php-fpm is spawning some processes that run for a very long time and never finish. Some have apparently been running for 3 days in your screenshot.

There are a couple ways to go about tuning this. I'd need to know what distro you're using to determine exactly what file to make the configuration changes in.

• One way is changing the max process execution time (php_admin_value[max_execution_time] = 30). Note: 30 seconds is the same timeout as if you were using PHP thru Apache's mod_php instead of php-fpm.
• Another way is to limit the number of child processes (pm = dynamic; pm.max_children = x) - Warning: This could lead to the server becoming unresponsive if tons of children are left running.
• Something else you could try is using static allocation (pm = static; pm.max_children = x).
• You could also try ditching PHP-FPM, and just using Apache's mod_php. FPM might be overkill anyway unless you have dozens of users connecting to Nagios XI at the same time.

[J.A.K]

Distro is RHEL 8.2. We'll have a lot hitting the API, but should only be 1 - 2 of us actually using the admin website.

/etc/php-fpm.d/www.conf
/etc/php.ini

Would be the two files I assume to modify. I can start with adding that:

php_admin_value[max_execution_time] = 30

To the conf since it doesn't appear to already exist. Is there anyway to see why the processes aren't finishing?

[dchurch]

FPM is kinda particular where you put configuration values. It would be under the [www] heading in /etc/php-fpm.d/www.conf. E.g.:

Code: Select all

[www]
# Now directives under this apply to the "www" pool, same as your screenshot
# ...
php_admin_value[max_execution_time] = 30

And there is a way to figure out what a PHP process is doing if you look at access_log and cross reference it with when the process started.

[J.A.K]

Got it I stuck it directly under [www] and restarted the php-frm service, so I'll see if it starts building again.

Capture.PNG

Are we talking about /etc/httpd/logs/access_log or /var/log/httpd/access_log for the access log? Checked those out, but nothing stood out to me as an issue.

[dchurch]

It'll be whichever access log is the access log for Apache. You'd be cross-referencing by the second the php-fpm process was kicked off.

For instance if ps -A x -f |grep php-fpm returns a bunch of pids, and you see one that's been running a long time, then you'd find out when it was spawned:

Code: Select all

ls -lad /proc/<PID>
dr-xr-xr-x 9 nagios nagios 0 Dec 29 18:18 /proc/1058

Then you'd look for when that pid was created, in this case 2020-12-29 18:18 in the access_log to see what request was made that spawned that process.

[J.A.K]

/etc/httpd/logs/access_log seems to be correct for RHEL, but whenever php-fpm restarts this is all that's in there

Capture3.PNG

Would maybe turning on the php-fpm status page help? Does Nagios XI have any issues with doing that?

[dchurch]

No, turning on the PHP-FPM status page shouldn't affect Nagios XI in any way unless it's either set to /nagiosxi or /nagios.

Setting it to /status might constitute a security risk if it's not password-protected and if the IP address of your Nagios XI box is exposed on the public internet (scanners may target that URL and be able to see your FPM stats).

Something like this may be more appropriate from a security perspective in the php-fpm config file:

Code: Select all

pm.status_path = /status-hFNKMuo4

[J.A.K]

Interesting so they seem to be stuck on one of our hosts:


************************
pid: 2493300
state: Idle
start time: 30/Dec/2020:11:44:53 -0500
start since: 97
requests: 14
request duration: 159244
request method: GET
request URI: /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_host_status_detailed_info_html%22%2C%22args%22%3A%7B%22hostname%22%3A%22
CWWCTX595%22%2C%22host_id%22%3A%22924%22%2C%22display%22%3A%22simple%22%7D%7D&nsp=5413e7e1608075be9a5c5ed19c3a840fc68a18712260c918b5f24f4c4bb768c5
content length: 0
user: -
script: /usr/local/nagiosxi/html/ajaxhelper.php
last request cpu: 75.36
last request memory: 2097152

************************
pid: 2493391
state: Idle
start time: 30/Dec/2020:11:45:10 -0500
start since: 80
requests: 11
request duration: 156531
request method: GET
request URI: /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_host_status_attributes_html%22%2C%22args%22%3A%7B%22hostname%22%3A%22
CWWCTX595%22%2C%22host_id%22%3A%22924%22%2C%22display%22%3A%22all%22%7D%7D&nsp=5413e7e1608075be9a5c5ed19c3a840fc68a18712260c918b5f24f4c4bb768c5
content length: 0
user: -
script: /usr/local/nagiosxi/html/ajaxhelper.php
last request cpu: 63.89
last request memory: 2097152

************************
pid: 2493461
state: Idle
start time: 30/Dec/2020:11:45:19 -0500
start since: 71
requests: 9
request duration: 141433
request method: GET
request URI: /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_host_status_detailed_info_html%22%2C%22args%22%3A%7B%22hostname%22%3A%22
CWWCTX595%22%2C%22host_id%22%3A%22924%22%2C%22display%22%3A%22advanced%22%7D%7D&nsp=5413e7e1608075be9a5c5ed19c3a840fc68a18712260c918b5f24f4c4bb768c5
content length: 0
user: -
script: /usr/local/nagiosxi/html/ajaxhelper.php
last request cpu: 77.78
last request memory: 2097152

************************
pid: 2493790
state: Idle
start time: 30/Dec/2020:11:46:19 -0500
start since: 11
requests: 1
request duration: 172238
request method: GET
request URI: /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_host_status_quick_actions_html%22%2C%22args%22%3A%7B%22hostname%22%3A%22
CWWCTX595%22%2C%22host_id%22%3A%22924%22%2C%22display%22%3A%22simple%22%7D%7D&nsp=5413e7e1608075be9a5c5ed19c3a840fc68a18712260c918b5f24f4c4bb768c5
content length: 0
user: -
script: /usr/local/nagiosxi/html/ajaxhelper.php
last request cpu: 63.87
last request memory: 18874368

Capture3.PNG

Should I maybe just try to recreate it?

[J.A.K]

Well even after deleting the host and services, applying configuration, and restarting php-fpm it still keeps trying those calls. Also tried reinstalling the NCPA agent on the CWWCTX595 server, restarting the XI server, and restarting the mod gearman worker services on the worker making the calls.