Apache Running High on Memory Usage

[ssax]

Please PM me a copy of your profile, you can download it from Admin > System Profile by clicking the Download Profile button.

Run this command and then attach the resulting /tmp/HTTPDFILES.zip file:

Code: Select all

zip -r /tmp/HTTPDFILES.zip /etc/php.ini /etc/httpd/conf/httpd.conf /etc/httpd/conf.d /etc/php-fpm.conf /etc/php-fpm.d/www.conf

Additionally, please send the output of these commands:
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

This next command may fail, that's okay, not all systems run postgresql, send the output anyways:

Code: Select all

echo "SELECT relname as Table, pg_size_pretty(pg_total_relation_size(relid)) As Size, pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as ExternalSize FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;" | psql nagiosxi nagiosxi

Include the output of these commands as root:

Code: Select all

sysctl -p
sar
ulimit -a
su -s /bin/bash -c 'ulimit -a' nagios
su -s /bin/bash -c 'ulimit -a' mysql

[J.A.K]

Sent the profile. Put the HTTPD zip and the output from the cmds into this single zip. sysctl -p didn't give me any output though.

output.zip

I was also informed by our server team that CWWCTX595 actually had an issue and had to be restored from a VMWare snapshot around when the issue started. Not sure if the revert caused something to be hung in the XI server, but wanted to mention it.

[J.A.K]

For the moment I set up a cron job to restart the PHP-FPM service nightly. Not idea, but I just can't get it to stop trying those calls.

[ssax]

The php-fpm logs should be here:

Code: Select all

/var/log/php-fpm/

Edit /etc/php-fpm.d/www.conf and change this:

Code: Select all

;pm.max_requests = 500

To this:

Code: Select all

pm.max_requests = 50

Then restart php-fpm:

Code: Select all

systemctl restart php-fpm

What that will do is restart the child process after 50 requests which should free the memory.

See if that helps.

I'm wondering if changing the pm from dynamic to ondemand and then setting the process_idle_timeout would also work in this case:

https://serverfault.com/a/939822

[J.A.K]

I'll try that in the morning to see if it can limit it.

Is there anyway to see why it continuously is trying those ajax calls for that host even when it doesn't exist? If it can happen with one host I'm worried it can happen with even more down the line. The only thing I see in the logs at /var/log/php-fpm are these two errors over and over in www-error.log

[04-Jan-2021 17:34:30 US/Eastern] PHP Warning: A non-numeric value encountered in /Nagios/usr/local/nagiosxi/html/includes/components/metrics/metrics.inc.php on line 179
[04-Jan-2021 17:34:30 US/Eastern] PHP Warning: A non-numeric value encountered in /Nagios/usr/local/nagiosxi/html/includes/components/metrics/metrics.inc.php on line 182

But that seems unrelated.

[J.A.K]

Actually going into "Config File Management". Deleting all files. Writing. Then restarting the service may have fixed it. I no longer see PHP-FPM spawning processes trying to check for that one host over and over. I'll check again in the morning.

[J.A.K]

Yeah that seems to have done it. It's no longer attempting those aJax calls to that singular host over and over, so PHP-FPM hasn't gotten over 1GB and has idled around a few processes. I guess even when I was applying configuration something wasn't being updated in the files themselves until they were purged? I tried this under the recommendation of a colleague of mine using Nagios XI at another company who stated he had ran into something similar.

[dchurch]

While that "non-numeric value" warning message is likely a red herring there's something you can do to fix it.

The piece of code that calculates the data to go into the CPU stats dashlet isn't compatible with newer versions of psutils. There's a fix that will be included in the next release cycle of Nagios XI 5.7.6. In the mean time, here are steps you can follow as a temporary workaround:

1. Edit /usr/local/nagiosxi/cron/sysstat.php
2. On line 268, where it says:

   Code: Select all

   $cmdline = sprintf("/usr/bin/iostat -c 5 2 | tail --lines=2 | head --lines=1 | awk '{ print $1,$2,$3,$4,$5,$6 }'");

   Replace this with:

   Code: Select all

   $cmdline = '/usr/bin/iostat -c 5 2 | grep -v \'^ *$\' | tail -1 | awk \'{ print $1,$2,$3,$4,$5,$6 }\'';

[J.A.K]

That's the one you helped me fix in our other thread right?

https://support.nagios.com/forum/viewto ... 16&t=61086

I've got that modified and it works but doesn't stop the errors from outputting to that log. No worries there though I assume just a red herring like you said. Deleting and rewriting the conf files has definitely solved the issue I was having with those PHP-FPM calls though if we want to lock this thread. Thanks again for all the help.