Page 1 of 2
constant increase in memory usage
Posted: Wed Mar 17, 2021 3:30 pm
by supportcl
Hello, since we updated to the latest version, we are having a RAM use problem, its use is slowly increasing until it ends and it begins to give ram usage alerts, we restart the server and it starts again, this takes days but in the graph can be seen as it increases as time passes. What could be happening?
I am talking about my nagiosxi server, in the old version it had half the resources and it worked without problems
Re: constant increase in memory usage
Posted: Thu Mar 18, 2021 12:58 pm
by dchurch
What specifically is the process taking up all the RAM?
If you don't know how to figure that out, you can PM me a profile so I can diagnose further. Get one by going to Admin (top menu) => System Profile (in the left menu), then clicking the blue button.
If you're unable to generate the the profile through the web interface, please try generating it from the command line by running these commands as root:
Code: Select all
rm -rf /usr/local/nagiosxi/var/components/profile*
/usr/local/nagiosxi/scripts/components/getprofile.sh SUPPORT
Then send me the resulting
/usr/local/nagiosxi/var/components/profile.zip file.
If the profile script fails, please include the ENTIRE output.
Re: constant increase in memory usage
Posted: Thu Mar 18, 2021 1:20 pm
by supportcl
I have sent you the requested profile file.
The strange thing is that I could not see what is consuming the RAM, now I had to restart the system since we were having problems.
Check the server log file
Re: constant increase in memory usage
Posted: Thu Mar 18, 2021 2:09 pm
by dchurch
Looks like mysqld was using 4.9GB of memory or more when the OOM killer shot it in the head, and even after a reboot it's still using 118% of a CPU. Maybe there's a rogue index somewhere.
What's the output from the following command?
Code: Select all
mysql -unagiosxi -pn@gweb --table <<< "select table_name as 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' from information_schema.TABLES where table_schema in ('nagios', 'nagiosql', 'nagiosxi');"
Re: constant increase in memory usage
Posted: Thu Mar 18, 2021 2:12 pm
by supportcl
Code: Select all
[root@nagiosxi log]# mysql -unagiosxi -pn@gweb --table <<< "select table_name as 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' from information_schema.TABLES where table_schema in ('nagios', 'nagiosql', 'nagiosxi');"
mysql: [Warning] Using a password on the command line interface can be insecure.
+-----------------------------+------------+
| Table | Size in MB |
+-----------------------------+------------+
| xi_auditlog | 7.03 |
| xi_auth_tokens | 1.56 |
| xi_cmp_ccm_backups | 0.02 |
| xi_cmp_favorites | 0.03 |
| xi_cmp_nagiosbpi_backups | 0.27 |
| xi_cmp_scheduledreports_log | 0.02 |
| xi_cmp_trapdata | 0.03 |
| xi_cmp_trapdata_log | 0.03 |
| xi_commands | 0.01 |
| xi_deploy_agents | 0.02 |
| xi_deploy_jobs | 0.02 |
| xi_eventqueue | 0.10 |
| xi_events | 0.14 |
| xi_incidents | 0.00 |
| xi_meta | 8.31 |
| xi_mibs | 0.05 |
| xi_options | 0.03 |
| xi_sessions | 0.03 |
| xi_sysstat | 0.01 |
| xi_usermeta | 0.56 |
| xi_users | 0.02 |
+-----------------------------+------------+
[root@nagiosxi log]#
Re: constant increase in memory usage
Posted: Fri Mar 19, 2021 12:41 pm
by dchurch
Try running the database repair script, and let me know if that is successful. Run the following as root from the terminal.
Code: Select all
/usr/local/nagiosxi/scripts/repair_databases.sh
See here for complete instructions:
run the database repair
Re: constant increase in memory usage
Posted: Mon Mar 22, 2021 3:53 pm
by dchurch
Did you get a chance to do this? Let us know if you have any related questions or if we're okay to lock this up and mark it as resolved.
Re: constant increase in memory usage
Posted: Wed Mar 24, 2021 1:39 am
by jweijters
Hi I have similar problem, however memory usage goes pretty fast.
It is the php-fpm processes using up the memory:
I ran a repair database. for a moment the usage was gone, however after approx 15 minutes the memory usage is very high again.
see the 4 processes using 15.5% memory
Code: Select all
$ free -h
total used free shared buff/cache available
Mem: 15G 12G 1.4G 262M 1.8G 2.6G
Swap: 5.9G 647M 5.2G
$ ps aux |grep -i php-fpm
apache 625 0.5 0.2 547104 36820 ? S 07:12 0:10 php-fpm: pool www
apache 633 2.7 15.4 3023704 2513976 ? S 07:12 0:48 php-fpm: pool www
apache 634 0.6 0.2 545736 36040 ? R 07:12 0:12 php-fpm: pool www
apache 645 0.5 0.2 544676 35080 ? S 07:12 0:10 php-fpm: pool www
apache 1433 2.5 15.5 3028084 2516676 ? S 07:13 0:43 php-fpm: pool www
apache 1770 0.5 0.2 543236 33880 ? S 07:14 0:08 php-fpm: pool www
apache 1793 0.5 0.2 544872 35892 ? S 07:14 0:09 php-fpm: pool www
apache 1798 0.6 0.2 546440 37124 ? S 07:14 0:11 php-fpm: pool www
apache 1799 0.6 0.2 546080 34860 ? S 07:14 0:10 php-fpm: pool www
apache 1822 0.5 0.4 577280 67488 ? S 07:14 0:09 php-fpm: pool www
apache 1823 0.5 0.2 544316 32740 ? S 07:14 0:09 php-fpm: pool www
apache 1824 0.5 0.2 546120 35404 ? S 07:14 0:09 php-fpm: pool www
apache 1825 0.5 0.2 631816 39812 ? S 07:14 0:09 php-fpm: pool www
apache 2107 0.5 0.2 544692 35060 ? S 07:14 0:10 php-fpm: pool www
apache 9366 2.8 15.5 3031320 2519108 ? S 07:18 0:40 php-fpm: pool www
apache 15859 3.8 15.4 3025812 2515480 ? S 07:24 0:40 php-fpm: pool www
root 33761 0.0 0.0 112816 984 pts/1 R+ 07:42 0:00 grep --color=auto -i php-fpm
root 201405 0.0 0.0 525328 3068 ? Ss 06:25 0:00 php-fpm: master process (/etc/php-fpm.conf)
apache 245255 0.4 0.2 543424 33824 ? S 07:10 0:09 php-fpm: pool www
apache 245258 0.4 0.2 544884 35316 ? S 07:11 0:08 php-fpm: pool www
apache 245259 0.5 0.2 550256 40580 ? S 07:11 0:09 php-fpm: pool www
apache 245260 0.5 0.2 543576 33856 ? S 07:11 0:11 php-fpm: pool www
Re: constant increase in memory usage
Posted: Wed Mar 24, 2021 4:47 pm
by dchurch
PHP is ballooning out to ~500MB-3GB per process, and your PHP-FPM configuration is either set to spawn too many of them, or they're not being reaped. My idea is to change how the children are created, and reap them more effectively.
Edit under the
[www] heading in
/etc/php-fpm.d/www.conf (replace and/or uncomment values rather than adding them):
Code: Select all
[www]
# Now directives under this apply to the "www" pool, same as your screenshot
# ...
pm = ondemand
pm.max_children = 5
pm.process_idle_timeout = 10s
pm.max_requests = 500
Re: constant increase in memory usage
Posted: Wed Apr 07, 2021 5:41 am
by jweijters
I Adjusted these Parameters,
However it doesn't help.
Because it is the php-fpm, which is started from the http-server so browser initiated, how can I find out which user is causing this high amount of memory usage?
kind regards,
Joris Weijters