Page 1 of 1
Memory utilization rising over time
Posted: Fri Feb 27, 2015 12:27 pm
by krobertson71
Nagios 2014 r2.5 - RedHat Enterprise EL5
Notice the longer Nagios XI runs (as a whole) the more ram memory is utilized. When server is fully rebooted, it starts around 2 gig (out of 16). Today it's over 9 gig and we haven't added more than 20 hosts about about 100 services. Now it is caching over 7 gigs of that. No memory swapping is occuring. I know cached memory is not bad, but they seem to keep rising together consistently.
Just worried about a memory leak and what the impact is going to be when we add another 100 hosts and about 500 services.
Here are screenshots of what I am seeing.
Selection_165.png
Selection_166.png
Re: Memory utilization rising over time
Posted: Fri Feb 27, 2015 12:56 pm
by cmerchant
You can get an idea of what processes are using memory by invoking this command:
The cached memory usage is ok. The growth of memory usage will be from the concurrent number processes (checks) related to the number of workers defined.
Re: Memory utilization rising over time
Posted: Fri Feb 27, 2015 2:47 pm
by krobertson71
Where are workers defined?
Here is output from the 'Top' of the list'
Code: Select all
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mysql 1519 0.7 1.0 4196760 163548 ? Sl Feb03 257:54 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
apache 31119 0.1 0.2 461368 40580 ? S Feb24 8:04 /usr/sbin/httpd
apache 31120 0.1 0.2 460892 40196 ? S Feb24 7:52 /usr/sbin/httpd
apache 11624 0.1 0.2 460512 39428 ? S Feb25 5:37 /usr/sbin/httpd
apache 31116 0.1 0.2 460356 39252 ? S Feb24 7:57 /usr/sbin/httpd
apache 11623 0.1 0.2 459972 39188 ? S Feb25 5:38 /usr/sbin/httpd
apache 4144 0.1 0.2 459852 38944 ? S Feb24 7:26 /usr/sbin/httpd
apache 11620 0.1 0.2 458824 37992 ? S Feb25 5:46 /usr/sbin/httpd
apache 11645 0.1 0.2 458400 37416 ? S Feb25 5:40 /usr/sbin/httpd
apache 31115 0.1 0.2 458008 37352 ? S Feb24 8:00 /usr/sbin/httpd
apache 27985 0.1 0.2 458188 37048 ? S Feb25 5:00 /usr/sbin/httpd
apache 31122 0.1 0.2 455736 35020 ? S Feb24 8:05 /usr/sbin/httpd
apache 31118 0.1 0.2 455192 34480 ? S Feb24 8:00 /usr/sbin/httpd
apache 31114 0.1 0.2 455192 34236 ? S Feb24 7:56 /usr/sbin/httpd
apache 31117 0.1 0.2 454936 34036 ? S Feb24 7:54 /usr/sbin/httpd
nagios 28840 1.1 0.1 327008 30212 ? S 14:43 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
apache 30934 0.1 0.1 449372 28552 ? S Feb24 7:47 /usr/sbin/httpd
root 1667 0.0 0.1 2651124 28460 ? Ssl Feb03 16:51 /opt/numara-software/footprints-asset-core/client/bin/mtxagent
nagios 28836 0.9 0.1 319228 22916 ? S 14:43 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios 28841 0.8 0.1 319468 22820 ? S 14:43 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios 28839 0.8 0.1 319176 22480 ? S 14:43 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
root 21634 0.0 0.1 340728 19172 ? Ss Feb06 1:08 /usr/sbin/httpd
root 1770 2.1 0.1 461764 17628 ? Sl Feb03 735:12 /opt/BESClient/bin/BESClient
root 1724 0.0 0.0 564928 14244 ? Ssl Feb03 1:14 /usr/sbin/nsrexecd
nagios 19193 0.3 0.0 34960 12156 ? Ss Feb26 4:17 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 28981 4.0 0.0 184320 9408 ? S 14:43 0:00 python /usr/local/nagios/libexec/check_ncpa.py -H vmhydrogen.dcri.duke.edu -t mytoken -P 5693 -M cpu/percent -w 90 -c 97
postgres 30935 0.0 0.0 217920 8132 ? Ss Feb24 0:07 postgres: nagiosxi nagiosxi [local] idle
postgres 31373 0.0 0.0 217924 8132 ? Ss Feb24 0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 32109 0.0 0.0 217832 8104 ? Ss Feb24 0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 31898 0.0 0.0 217864 8096 ? Ss Feb24 0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 12630 0.0 0.0 217872 8076 ? Ss Feb25 0:06 postgres: nagiosxi nagiosxi [local] idle
postgres 11642 0.0 0.0 217868 8056 ? Ss Feb25 0:06 postgres: nagiosxi nagiosxi [local] idle
postgres 4154 0.0 0.0 217884 8028 ? Ss Feb24 0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 11668 0.0 0.0 217828 8012 ? Ss Feb25 0:06 postgres: nagiosxi nagiosxi [local] idle
postgres 31741 0.0 0.0 217872 7992 ? Ss Feb24 0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 31635 0.0 0.0 217880 7980 ? Ss Feb24 0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 11662 0.0 0.0 217868 7960 ? Ss Feb25 0:06 postgres: nagiosxi nagiosxi [local] idle
postgres 31534 0.0 0.0 217840 7912 ? Ss Feb24 0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 28126 0.0 0.0 217836 7900 ? Ss Feb25 0:05 postgres: nagiosxi nagiosxi [local] idle
postgres 32004 0.0 0.0 217840 7852 ? Ss Feb24 0:08 postgres: nagiosxi nagiosxi [local] idle
ajaxterm 1759 0.0 0.0 170340 7824 ? Sl Feb03 9:44 python /usr/share/ajaxterm/ajaxterm.py --daemon --port=8022 --uid=ajaxterm
postgres 31266 0.0 0.0 217804 7304 ? Ss Feb24 0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 1567 0.0 0.0 215996 6696 ? Ss Feb03 5:23 postgres: writer process
Here are the number of worker processes I found:
Code: Select all
nagios 19195 0.0 0.0 10016 956 ? S Feb26 0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 19198 0.0 0.0 10016 956 ? S Feb26 0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 19201 0.0 0.0 10016 956 ? S Feb26 0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 19203 0.0 0.0 10016 956 ? S Feb26 0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 1707 0.0 0.0 303352 952 ? S Feb03 5:37 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
nagios 19196 0.0 0.0 10016 952 ? S Feb26 0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 19197 0.0 0.0 10016 952 ? S Feb26 0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 19199 0.0 0.0 10016 952 ? S Feb26 0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 19200 0.0 0.0 10016 952 ? S Feb26 0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 19202 0.0 0.0 10016 952 ? S Feb26 0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 19204 0.0 0.0 10016 952 ? S Feb26 0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 19205 0.0 0.0 10016 952 ? S Feb26 0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 19206 0.0 0.0 10016 952 ? S Feb26 0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
Re: Memory utilization rising over time
Posted: Fri Feb 27, 2015 3:11 pm
by abrist
This is most likely due to disk cache/buffers. Linux servers get overzealous with using free memory for buffers. It causes no harm, and can significantly help certain areas of performance. What is the output of:
Re: Memory utilization rising over time
Posted: Fri Feb 27, 2015 3:20 pm
by cmerchant
Your usage is normal, for a historical look at your memory usage, you might want to look at your earlier thread:
http://support.nagios.com/forum/viewtop ... 16&t=30524
Re: Memory utilization rising over time
Posted: Fri Feb 27, 2015 4:50 pm
by krobertson71
Sorry I forgot about that.
Free memory after -buffers and cache is 2.3 gig used.
Thanks for the input.
I will keep an eye on it and let you know if it turns into an issue.
Thanks again.