Memory utilization rising over time

[krobertson71]

Nagios 2014 r2.5 - RedHat Enterprise EL5

Notice the longer Nagios XI runs (as a whole) the more ram memory is utilized. When server is fully rebooted, it starts around 2 gig (out of 16). Today it's over 9 gig and we haven't added more than 20 hosts about about 100 services. Now it is caching over 7 gigs of that. No memory swapping is occuring. I know cached memory is not bad, but they seem to keep rising together consistently.

Just worried about a memory leak and what the impact is going to be when we add another 100 hosts and about 500 services.

Here are screenshots of what I am seeing.

Selection_165.png

Selection_166.png

[cmerchant]

You can get an idea of what processes are using memory by invoking this command:

Code: Select all

ps aux --sort -rss

The cached memory usage is ok. The growth of memory usage will be from the concurrent number processes (checks) related to the number of workers defined.

[krobertson71]

Where are workers defined?

Here is output from the 'Top' of the list'

Code: Select all

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mysql     1519  0.7  1.0 4196760 163548 ?      Sl   Feb03 257:54 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
apache   31119  0.1  0.2 461368 40580 ?        S    Feb24   8:04 /usr/sbin/httpd
apache   31120  0.1  0.2 460892 40196 ?        S    Feb24   7:52 /usr/sbin/httpd
apache   11624  0.1  0.2 460512 39428 ?        S    Feb25   5:37 /usr/sbin/httpd
apache   31116  0.1  0.2 460356 39252 ?        S    Feb24   7:57 /usr/sbin/httpd
apache   11623  0.1  0.2 459972 39188 ?        S    Feb25   5:38 /usr/sbin/httpd
apache    4144  0.1  0.2 459852 38944 ?        S    Feb24   7:26 /usr/sbin/httpd
apache   11620  0.1  0.2 458824 37992 ?        S    Feb25   5:46 /usr/sbin/httpd
apache   11645  0.1  0.2 458400 37416 ?        S    Feb25   5:40 /usr/sbin/httpd
apache   31115  0.1  0.2 458008 37352 ?        S    Feb24   8:00 /usr/sbin/httpd
apache   27985  0.1  0.2 458188 37048 ?        S    Feb25   5:00 /usr/sbin/httpd
apache   31122  0.1  0.2 455736 35020 ?        S    Feb24   8:05 /usr/sbin/httpd
apache   31118  0.1  0.2 455192 34480 ?        S    Feb24   8:00 /usr/sbin/httpd
apache   31114  0.1  0.2 455192 34236 ?        S    Feb24   7:56 /usr/sbin/httpd
apache   31117  0.1  0.2 454936 34036 ?        S    Feb24   7:54 /usr/sbin/httpd
nagios   28840  1.1  0.1 327008 30212 ?        S    14:43   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
apache   30934  0.1  0.1 449372 28552 ?        S    Feb24   7:47 /usr/sbin/httpd
root      1667  0.0  0.1 2651124 28460 ?       Ssl  Feb03  16:51 /opt/numara-software/footprints-asset-core/client/bin/mtxagent
nagios   28836  0.9  0.1 319228 22916 ?        S    14:43   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios   28841  0.8  0.1 319468 22820 ?        S    14:43   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios   28839  0.8  0.1 319176 22480 ?        S    14:43   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
root     21634  0.0  0.1 340728 19172 ?        Ss   Feb06   1:08 /usr/sbin/httpd
root      1770  2.1  0.1 461764 17628 ?        Sl   Feb03 735:12 /opt/BESClient/bin/BESClient
root      1724  0.0  0.0 564928 14244 ?        Ssl  Feb03   1:14 /usr/sbin/nsrexecd
nagios   19193  0.3  0.0  34960 12156 ?        Ss   Feb26   4:17 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   28981  4.0  0.0 184320  9408 ?        S    14:43   0:00 python /usr/local/nagios/libexec/check_ncpa.py -H vmhydrogen.dcri.duke.edu -t mytoken -P 5693 -M cpu/percent -w 90 -c 97
postgres 30935  0.0  0.0 217920  8132 ?        Ss   Feb24   0:07 postgres: nagiosxi nagiosxi [local] idle
postgres 31373  0.0  0.0 217924  8132 ?        Ss   Feb24   0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 32109  0.0  0.0 217832  8104 ?        Ss   Feb24   0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 31898  0.0  0.0 217864  8096 ?        Ss   Feb24   0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 12630  0.0  0.0 217872  8076 ?        Ss   Feb25   0:06 postgres: nagiosxi nagiosxi [local] idle
postgres 11642  0.0  0.0 217868  8056 ?        Ss   Feb25   0:06 postgres: nagiosxi nagiosxi [local] idle
postgres  4154  0.0  0.0 217884  8028 ?        Ss   Feb24   0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 11668  0.0  0.0 217828  8012 ?        Ss   Feb25   0:06 postgres: nagiosxi nagiosxi [local] idle
postgres 31741  0.0  0.0 217872  7992 ?        Ss   Feb24   0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 31635  0.0  0.0 217880  7980 ?        Ss   Feb24   0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 11662  0.0  0.0 217868  7960 ?        Ss   Feb25   0:06 postgres: nagiosxi nagiosxi [local] idle
postgres 31534  0.0  0.0 217840  7912 ?        Ss   Feb24   0:08 postgres: nagiosxi nagiosxi [local] idle
postgres 28126  0.0  0.0 217836  7900 ?        Ss   Feb25   0:05 postgres: nagiosxi nagiosxi [local] idle
postgres 32004  0.0  0.0 217840  7852 ?        Ss   Feb24   0:08 postgres: nagiosxi nagiosxi [local] idle
ajaxterm  1759  0.0  0.0 170340  7824 ?        Sl   Feb03   9:44 python /usr/share/ajaxterm/ajaxterm.py --daemon --port=8022 --uid=ajaxterm
postgres 31266  0.0  0.0 217804  7304 ?        Ss   Feb24   0:08 postgres: nagiosxi nagiosxi [local] idle
postgres  1567  0.0  0.0 215996  6696 ?        Ss   Feb03   5:23 postgres: writer process

Here are the number of worker processes I found:

Code: Select all

nagios   19195  0.0  0.0  10016   956 ?        S    Feb26   0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   19198  0.0  0.0  10016   956 ?        S    Feb26   0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   19201  0.0  0.0  10016   956 ?        S    Feb26   0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   19203  0.0  0.0  10016   956 ?        S    Feb26   0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios    1707  0.0  0.0 303352   952 ?        S    Feb03   5:37 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
nagios   19196  0.0  0.0  10016   952 ?        S    Feb26   0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   19197  0.0  0.0  10016   952 ?        S    Feb26   0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   19199  0.0  0.0  10016   952 ?        S    Feb26   0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   19200  0.0  0.0  10016   952 ?        S    Feb26   0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   19202  0.0  0.0  10016   952 ?        S    Feb26   0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   19204  0.0  0.0  10016   952 ?        S    Feb26   0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   19205  0.0  0.0  10016   952 ?        S    Feb26   0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   19206  0.0  0.0  10016   952 ?        S    Feb26   0:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh

[abrist]

This is most likely due to disk cache/buffers. Linux servers get overzealous with using free memory for buffers. It causes no harm, and can significantly help certain areas of performance. What is the output of:

Code: Select all

free -m

[cmerchant]

Your usage is normal, for a historical look at your memory usage, you might want to look at your earlier thread:

http://support.nagios.com/forum/viewtop ... 16&t=30524

[krobertson71]

Sorry I forgot about that.

Free memory after -buffers and cache is 2.3 gig used.

Thanks for the input.

I will keep an eye on it and let you know if it turns into an issue.

Thanks again.