Page 1 of 2

Linux memory monitoring and slab?

Posted: Thu Jun 09, 2016 3:36 pm
by gormank
Has anyone located a Linux memory monitor that adds the slab reclaimable RAM back so when slab doesn't free memory, there are no false alerts?
Personally I think a process that sucks up 200G RAM in a month, whether reclaimable or not is the problem, but that's me...

Re: Linux memory monitoring and slab?

Posted: Thu Jun 09, 2016 6:49 pm
by Box293

Re: Linux memory monitoring and slab?

Posted: Fri Jun 10, 2016 6:05 pm
by gormank
My memory check is as follows:

/usr/local/nagios/libexec/check_nrpe -H <host> -u -t 45 -c check_mem -a '-w 25 -c 10'

It and other memory checks don't add back the allocated but freeable RAM used by slab and will alert. I have no need to check slab usage itself.

Re: Linux memory monitoring and slab?

Posted: Mon Jun 13, 2016 12:30 pm
by gormank
Somewhat unrelated, but it looks like the check_mem plugin isn't always accurate. I'm not sure if its standard or not.

OK - 503843 / 257751 MB (195%) Free Memory, Used: 3550 MB, Shared: 4097 MB, Buffers: 253500 MB, Cached: 249642 MB

$ free
total used free shared buff/cache available
Mem: 263937608 3617244 739340 4196120 259581024 255652432
Swap: 33554428 28856 33525572

$ grep check_mem /usr/local/nagios/etc/nrpe.cfg
$ grep check_mem /usr/local/nagios/etc/nrpe/common.cfg
command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$

$ file /usr/local/nagios/libexec/custom_check_mem
/usr/local/nagios/libexec/custom_check_mem: Bourne-Again shell script, ASCII text executable

$ ll /usr/local/nagios/libexec/custom_check_mem
-rwxr-xr-x. 1 root root 3060 Nov 11 2015 /usr/local/nagios/libexec/custom_check_mem

Re: Linux memory monitoring and slab?

Posted: Mon Jun 13, 2016 3:39 pm
by tgriep
The -n option adds in the cached memory back in to the formula which could account for the discrepancy.
Even the output of the free command doesn't look right.
263 gig total, 3.6 gig used, 739 meg free.

Re: Linux memory monitoring and slab?

Posted: Mon Jun 13, 2016 3:46 pm
by gormank
From what I've seen, that has no effect.

COMMAND: /usr/local/nagios/libexec/check_nrpe -H txslm2mlpxy003 -u -t 45 -c check_mem -a '-w 25 -c 10 -n'
OUTPUT: OK - 503866 / 257751 MB (195%) Free Memory, Used: 3554 MB, Shared: 4065 MB, Buffers: 253449 MB, Cached: 249669 MB | total=257751MB free=503866MB used=3554MB shared=4065 buffers=253449MB cached=249669MB

This is an RHEL 7.2 box and the output of free is different in 5 and 6 as well.

buff/cache is using the memory and its available for reuse.

Re: Linux memory monitoring and slab?

Posted: Mon Jun 13, 2016 8:35 pm
by Box293
gormank wrote:This is an RHEL 7.2 box and the output of free is different in 5 and 6 as well.
Here's an updated version of the plugin that should fix these issues.
custom_check_mem.txt

Re: Linux memory monitoring and slab?

Posted: Tue Jun 14, 2016 1:36 pm
by gormank
Thanks, but we've deviated from the original question (my fault).
In RHEL 6 and 7 at least if slab reclaimable is big, the memory monitor can alert when there's freeable memory.

Re: Linux memory monitoring and slab?

Posted: Tue Jun 14, 2016 5:09 pm
by Box293
I suspect you're going to need to write your own plugin to do this.

Re: Linux memory monitoring and slab?

Posted: Tue Jun 14, 2016 6:28 pm
by gormank
I already have, via /proc/meminfo, but I guess no one else has elasticsearch, or anything else on RHEL 6 or 7 that burns through RAM, or people just don't monitor memory.