Linux Cached Memory

[saber]

Hi,

We are using the latest "custom_check_mem" and even the "-n" param but no success. It's still show Critical even if all the RAM is in cache.

CentOS 7.8 x86_64 latest version of everything.

Code: Select all

# cat /usr/local/nagios/etc/nrpe/common.cfg
(...)
command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$
(...)

# /usr/local/nagios/libexec/custom_check_mem -n -w 20 -c 10
CRITICAL - 12432 / 128675 MB (9%) Free Memory, Used: 37987 MB, Shared: 218 MB, Buffers + Cached: 80799 MB | total=128675MB free=12432MB used=37987MB shared=218MB buffers_and_cached=80799MB

Any idea?

Thanks,
Saber

[lmiltchev]

Can you run the following commands on the remote machine, and show the output?

Code: Select all

free -m
/usr/local/nagios/libexec/custom_check_mem -w 20 -c 10
/usr/local/nagios/libexec/custom_check_mem -n -w 20 -c 10

Also, upload the entire /usr/local/nagios/libexec/custom_check_mem script that you are using on the forum.

[saber]

Hi,

I appreciate the answer!

Code: Select all

# free -m
              total        used        free      shared  buff/cache   available
Mem:         128675       37092       10767         218       80816       90590
Swap:          8191           0        8191

# /usr/local/nagios/libexec/custom_check_mem -w 20 -c 10
CRITICAL - 10751 / 128675 MB (8%) Free Memory, Used: 37107 MB, Shared: 218 MB, Buffers + Cached: 80815 MB | total=128675MB free=10751MB used=37107MB shared=218MB buffers_and_cached=80815MB

# /usr/local/nagios/libexec/custom_check_mem -n -w 20 -c 10
CRITICAL - 13263 / 128675 MB (10%) Free Memory, Used: 37153 MB, Shared: 218 MB, Buffers + Cached: 80815 MB | total=128675MB free=13263MB used=37153MB shared=218MB buffers_and_cached=80815MB

[lmiltchev]

Hmm, the free memory in your last command is definitely wrong... I wonder if you have some of the programs that the plugin uses installed in a "non-default" location.

Please run the commands below, and show the output.

Code: Select all

which gawk
which dc
which sed
memoutput=`free -m -w | head -2 | tail -1`
free=`echo $memoutput | gawk '{print $4+$7}'`
echo $memoutput
echo $free

[saber]

Hi,

Thanks for the reply.

Code: Select all

# which gawk
/usr/bin/gawk

# which dc
/usr/bin/dc

# which sed
/usr/bin/sed

# memoutput=`free -m -w | head -2 | tail -1`

# free=`echo $memoutput | gawk '{print $4+$7}'`

# echo $memoutput
Mem: 128675 38105 9736 226 78276 2557 89568

# echo $free
12293

[lmiltchev]

The "free" memory seems to be correct in the latest output. When you run:

Code: Select all

free -m -w

the "-w" flag (wide output) shows buffers and cache memory separated (in different columns). So, when you use the "custom_check_mem" plugin with "-n", only the "cache" memory is added to the "free" memory.

In your example:

# memoutput=`free -m -w | head -2 | tail -1`

# free=`echo $memoutput | gawk '{print $4+$7}'`

# echo $memoutput

Mem: 128675 38105 9736 226 78276 2557 89568

free=`echo $memoutput | gawk '{print $4+$7}'`

free = free + cache = 9736 + 2557 = 12,293

# echo $free
12293

When you run your check:

Code: Select all

/usr/local/nagios/libexec/custom_check_mem -n -w 20 -c 10

the output would be CRITICAL as:

free mem % = (free / total) x 100% = (12293 / 128675) x 100% = 9% (some rounding)

which is less than your critical threshold.

Can you try checking the "wide output" and running the check, verifying the output?

One more thing... the script has a hard coded path to sed (/bin/sed), but your path is /usr/bin/sed. I am not sure if this is causing the issue, but you could modify the plugin by entering the correct path to sed. Let us know if this helped.

[saber]

Hi,

There is nothing weird on the system and we have this issue on many servers too... Most of them have more than enough memory and this particular one has over 89GB in cached memory.

Sed binary seems fine too:

Code: Select all

# which sed
/usr/bin/sed




# sed --version
sed (GNU sed) 4.2.2
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jay Fenlason, Tom Lord, Ken Pizzini,
and Paolo Bonzini.
GNU sed home page: <http://www.gnu.org/software/sed/>.
General help using GNU software: <http://www.gnu.org/gethelp/>.
E-mail bug reports to: <[email protected]>.
Be sure to include the word ``sed'' somewhere in the ``Subject:'' field.




# /bin/sed --version
/bin/sed (GNU sed) 4.2.2
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jay Fenlason, Tom Lord, Ken Pizzini,
and Paolo Bonzini.
GNU sed home page: <http://www.gnu.org/software/sed/>.
General help using GNU software: <http://www.gnu.org/gethelp/>.
E-mail bug reports to: <[email protected]>.
Be sure to include the word ``sed'' somewhere in the ``Subject:'' field.

[lmiltchev]

OK, let's try one more time to view the resources, and run the check immediately after this. Please show the output of the command below.

Code: Select all

free -m -w;echo "";/usr/local/nagios/libexec/custom_check_mem -n -w 20 -c 10

If numbers don't make sense, we would have to schedule a remote session to further troubleshoot the issue.

[saber]

Hi,

Here is the output:

Code: Select all

# free -m -w;echo "";/usr/local/nagios/libexec/custom_check_mem -n -w 20 -c 10
              total        used        free      shared     buffers       cache   available
Mem:         128675       37392        9998         234       78707        2577       90273
Swap:          8191           0        8191

CRITICAL - 12573 / 128675 MB (9%) Free Memory, Used: 37394 MB, Shared: 234 MB, Buffers + Cached: 81284 MB | total=128675MB free=12573MB used=37394MB shared=234MB buffers_and_cached=81284MB

[lmiltchev]

This looks correct.

example-01.jpg

Your free memory + cache memory is 12573 (or pretty close to that).

free + cache = 9998 + 2577 = 12575

(free / total) x 100% = (12575 / 128675) x 100% = 9% (rounded)

9% < 10% (your critical threshold), that's why your status is CRITICAL.

You could adjust your warning and critical thresholds in order to eliminate unnecessary notifications.