Nagios XI swap usage

[kalyanpabolu]

Hello Team,

We are using Nagios XI 5.7.3 and Centos * server. We can see the swap usage for Nagios XI server is continuously critical.

Below are the details:

[root@HO1-NAGIOSXI ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 1.6G 15G 10% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/VG_Main-root 49G 11G 37G 22% /
/dev/mapper/VG_Main-usr 32G 14G 17G 45% /usr
/dev/mapper/VG_Main-home 12G 41M 12G 1% /home
/dev/mapper/VG_Main-var 22G 15G 6.3G 70% /var
/dev/sda1 2.0G 190M 1.7G 11% /boot
tmpfs 3.2G 0 3.2G 0% /run/user/1000
tmpfs 3.2G 0 3.2G 0% /run/user/0
[root@HO1-NAGIOSXI ~]#



[root@HO1-NAGIOSXI ~]# free -m
total used free shared buff/cache available
Mem: 31992 25530 676 1552 5786 4452
Swap: 10239 10220 19
[root@HO1-NAGIOSXI ~]#

[root@HO1-NAGIOSXI ~]# uname -a
Linux HO1-NAGIOSXI 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@HO1-NAGIOSXI ~]#


[root@HO1-NAGIOSXI ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

[root@HO1-NAGIOSXI ~]#

What exactly swap usage is consumed for? Could you please suggest ways to resolve it?

[benjaminsmith]

Hi,

The swap space on the disk is used when RAM memory is full. The system will move inactive memory into swap space to free up resources. What is the check load on this server, the total number of host and services checks?

I would run a top command to see what processes are using memory but overall it would be best to add more RAM to the server to resolve this issue.

Code: Select all

top -o +%MEM

Ben

[kalyanpabolu]

Hello,

Number of host: 2887
Number of services: 5285

[root@vmaz-nagiosxi libexec]# ./check_load 10.44.3.8
CRITICAL - load average: 2.90, 3.95, 4.10|load1=2.900;0.000;0.000;0; load5=3.950;0.000;0.000;0; load15=4.100;0.000;0.000;0;
[root@vmaz-nagiosxi libexec]#

Is there any way to clear the swap usage?

[jdunitz]

You could do a "swapoff -av", wait a minute or so, then do a "swapon -av".

Note, though, that if you happen to have all your memory consumed by actual processes when you do this, you might make the system temporarily unusable, and may even need to reboot to get it back. Under normal circumstances, though, it should be fine to to this.

--Jeffrey

[kalyanpabolu]

Hello,

Below are the processes we can see when we ran top -o +%MEM command.

top - 17:53:34 up 33 days, 4:36, 1 user, load average: 2.26, 3.76, 4.78
Tasks: 293 total, 2 running, 291 sleeping, 0 stopped, 0 zombie
%Cpu(s): 13.7 us, 2.7 sy, 0.0 ni, 83.1 id, 0.0 wa, 0.4 hi, 0.1 si, 0.0 st
MiB Mem : 32136.7 total, 1673.1 free, 27278.6 used, 3185.0 buff/cache
MiB Swap: 15356.0 total, 42.1 free, 15313.9 used. 3099.9 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1793381 apache 20 0 2383244 1.2g 5724 S 0.0 3.7 99:12.86 php-fpm
1788342 apache 20 0 2486452 1.2g 5784 S 0.0 3.7 99:18.48 php-fpm
1799520 apache 20 0 2386572 1.1g 5940 S 0.0 3.5 99:25.14 php-fpm
1791823 apache 20 0 2379720 1.1g 5752 S 0.0 3.5 99:12.88 php-fpm
1790506 apache 20 0 2409324 1.0g 5828 S 0.0 3.3 98:50.62 php-fpm
1788809 apache 20 0 2396800 1.0g 5728 S 0.0 3.2 99:08.79 php-fpm
3716516 apache 20 0 2569280 1.0g 5740 S 0.0 3.2 102:07.20 php-fpm
1789707 apache 20 0 2430196 988.7m 5904 S 0.0 3.1 98:55.81 php-fpm
1784732 apache 20 0 2398468 986.2m 5744 S 0.0 3.1 98:27.06 php-fpm
1789466 apache 20 0 2391300 980.7m 5824 S 0.0 3.1 98:53.36 php-fpm
1787441 apache 20 0 2387812 971364 5736 S 1.7 3.0 98:47.23 php-fpm
1800707 apache 20 0 2382868 962704 5944 S 0.0 2.9 99:10.33 php-fpm
1800410 apache 20 0 2408992 926060 5908 S 0.0 2.8 98:37.44 php-fpm
1789659 apache 20 0 2463576 925884 5764 S 0.0 2.8 100:45.07 php-fpm
1787656 apache 20 0 2375032 921608 5780 S 0.0 2.8 98:38.10 php-fpm
1633148 mysql 20 0 5106580 921440 0 S 4.7 2.8 649:22.74 mysqld
1787469 apache 20 0 2405332 884616 5752 S 0.0 2.7 98:36.77 php-fpm
1850387 apache 20 0 1876668 629380 5928 S 1.3 1.9 70:55.27 php-fpm
688531 apache 20 0 1281380 627484 5732 S 3.7 1.9 41:32.21 php-fpm
2952499 apache 20 0 1227216 578260 5708 S 0.0 1.8 38:53.05 php-fpm
3804629 apache 20 0 1203180 556676 6080 S 0.0 1.7 36:49.99 php-fpm
1765706 apache 20 0 1068972 549000 5880 S 0.0 1.7 26:13.20 php-fpm
866995 apache 20 0 1173688 548420 5748 S 0.0 1.7 33:16.70 php-fpm
3215074 apache 20 0 1078580 538404 5884 S 0.0 1.6 29:26.53 php-fpm
4165274 apache 20 0 1051204 532776 5764 S 4.3 1.6 27:50.91 php-fpm
519458 apache 20 0 1147920 522184 5752 S 0.3 1.6 34:08.91 php-fpm
937652 apache 20 0 1137932 490680 5804 S 0.0 1.5 32:58.69 php-fpm
932708 apache 20 0 1129212 476700 5820 S 0.0 1.4 32:22.54 php-fpm
233642 apache 20 0 990872 472612 5764 S 0.0 1.4 22:31.51 php-fpm
3919100 apache 20 0 976624 458592 5940 S 0.0 1.4 19:14.40 php-fpm
3870359 apache 20 0 917220 399216 5820 S 0.0 1.2 16:13.30 php-fpm
1368361 apache 20 0 891528 373444 5724 S 0.0 1.1 14:52.13 php-fpm
684717 apache 20 0 840840 322792 5904 S 0.0 1.0 10:25.85 php-fpm
1397781 apache 20 0 769372 251072 5724 S 0.0 0.8 8:41.78 php-fpm
3132703 apache 20 0 756096 237812 5720 S 0.0 0.7 6:47.52 php-fpm
697 root 20 0 406104 223300 179588 S 0.0 0.7 154:18.22 systemd-journal
3054863 nagios 20 0 336076 198784 9920 S 0.7 0.6 0:09.15 mrtg


Please let us know how we can optimize the apache processes and php processes.

[ssax]

Please PM me a copy of your profile, you can download it from Admin > System Profile > Download Profile button.

Attach these files:

Code: Select all

/etc/php.ini
/etc/my.cnf.d/mysql-server.cnf

Are you seeing any errors in this file?

Code: Select all

/var/log/php-fpm/error.log

Additionally, please send the output of these commands (as root):
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

This next command may fail, that's okay, not all systems run postgresql, send the output anyways:

Code: Select all

echo "SELECT relname as Table, pg_size_pretty(pg_total_relation_size(relid)) As Size, pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as ExternalSize FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;" | psql nagiosxi nagiosxi