Increasing swap usage for Nagios XI host

[atc0005]

Hi,

We recently migrated from a Nagios XI 5.6.14 installation running on a RHEL 6 OS to a 5.8.7 instance running on RHEL 8.

We provisioned the new system with 12 GB of memory and 6 GB of swap and enabled nightly local backups (separate logical volume for created backups).

From what I can tell, the original resource configuration exceeded the system requirements noted on https://support.nagios.com/kb/article.php?id=12 and https://support.nagios.com/kb/article.p ... ategory=83.

We're monitoring 438 hosts, 475 services at present.

We deployed the system Dec 16th 2021 and the nagios service was targeted by OOM (along with mysqld) around January 3rd 2022.

The workaround was to reboot, increase memory and swap temporarily while we dug further into the issue. The system is currently provisioned with 8 vCPUs, 14 GB RAM and 10 GB swap.

One of the enabled service checks for the Nagios XI instance is to monitor swap usage. We're seeing a marked increase in swap usage each day during scheduled backups, then a slight recovery the next morning when cron jobs run. We moved the backup time from 12 am to 3 pm and observed the same drop.

nagiosxi-swap-usage-last-7-days-2022-01-24.png

Modifications applied thus far:

• modified php-fpm pm.max_children (50 to 15)
• modified php-fpm pm.max_spare_servers (35 to 8)
• modified mysqld to disable name resolution for mysql client connections

I've looked at various MySQL tuning options, but I'd held off changing any aside from disabling name resolution.

When using the mysqltuner.pl script from https://github.com/major/MySQLTuner-perl it cautions that MySQL memory usage has the potential to greatly exceed system memory:

Code: Select all

[!!] Maximum reached memory usage: 29.4G (217.66% of installed RAM)
[!!] Maximum possible memory usage: 38.2G (282.81% of installed RAM)

These are the settings currently applied in /etc/my.cnf.d/mysql-server.cnf:

Code: Select all

tmp_table_size=64M
max_heap_table_size=64M
key_buffer_size=32M
table_open_cache=32
innodb_file_per_table=1
binlog_expire_logs_seconds=86400
sql_mode=NO_ENGINE_SUBSTITUTION
max_allowed_packet=256M

I have not modified those values.

Our goal at this point is to reign in memory usage without unduly impacting Nagios XI operations. Presumably this involves further tuning of php-fpm and MySQL, but before I go any further down those paths I wanted to reach out and get your assistance first.

Thanks in advance for your help.

[ssax]

When the system is under high swap, please send the output of these commands:

Code: Select all

ps aux
sar -A

Try following the instructions in the attached file, these are usually the changes I have to do on EL8 when I see this issue.

php-fpm timeout issue.txt

Let us know if that resolves the issue.

Thank you!

[atc0005]

Thanks for the reply and for the suggested changes.

I'll apply them and watch the box for a few days before responding back.

[kfanselow]

Excellent - We'll keep the thread open then.

[atc0005]

Hi,

Our system has been a lot better since applying the changes. Swap is barely touched at all at this point.

Please consider this issue resolved. I'll open a new one if the problem resurfaces in the future.

Thanks for your help.

[kfanselow]

Excellent - We're glad to hear you got it resolved !!!

We'll get the feedback to ssax and lock the thread.

Thanks and Best Regards,
Keith