Warning: fork() in my_system_r() failed for command "/bin/mv /var/nagiosramdisk/service-perfdata /var/nagiosramdisk/spoo

[g-nagios-support]

Hello,

I keep getting:

Warning: fork() in my_system_r() failed for command "/bin/mv /var/nagiosramdisk/service-perfdata /var/nagiosramdisk/spool/xidpe/1684184776.perfdata.service" - errno: Cannot allocate memory

It stops monitoring engine and then stops performance grapher. The system has enough memory at the moment. I restarted npcd service and restarted nagios, it would be ok for less than 10 mins, then nagios fails again.

I also saw:

WARNING: RLIMIT_NPROC is 63448, total max estimated processes is 63856! You should increase your limits (uli...its.conf)

checked /etc/security/limits.conf, I increased nproc to 100000, commented nagios maxprocs, reboot the system. I still got same errors. /etc/security/limits.conf looks like below now:

* hard nofile 10000
* soft nofile 10000

root hard nofile 10000
root soft nofile 10000

##* soft nproc 262144
* soft nproc 100000
##* hard nproc 262144
* hard nproc 100000
##nagios - maxprocs 14865

Could it be related to DB? I searched and tried different things, nothing helps so far. Can anyone advise please? Thank you!

After restart nagios, used swap increased quickly and then monitoring engine stopped.

Saw Timeout after 5 seconds, so increased TIMEOUT from 5 to 20 in /usr/local/nagios/etc/pnp/process_perfdata.cfg, didn't help.

Changed npcd log to level 2, the highest load was 8.78. The load is increased to 20.0 though. There were 2 files in /var/nagiosramdisk/spool/perfdata/.

Our other instances are running well while they have more hosts and services monitored with same config. This issue started last weekend. What did eat all SWAP suddenly?

[g-nagios-support]

Some DB tables crushed caused the issue. Clean and repair DB fixed the issue.