High CPU/Load on npcd

[cambevis]

That did it. I tried cutting down to 14 days, then 7, then 3, it would still go to a load of 6.
Deleting them all fixed it.
I don't know what was in those files but it did not like it.

Thanks!

Cam

[tgriep]

Those files contained the performance data for your host and service checks that are run on your system.
Shall I close the post for you?

[abrist]

Those files contain spooled performance data. The npcd daemon reaps them. If there are too many queued (due to npcd going down, not processing them on the configured interval), they stack up in the spool folder. Some systems can catch up from this happening, while others cannot (it depends on the average system load, resources, disk speeds, etc). Removing them is usually the best bet to get graphing working again, but you will end up with a gap in your graphs by doing so.

[cambevis]

This is now happening again after turning npcd back on.

[tgriep]

Did you remember to turn off the debugging for npcd?
Can you run the following and post back the results?

Code: Select all

ps aux
tail --100 /usr/local/nagios/var/npcd.log
tail --100 /usr/local/nagios/var/perfdata.log

[cambevis]

The debug is turned off. I do think the root of the issue is this error, there are 100s per second when npcd is turned on. I'm guessing the files never get cleaned out as these never exit cleanly.

See below this for tail of perfdata.

[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985886.perfdata.service'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985871.perfdata.host'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985916.perfdata.host'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985916.perfdata.service'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985901.perfdata.host'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985901.perfdata.service'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985931.perfdata.service'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985932.perfdata.host'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985961.perfdata.host'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985961.perfdata.service'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985946.perfdata.service'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985946.perfdata.host'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985991.perfdata.host'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433986006.perfdata.service'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985976.perfdata.service'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985976.perfdata.host'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433985991.perfdata.service'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433986022.perfdata.host'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433986036.perfdata.host'
[06-11-2015 09:16:11] NPCD: ERROR: Executed command exits with return code '2'
[06-11-2015 09:16:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1433986007.perfdata.host'


2015-06-08 13:32:32 [19580] [2] Using Config File /usr/local/nagios/etc/pnp/process_perfdata.cfg parameters
2015-06-08 13:32:32 [19579] [2] Using Config File /usr/local/nagios/etc/pnp/process_perfdata.cfg parameters
2015-06-08 13:32:32 [19580] [1] process_perfdata.pl-0.4.14 starting in BULK Mode called by NPCD
2015-06-08 13:32:32 [19580] [2] searching for /usr/local/nagios/var/spool/perfdata//1431822717.perfdata.host
2015-06-08 13:32:32 [19580] [2] renaming /usr/local/nagios/var/spool/perfdata//1431822717.perfdata.host to /usr/local/nagios/var/spool/perfdata//1431822717.perfdata.host-PID-19580 for bulk update
2015-06-08 13:32:32 [19579] [1] process_perfdata.pl-0.4.14 starting in BULK Mode called by NPCD
2015-06-08 13:32:32 [19579] [2] searching for /usr/local/nagios/var/spool/perfdata//1431822702.perfdata.service
2015-06-08 13:32:32 [19579] [2] renaming /usr/local/nagios/var/spool/perfdata//1431822702.perfdata.service to /usr/local/nagios/var/spool/perfdata//1431822702.perfdata.service-PID-19579 for bulk update
2015-06-08 13:32:32 [19588] [2] Using Config File /usr/local/nagios/etc/pnp/process_perfdata.cfg parameters
2015-06-08 13:32:32 [19588] [1] process_perfdata.pl-0.4.14 starting in BULK Mode called by NPCD
2015-06-08 13:32:32 [19588] [2] searching for /usr/local/nagios/var/spool/perfdata//1431822732.perfdata.service
2015-06-08 13:32:32 [19588] [2] renaming /usr/local/nagios/var/spool/perfdata//1431822732.perfdata.service to /usr/local/nagios/var/spool/perfdata//1431822732.perfdata.service-PID-19588 for bulk update
2015-06-08 13:32:32 [19589] [2] Using Config File /usr/local/nagios/etc/pnp/process_perfdata.cfg parameters
2015-06-08 13:32:32 [19589] [1] process_perfdata.pl-0.4.14 starting in BULK Mode called by NPCD
2015-06-08 13:32:32 [19589] [2] searching for /usr/local/nagios/var/spool/perfdata//1431822747.perfdata.host
2015-06-08 13:32:32 [19589] [2] renaming /usr/local/nagios/var/spool/perfdata//1431822747.perfdata.host to /usr/local/nagios/var/spool/perfdata//1431822747.perfdata.host-PID-19589 for bulk update
2015-06-08 13:32:32 [19585] [2] Using Config File /usr/local/nagios/etc/pnp/process_perfdata.cfg parameters
2015-06-08 13:32:32 [19584] [2] Using Config File /usr/local/nagios/etc/pnp/process_perfdata.cfg parameters
2015-06-08 13:32:32 [19585] [1] process_perfdata.pl-0.4.14 starting in BULK Mode called by NPCD
2015-06-08 13:32:32 [19585] [2] searching for /usr/local/nagios/var/spool/perfdata//1431822732.perfdata.host
2015-06-08 13:32:32 [19585] [2] renaming /usr/local/nagios/var/spool/perfdata//1431822732.perfdata.host to /usr/local/nagios/var/spool/perfdata//1431822732.perfdata.host-PID-19585 for bulk update
2015-06-08 13:32:32 [19584] [1] process_perfdata.pl-0.4.14 starting in BULK Mode called by NPCD
2015-06-08 13:32:32 [19584] [2] searching for /usr/local/nagios/var/spool/perfdata//1431822717.perfdata.service
2015-06-08 13:32:32 [19584] [2] renaming /usr/local/nagios/var/spool/perfdata//1431822717.perfdata.service to /usr/local/nagios/var/spool/perfdata//1431822717.perfdata.service-PID-19584 for bulk update
2015-06-08 13:32:32 [19590] [2] Using Config File /usr/local/nagios/etc/pnp/process_perfdata.cfg parameters
2015-06-08 13:32:32 [19590] [1] process_perfdata.pl-0.4.14 starting in BULK Mode called by NPCD
2015-06-08 13:32:32 [19590] [2] searching for /usr/local/nagios/var/spool/perfdata//1431822747.perfdata.service
2015-06-08 13:32:32 [19590] [2] renaming /usr/local/nagios/var/spool/perfdata//1431822747.perfdata.service to /usr/local/nagios/var/spool/perfdata//1431822747.perfdata.service-PID-19590 for bulk update
2015-06-08 13:32:32 [19594] [2] Using Config File /usr/local/nagios/etc/pnp/process_perfdata.cfg parameters
2015-06-08 13:32:32 [19594] [1] process_perfdata.pl-0.4.14 starting in BULK Mode called by NPCD
2015-06-08 13:32:32 [19594] [2] searching for /usr/local/nagios/var/spool/perfdata//1431822762.perfdata.host
2015-06-08 13:32:32 [19594] [2] renaming /usr/local/nagios/var/spool/perfdata//1431822762.perfdata.host to /usr/local/nagios/var/spool/perfdata//1431822762.perfdata.host-PID-19594 for bulk update
2015-06-08 13:32:32 [19595] [2] Using Config File /usr/local/nagios/etc/pnp/process_perfdata.cfg parameters
2015-06-08 13:32:32 [19595] [1] process_perfdata.pl-0.4.14 starting in BULK Mode called by NPCD
2015-06-08 13:32:32 [19595] [2] searching for /usr/local/nagios/var/spool/perfdata//1431822762.perfdata.service
2015-06-08 13:32:32 [19595] [2] renaming /usr/local/nagios/var/spool/perfdata//1431822762.perfdata.service to /usr/local/nagios/var/spool/perfdata//1431822762.perfdata.service-PID-19595 for bulk update
2015-06-08 13:32:32 [19598] [2] Using Config File /usr/local/nagios/etc/pnp/process_perfdata.cfg parameters
2015-06-08 13:32:32 [19598] [1] process_perfdata.pl-0.4.14 starting in BULK Mode called by NPCD
2015-06-08 13:32:32 [19598] [2] searching for /usr/local/nagios/var/spool/perfdata//1431822777.perfdata.host
2015-06-08 13:32:32 [19598] [2] renaming /usr/local/nagios/var/spool/perfdata//1431822777.perfdata.host to /usr/local/nagios/var/spool/perfdata//1431822777.perfdata.host-PID-19598 for bulk update
2015-06-08 13:32:32 [19600] [2] Using Config File /usr/local/nagios/etc/pnp/process_perfdata.cfg parameters
2015-06-08 13:32:32 [19599] [2] Using Config File /usr/local/nagios/etc/pnp/process_perfdata.cfg parameters
2015-06-08 13:32:32 [19600] [1] process_perfdata.pl-0.4.14 starting in BULK Mode called by NPCD
2015-06-08 13:32:32 [19600] [2] searching for /usr/local/nagios/var/spool/perfdata//1431822792.perfdata.host
2015-06-08 13:32:32 [19600] [2] renaming /usr/local/nagios/var/spool/perfdata//1431822792.perfdata.host to /usr/local/nagios/var/spool/perfdata//1431822792.perfdata.host-PID-19600 for bulk update
2015-06-08 13:32:32 [19599] [1] process_perfdata.pl-0.4.14 starting in BULK Mode called by NPCD
2015-06-08 13:32:32 [19599] [2] searching for /usr/local/nagios/var/spool/perfdata//1431822777.perfdata.service
2015-06-08 13:32:32 [19599] [2] renaming /usr/local/nagios/var/spool/perfdata//1431822777.perfdata.service to /usr/local/nagios/var/spool/perfdata//1431822777.perfdata.service-PID-19599 for bulk update

[abrist]

What are the permissions on the perfdata files?

Code: Select all

ls -la /usr/local/nagios/var/spool/perfdata | head

[cambevis]

drwxr-xr-x. 2 nagios nagios 14151680 Jun 11 11:11 .
drwxr-xr-x. 5 root root 4096 Jun 6 2012 ..
-rw-r--r-- 1 nagios nagios 316 Jun 10 10:03 1433948626.perfdata.host
-rw-r--r-- 1 nagios nagios 2448 Jun 10 10:03 1433948626.perfdata.service
-rw-r--r-- 1 nagios nagios 618 Jun 10 10:03 1433948641.perfdata.host
-rw-r--r-- 1 nagios nagios 2343 Jun 10 10:04 1433948642.perfdata.service
-rw-r--r-- 1 nagios nagios 944 Jun 10 10:04 1433948656.perfdata.host
-rw-r--r-- 1 nagios nagios 2332 Jun 10 10:04 1433948656.perfdata.service
-rw-r--r-- 1 nagios nagios 0 Jun 10 10:04 1433948671.perfdata.host

[cambevis]

I wonder what happened around 5/16? There's nothing in the logs, and we haven't made any system changes recently.
I did upgrade on 5/8 to current version.

[abrist]

Have you ever implemented a ramdisk on this server?
What are the permissions on the directories:
ls -lad /usr/local/nagios/var/spool/perfdata

Code: Select all

ls -lad /usr/local/nagios/var/spool/
ls -lad /usr/local/nagios/var/