checkresults issue

[sandeepatil]

Hi,

We faced issue with checkresults directory keep max data and partition was 100% full. After removing all data from checkresults, nagios working fine.

During troubleshooting inode was shown 100% full.

Please explain the issue, Why large amount of files are created in checkresults directory ? and What is preventive steps for avoiding the same ?

Thanks,
Sandeep.

[dwasswa]

Hi @sandeepatil,

Issues with checkresults can be caused by a couple of things:

1.npcd daemon not running. To check that, please run the commands below:

service npcd restart then service npcd status


Please attach npcd.log file that contains your npcd logs. This file is located at /usr/local/nagios/var

Please attach perfdataproc.log file located at /usr/local/nagiosxi/var

2.Have you installed ramdisk on your Nagios XI server recently?

3.Expired Nagios user account. Check this with this command...chage -l nagios

4.Wrong group permissions or permission changes

5. enabled selinux; check this by running this command.../usr/sbin/sestatus

Lastly, please PM your system profile by following the steps below:

Go to Admin >> System Profile >> Download Profile

Download_system_profile.PNG

[lmiltchev]

This could be caused by permissions on the checkresults directory. Please check out the KB article below:
https://support.nagios.com/kb/article/n ... d-416.html

Are you using a ramdisk with Nagios XI? Can you post the nagios.cfg file, so that we could review it?

What is the output of the following command?

Code: Select all

ls /usr/local/nagios/var/spool/checkresults | wc -l

... or this one (if you are using a ramdisk):

Code: Select all

ls /var/nagiosramdisk/spool/checkresults/ | wc -l

[sandeepatil]

We are not using ramdisk with Nagios XI.

Please find status.

Code: Select all

[nagios@abcd123]$ ls /usr/local/nagios/var/spool/checkresults | wc -l
360

Nagios user account not expire.

selinux disabled.

[lmiltchev]

1. Can you run the following commands and show the output in code wraps?

Code: Select all

service npcd restart
service npcd status
ps -ef | grep perf
ls -lad /usr/local/nagios/var/spool/checkresults
tail -100 /usr/local/nagios/var/npcd.log
tail -100 /usr/local/nagios/var/perfdataproc.log

Note: The commands above will show us if ncpd and perdataproc are running, what are the permissions of the checkresults directory, and if there are any clues in the log files.

2. PM me or any other member of the Nagios Support Team your profile (profile.zip).

Note: We will review your configs and various logs for issues.

3. Also, run the following command:

Code: Select all

watch -n 1 'ls /usr/local/nagios/var/spool/checkresults | wc -l'

watch the output for a while (1-2 min), then exit by pressing Ctrl+c.

Let us know if the number of the files in the checkresults directory is increasing or decreasing over time.

[sandeepatil]

Please find requested details,

ps -ef | grep perf

Code: Select all

$ ps -ef | grep perf
root        177      2  0 Nov20 ?        00:00:00 [perf]
nagios     9955   9930  0 11:05 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios     9958   9955  0 11:05 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
sp470a    10227  27064  0 11:05 pts/9    00:00:00 grep --color=auto perf

ls -lad /usr/local/nagios/var/spool/checkresults

Code: Select all

$ tail -100 /usr/local/nagios/var/npcd.log
[11-21-2017 07:15:00] NPCD: WARN: MAX load reached: load 121.680000/10.000000 at i=1
[11-21-2017 07:15:15] NPCD: WARN: MAX load reached: load 122.130000/10.000000 at i=1
[11-21-2017 07:15:30] NPCD: WARN: MAX load reached: load 122.950000/10.000000 at i=1
[11-21-2017 07:15:45] NPCD: WARN: MAX load reached: load 123.340000/10.000000 at i=1
[11-21-2017 07:16:00] NPCD: WARN: MAX load reached: load 123.480000/10.000000 at i=1
[11-21-2017 07:16:15] NPCD: WARN: MAX load reached: load 124.060000/10.000000 at i=1
[11-21-2017 07:16:30] NPCD: WARN: MAX load reached: load 124.900000/10.000000 at i=1
[11-21-2017 07:16:45] NPCD: WARN: MAX load reached: load 125.650000/10.000000 at i=1
[11-21-2017 07:17:00] NPCD: WARN: MAX load reached: load 126.170000/10.000000 at i=1
[11-21-2017 07:17:15] NPCD: WARN: MAX load reached: load 128.730000/10.000000 at i=1
[11-21-2017 07:17:30] NPCD: WARN: MAX load reached: load 130.050000/10.000000 at i=1
[11-21-2017 07:17:45] NPCD: WARN: MAX load reached: load 129.680000/10.000000 at i=1
[11-21-2017 07:18:00] NPCD: WARN: MAX load reached: load 129.560000/10.000000 at i=1
[11-21-2017 07:18:15] NPCD: WARN: MAX load reached: load 130.090000/10.000000 at i=1
[11-21-2017 07:18:31] NPCD: WARN: MAX load reached: load 130.440000/10.000000 at i=1
[11-21-2017 07:18:46] NPCD: WARN: MAX load reached: load 130.320000/10.000000 at i=1
[11-21-2017 07:19:01] NPCD: WARN: MAX load reached: load 130.260000/10.000000 at i=1
[11-21-2017 07:19:16] NPCD: WARN: MAX load reached: load 130.440000/10.000000 at i=1
[11-21-2017 07:19:31] NPCD: WARN: MAX load reached: load 131.020000/10.000000 at i=1
[11-21-2017 07:19:46] NPCD: WARN: MAX load reached: load 131.800000/10.000000 at i=1
[11-21-2017 07:20:01] NPCD: WARN: MAX load reached: load 132.130000/10.000000 at i=1
[11-21-2017 07:20:16] NPCD: WARN: MAX load reached: load 133.390000/10.000000 at i=1
[11-21-2017 07:20:31] NPCD: WARN: MAX load reached: load 133.820000/10.000000 at i=1
[11-21-2017 07:20:46] NPCD: WARN: MAX load reached: load 133.850000/10.000000 at i=1
[11-21-2017 07:21:01] NPCD: WARN: MAX load reached: load 135.630000/10.000000 at i=1
[11-21-2017 07:21:16] NPCD: WARN: MAX load reached: load 136.280000/10.000000 at i=1
[11-21-2017 07:21:31] NPCD: WARN: MAX load reached: load 136.530000/10.000000 at i=1
[11-21-2017 07:21:46] NPCD: WARN: MAX load reached: load 136.730000/10.000000 at i=1
[11-21-2017 07:22:01] NPCD: WARN: MAX load reached: load 136.970000/10.000000 at i=1
[11-21-2017 07:22:16] NPCD: WARN: MAX load reached: load 137.540000/10.000000 at i=1
[11-21-2017 07:22:31] NPCD: WARN: MAX load reached: load 138.650000/10.000000 at i=1
[11-21-2017 07:22:46] NPCD: WARN: MAX load reached: load 138.440000/10.000000 at i=1
[11-21-2017 07:23:01] NPCD: WARN: MAX load reached: load 136.600000/10.000000 at i=1
[11-21-2017 07:23:16] NPCD: WARN: MAX load reached: load 132.470000/10.000000 at i=1
[11-21-2017 07:23:31] NPCD: WARN: MAX load reached: load 129.310000/10.000000 at i=1
[11-21-2017 07:23:47] NPCD: WARN: MAX load reached: load 127.090000/10.000000 at i=1
[11-21-2017 07:24:02] NPCD: WARN: MAX load reached: load 125.950000/10.000000 at i=1
[11-21-2017 07:24:17] NPCD: WARN: MAX load reached: load 124.460000/10.000000 at i=1
[11-21-2017 07:24:32] NPCD: WARN: MAX load reached: load 123.800000/10.000000 at i=1
[11-21-2017 07:24:47] NPCD: WARN: MAX load reached: load 122.510000/10.000000 at i=1
[11-21-2017 07:25:02] NPCD: WARN: MAX load reached: load 121.520000/10.000000 at i=1
[11-21-2017 07:25:17] NPCD: WARN: MAX load reached: load 123.800000/10.000000 at i=1
[11-21-2017 07:25:32] NPCD: WARN: MAX load reached: load 127.040000/10.000000 at i=1
[11-21-2017 07:25:47] NPCD: WARN: MAX load reached: load 124.150000/10.000000 at i=1
[11-21-2017 07:26:02] NPCD: WARN: MAX load reached: load 122.170000/10.000000 at i=1
[11-21-2017 07:26:17] NPCD: WARN: MAX load reached: load 120.430000/10.000000 at i=1
[11-21-2017 07:26:32] NPCD: WARN: MAX load reached: load 119.400000/10.000000 at i=1
[11-21-2017 07:26:47] NPCD: WARN: MAX load reached: load 119.550000/10.000000 at i=1
[11-21-2017 07:27:02] NPCD: WARN: MAX load reached: load 119.800000/10.000000 at i=1
[11-21-2017 07:27:17] NPCD: WARN: MAX load reached: load 119.270000/10.000000 at i=1
[11-21-2017 07:27:32] NPCD: WARN: MAX load reached: load 118.900000/10.000000 at i=1
[11-21-2017 07:27:47] NPCD: WARN: MAX load reached: load 118.400000/10.000000 at i=1
[11-21-2017 07:28:02] NPCD: WARN: MAX load reached: load 118.220000/10.000000 at i=1
[11-21-2017 07:28:03] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-21-2017 08:19:09] NPCD: npcd Daemon (0.4.14) started with PID=78114
[11-21-2017 08:19:09] NPCD: Please have a look at 'npcd -V' to get license information
[11-21-2017 08:19:09] NPCD: HINT: load_threshold is enabled - ('10.000000')
[11-21-2017 11:54:46] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-21-2017 11:54:46] NPCD: npcd Daemon (0.4.14) started with PID=2531
[11-21-2017 11:54:46] NPCD: Please have a look at 'npcd -V' to get license information
[11-21-2017 11:54:46] NPCD: HINT: load_threshold is enabled - ('10.000000')
[11-21-2017 12:39:31] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-21-2017 12:39:31] NPCD: npcd Daemon (0.4.14) started with PID=34493
[11-21-2017 12:39:31] NPCD: Please have a look at 'npcd -V' to get license information
[11-21-2017 12:39:31] NPCD: HINT: load_threshold is enabled - ('10.000000')
[11-21-2017 12:52:51] NPCD: WARN: MAX load reached: load 10.100000/10.000000 at i=0
[11-21-2017 12:53:06] NPCD: WARN: MAX load reached: load 10.120000/10.000000 at i=1
[11-21-2017 12:53:21] NPCD: WARN: MAX load reached: load 10.850000/10.000000 at i=1
[11-21-2017 12:53:36] NPCD: WARN: MAX load reached: load 10.220000/10.000000 at i=1
[11-21-2017 12:53:51] NPCD: WARN: MAX load reached: load 10.090000/10.000000 at i=1
[11-21-2017 12:54:06] NPCD: WARN: MAX load reached: load 10.070000/10.000000 at i=1
[11-21-2017 12:54:21] NPCD: WARN: MAX load reached: load 10.310000/10.000000 at i=1
[11-21-2017 13:46:40] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-21-2017 13:46:40] NPCD: npcd Daemon (0.4.14) started with PID=69973
[11-21-2017 13:46:40] NPCD: Please have a look at 'npcd -V' to get license information
[11-21-2017 13:46:40] NPCD: HINT: load_threshold is enabled - ('10.000000')
[11-21-2017 14:09:46] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-21-2017 14:15:53] NPCD: npcd Daemon (0.4.14) started with PID=98403
[11-21-2017 14:15:53] NPCD: Please have a look at 'npcd -V' to get license information
[11-21-2017 14:15:53] NPCD: HINT: load_threshold is enabled - ('10.000000')
[11-21-2017 14:41:13] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-21-2017 14:41:13] NPCD: npcd Daemon (0.4.14) started with PID=113553
[11-21-2017 14:41:13] NPCD: Please have a look at 'npcd -V' to get license information
[11-21-2017 14:41:13] NPCD: HINT: load_threshold is enabled - ('10.000000')
[11-21-2017 14:46:01] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-21-2017 14:46:01] NPCD: npcd Daemon (0.4.14) started with PID=117277
[11-21-2017 14:46:01] NPCD: Please have a look at 'npcd -V' to get license information
[11-21-2017 14:46:01] NPCD: HINT: load_threshold is enabled - ('10.000000')
[11-21-2017 14:47:52] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-21-2017 14:48:06] NPCD: npcd Daemon (0.4.14) started with PID=119065
[11-21-2017 14:48:06] NPCD: Please have a look at 'npcd -V' to get license information
[11-21-2017 14:48:06] NPCD: HINT: load_threshold is enabled - ('10.000000')
[11-21-2017 14:58:57] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-21-2017 14:58:57] NPCD: npcd Daemon (0.4.14) started with PID=124355
[11-21-2017 14:58:57] NPCD: Please have a look at 'npcd -V' to get license information
[11-21-2017 14:58:57] NPCD: HINT: load_threshold is enabled - ('10.000000')
[11-22-2017 08:57:18] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-22-2017 08:57:18] NPCD: npcd Daemon (0.4.14) started with PID=73070
[11-22-2017 08:57:18] NPCD: Please have a look at 'npcd -V' to get license information
[11-22-2017 08:57:18] NPCD: HINT: load_threshold is enabled - ('10.000000')

tail -100 /usr/local/nagios/var/perfdataproc.log

Code: Select all

$: tail -100 /usr/local/nagios/var/perfdata.log
2017-02-22 11:59:52 [55498] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-02-22 11:59:52 [55499] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-02-22 11:59:52 [55498] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-02-22 11:59:52 [55499] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-02-22 11:59:52 [55498] [0] *** TIMEOUT: Please check your npcd.cfg
2017-02-22 11:59:52 [55499] [0] *** TIMEOUT: Please check your npcd.cfg
2017-02-22 11:59:52 [55498] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1487782769.perfdata.host-PID-55498 deleted
2017-02-22 11:59:52 [55499] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1487782769.perfdata.service-PID-55499 deleted
2017-02-22 11:59:52 [55498] [0] *** Timeout while processing Host: "" Service: ""
2017-02-22 11:59:52 [55499] [0] *** Timeout while processing Host: "" Service: ""
2017-02-22 11:59:52 [55498] [0] *** process_perfdata.pl terminated on signal ALRM
2017-02-22 11:59:52 [55499] [0] *** process_perfdata.pl terminated on signal ALRM
2017-02-23 01:07:41 [122913] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-02-23 01:07:41 [122913] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-02-23 01:07:41 [122913] [0] *** TIMEOUT: Please check your npcd.cfg
2017-02-23 01:07:41 [122912] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-02-23 01:07:41 [122913] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1487830032.perfdata.service-PID-122913 deleted
2017-02-23 01:07:41 [122912] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-02-23 01:07:41 [122912] [0] *** TIMEOUT: Please check your npcd.cfg
2017-02-23 01:07:41 [122915] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-02-23 01:07:41 [122913] [0] *** Timeout while processing Host: "" Service: ""
2017-02-23 01:07:41 [122914] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-02-23 01:07:41 [122913] [0] *** process_perfdata.pl terminated on signal ALRM
2017-02-23 01:07:41 [122915] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-02-23 01:07:41 [122914] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-02-23 01:07:41 [122912] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1487830032.perfdata.host-PID-122912 deleted
2017-02-23 01:07:41 [122915] [0] *** TIMEOUT: Please check your npcd.cfg
2017-02-23 01:07:41 [122914] [0] *** TIMEOUT: Please check your npcd.cfg
2017-02-23 01:07:41 [122912] [0] *** Timeout while processing Host: "" Service: ""
2017-02-23 01:07:41 [122915] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1487830047.perfdata.service-PID-122915 deleted
2017-02-23 01:07:41 [122914] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1487830047.perfdata.host-PID-122914 deleted
2017-02-23 01:07:41 [122912] [0] *** process_perfdata.pl terminated on signal ALRM
2017-02-23 01:07:41 [122915] [0] *** Timeout while processing Host: "" Service: ""
2017-02-23 01:07:41 [122914] [0] *** Timeout while processing Host: "" Service: ""
2017-02-23 01:07:41 [122915] [0] *** process_perfdata.pl terminated on signal ALRM
2017-02-23 01:07:41 [122914] [0] *** process_perfdata.pl terminated on signal ALRM
2017-03-10 15:47:29 [22708] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-03-10 15:47:29 [22709] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-03-10 15:47:29 [22708] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-03-10 15:47:29 [22708] [0] *** TIMEOUT: Please check your npcd.cfg
2017-03-10 15:47:29 [22709] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-03-10 15:47:29 [22709] [0] *** TIMEOUT: Please check your npcd.cfg
2017-03-10 15:47:29 [22708] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1489178832.perfdata.host-PID-22708 deleted
2017-03-10 15:47:29 [22709] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1489178832.perfdata.service-PID-22709 deleted
2017-03-10 15:47:29 [22708] [0] *** Timeout while processing Host: "" Service: ""
2017-03-10 15:47:29 [22708] [0] *** process_perfdata.pl terminated on signal ALRM
2017-03-10 15:47:29 [22709] [0] *** Timeout while processing Host: "" Service: ""
2017-03-10 15:47:29 [22709] [0] *** process_perfdata.pl terminated on signal ALRM
2017-03-12 23:37:48 [89092] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-03-12 23:37:48 [89092] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-03-12 23:37:48 [89092] [0] *** TIMEOUT: Please check your npcd.cfg
2017-03-12 23:37:48 [89092] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1489376247.perfdata.host-PID-89092 deleted
2017-03-12 23:37:48 [89093] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-03-12 23:37:48 [89093] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-03-12 23:37:48 [89092] [0] *** Timeout while processing Host: "" Service: ""
2017-03-12 23:37:48 [89092] [0] *** process_perfdata.pl terminated on signal ALRM
2017-03-12 23:37:48 [89093] [0] *** TIMEOUT: Please check your npcd.cfg
2017-03-12 23:37:48 [89093] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1489376247.perfdata.service-PID-89093 deleted
2017-03-12 23:37:48 [89093] [0] *** Timeout while processing Host: "" Service: ""
2017-03-12 23:37:48 [89093] [0] *** process_perfdata.pl terminated on signal ALRM
2017-06-08 12:31:54 [57173] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1496939510.perfdata.service-PID-57173 deleted
2017-06-08 12:31:54 [57173] [0] *** Timeout while processing Host: "csdev61.sbc.com" Service: "Filespace_STAR-extra"
2017-06-08 12:31:54 [57173] [0] *** process_perfdata.pl terminated on signal BUS
2017-11-17 11:03:41 [22173] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-11-17 11:03:41 [22173] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-11-17 11:03:41 [22173] [0] *** TIMEOUT: Please check your npcd.cfg
2017-11-17 11:03:41 [22173] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1510783605.perfdata.service-PID-22173 deleted
2017-11-17 11:03:41 [22173] [0] *** Timeout while processing Host: "zlp22389.vci.att.com" Service: "Filespace_STAR-root"
2017-11-17 11:03:41 [22173] [0] *** process_perfdata.pl terminated on signal ALRM
2017-11-18 11:28:06 [128831] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-11-18 11:28:06 [128831] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-11-18 11:28:06 [128831] [0] *** TIMEOUT: Please check your npcd.cfg
2017-11-18 11:28:06 [128831] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1511021791.perfdata.service-PID-128831 deleted
2017-11-18 11:28:06 [128831] [0] *** Timeout while processing Host: "hixa804.vtil.att.com" Service: "Filespace_STAR-opt-openv"
2017-11-18 11:28:06 [128831] [0] *** process_perfdata.pl terminated on signal ALRM
2017-11-20 11:55:05 [8739] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-11-20 11:55:05 [8741] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-11-20 11:55:05 [8739] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-11-20 11:55:05 [8741] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-11-20 11:55:05 [8739] [0] *** TIMEOUT: Please check your npcd.cfg
2017-11-20 11:55:05 [8741] [0] *** TIMEOUT: Please check your npcd.cfg
2017-11-20 11:55:05 [8739] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1511041590.perfdata.service-PID-8739 deleted
2017-11-20 11:55:05 [8741] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1511041561.perfdata.service-PID-8741 deleted
2017-11-20 11:55:05 [8739] [0] *** Timeout while processing Host: "hidh206.hydc.sbc.com" Service: "Filespace_STAR-opt-openv"
2017-11-20 11:55:05 [8741] [0] *** Timeout while processing Host: "hidh206.hydc.sbc.com" Service: "Filespace_STAR-opt"
2017-11-20 11:55:05 [8739] [0] *** process_perfdata.pl terminated on signal ALRM
2017-11-20 11:55:05 [8741] [0] *** process_perfdata.pl terminated on signal ALRM
2017-11-20 11:55:19 [9368] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-11-20 11:55:19 [9368] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-11-20 11:55:19 [9368] [0] *** TIMEOUT: Please check your npcd.cfg
2017-11-20 11:55:19 [9368] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1511074594.perfdata.service-PID-9368 deleted
2017-11-20 11:55:19 [9368] [0] *** Timeout while processing Host: "cspz0423.sldc.sbc.com" Service: "Filespace_STAR-usr-local"
2017-11-20 11:55:19 [9368] [0] *** process_perfdata.pl terminated on signal ALRM
2017-11-20 14:33:12 [62146] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-11-20 14:33:12 [62146] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-11-20 14:33:12 [62146] [0] *** TIMEOUT: Please check your npcd.cfg
2017-11-20 14:33:12 [62146] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1511196902.perfdata.service-PID-62146 deleted
2017-11-20 14:33:12 [62146] [0] *** Timeout while processing Host: "hixa105.vtil.att.com" Service: "Filespace_STAR-root"
2017-11-20 14:33:12 [62146] [0] *** process_perfdata.pl terminated on signal ALRM

watch -n 1 'ls /usr/local/nagios/var/spool/checkresults | wc -l'

Minimum count 0 and maximum count 230

For this issue need to monitor npcd service ? to avoid this issue.

[dwhitfield]

We are not using ramdisk with Nagios XI.

You should.

You were having a massive amount of load, but that could have very well been because of the inode issue. The load will

We haven't gotten the npcd status either time we asked, but given what we know right now, I think making sure npcd stays on is a good idea.

If you post the entire /usr/local/nagios/var/perfdataproc.log and /usr/local/nagios/var/npcd.log that will give us a better idea of what was going on when the inodes were full.

[sandeepatil]

Please find attached profile.zip file.

This server having checkresult directory issue.

Other details,

# ps -ef | grep perf

root 177 2 0 Dec21 ? 00:00:00 [perf]
nagios 87552 87539 0 10:04 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios 87555 87552 0 10:04 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios 88051 62937 0 10:04 pts/1 00:00:00 grep --color=auto perf

tail -100 /usr/local/nagios/var/npcd.log

[12-24-2017 07:02:02] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514116900.perfdata.service'
[12-24-2017 12:01:49] NPCD: ERROR: Executed command exits with return code '7'
[12-24-2017 12:01:49] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514134886.perfdata.service'
[12-24-2017 15:00:32] NPCD: ERROR: Executed command exits with return code '7'
[12-24-2017 15:00:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514145611.perfdata.service'
[12-25-2017 12:01:37] NPCD: ERROR: Executed command exits with return code '7'
[12-25-2017 12:01:37] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514221282.perfdata.service'
[12-26-2017 05:15:06] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 05:15:06] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514283283.perfdata.service'
[12-26-2017 05:15:41] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 05:15:41] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514283314.perfdata.service'
[12-26-2017 05:16:31] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 05:16:31] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514283379.perfdata.service'
[12-26-2017 05:17:06] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 05:17:06] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514283410.perfdata.service'
[12-26-2017 05:17:41] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 05:17:41] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514283443.perfdata.service'
[12-26-2017 05:18:32] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 05:18:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514283497.perfdata.service'
[12-26-2017 08:30:07] NPCD: Caught Termination Signal - Hasta la vista... baby
[12-26-2017 08:30:07] NPCD: npcd Daemon (0.4.14) started with PID=54721
[12-26-2017 08:30:07] NPCD: Please have a look at 'npcd -V' to get license information
[12-26-2017 08:30:07] NPCD: HINT: load_threshold is enabled - ('10.000000')
[12-26-2017 08:40:32] NPCD: npcd Daemon (0.4.14) started with PID=58026
[12-26-2017 08:40:32] NPCD: Please have a look at 'npcd -V' to get license information
[12-26-2017 08:40:32] NPCD: HINT: load_threshold is enabled - ('10.000000')
[12-26-2017 08:44:41] NPCD: Caught Termination Signal - Hasta la vista... baby
[12-26-2017 09:43:37] NPCD: npcd Daemon (0.4.14) started with PID=73392
[12-26-2017 09:43:37] NPCD: Please have a look at 'npcd -V' to get license information
[12-26-2017 09:43:37] NPCD: HINT: load_threshold is enabled - ('10.000000')
[12-26-2017 09:45:42] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:45:42] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299523.perfdata.service'
[12-26-2017 09:46:18] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:46:18] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299555.perfdata.service'
[12-26-2017 09:46:38] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:46:38] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299586.perfdata.service'
[12-26-2017 09:47:13] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:47:13] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299617.perfdata.service'
[12-26-2017 09:47:43] NPCD: WARN: MAX load reached: load 10.310000/10.000000 at i=0
[12-26-2017 09:47:58] NPCD: WARN: MAX load reached: load 10.190000/10.000000 at i=1
[12-26-2017 09:48:33] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:48:33] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299681.perfdata.service'
[12-26-2017 09:48:53] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:48:53] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299712.perfdata.service'
[12-26-2017 09:48:53] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:48:53] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299714.perfdata.service'
[12-26-2017 09:49:28] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:49:28] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299753.perfdata.service'
[12-26-2017 09:49:48] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:49:48] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299777.perfdata.service'
[12-26-2017 09:50:08] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:50:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299793.perfdata.service'
[12-26-2017 09:50:28] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:50:28] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299810.perfdata.service'
[12-26-2017 09:50:48] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:50:48] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299827.perfdata.service'
[12-26-2017 09:50:48] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:50:48] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299840.perfdata.service'
[12-26-2017 09:51:08] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:51:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299854.perfdata.service'
[12-26-2017 09:51:28] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:51:28] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299866.perfdata.service'
[12-26-2017 09:51:28] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:51:28] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299879.perfdata.service'
[12-26-2017 09:51:48] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:51:48] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299892.perfdata.service'
[12-26-2017 09:52:08] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:52:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299918.perfdata.service'
[12-26-2017 09:52:08] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:52:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299904.perfdata.service'
[12-26-2017 09:52:59] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:52:59] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514299952.perfdata.service'
[12-26-2017 09:54:34] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:54:34] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300053.perfdata.service'
[12-26-2017 09:54:54] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:54:54] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300079.perfdata.service'
[12-26-2017 09:54:54] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:54:54] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300067.perfdata.service'
[12-26-2017 09:55:14] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:55:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300099.perfdata.service'
[12-26-2017 09:55:14] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:55:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300091.perfdata.service'
[12-26-2017 09:55:34] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:55:34] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300112.perfdata.service'
[12-26-2017 09:56:14] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:56:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300142.perfdata.service'
[12-26-2017 09:56:14] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:56:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300157.perfdata.service'
[12-26-2017 09:56:34] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:56:34] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300173.perfdata.service'
[12-26-2017 09:56:54] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:56:54] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300187.perfdata.service'
[12-26-2017 09:57:34] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:57:34] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300232.perfdata.service'
[12-26-2017 09:58:10] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:58:10] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300262.perfdata.service'
[12-26-2017 09:58:10] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:58:10] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300277.perfdata.service'
[12-26-2017 09:58:30] NPCD: ERROR: Executed command exits with return code '7'
[12-26-2017 09:58:30] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1514300293.perfdata.service'

tail -100 /usr/local/nagios/var/perfdataproc.log

2017-12-26 09:52:08 [77606] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:52:08 [77606] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514299918.perfdata.service-PID-77606 deleted
2017-12-26 09:52:08 [77606] [0] *** Timeout while processing Host: "blsd001.bhdc.att.com" Service: "Filespace_STAR-opt-tools"
2017-12-26 09:52:08 [77606] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:52:08 [77603] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:52:08 [77603] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:52:08 [77603] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:52:08 [77603] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514299904.perfdata.service-PID-77603 deleted
2017-12-26 09:52:08 [77603] [0] *** Timeout while processing Host: "ulpv0691.madc.att.com" Service: "CPU_STAR-Composite"
2017-12-26 09:52:08 [77603] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:52:58 [77907] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:52:58 [77907] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:52:58 [77907] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:52:58 [77907] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514299952.perfdata.service-PID-77907 deleted
2017-12-26 09:52:58 [77907] [0] *** Timeout while processing Host: "flth058.ffdc.sbc.com" Service: "Filespace_STAR-usr"
2017-12-26 09:52:58 [77907] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:54:34 [78655] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:54:34 [78655] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:54:34 [78655] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:54:34 [78655] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300053.perfdata.service-PID-78655 deleted
2017-12-26 09:54:34 [78655] [0] *** Timeout while processing Host: "zlt00268.vci.att.com" Service: "Filespace_STAR-boot"
2017-12-26 09:54:34 [78655] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:54:54 [78787] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:54:54 [78787] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:54:54 [78787] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:54:54 [78787] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300079.perfdata.service-PID-78787 deleted
2017-12-26 09:54:54 [78787] [0] *** Timeout while processing Host: "zldv7321.vci.att.com" Service: "Filespace_STAR-var"
2017-12-26 09:54:54 [78787] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:54:54 [78788] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:54:54 [78788] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:54:54 [78788] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:54:54 [78788] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300067.perfdata.service-PID-78788 deleted
2017-12-26 09:54:54 [78788] [0] *** Timeout while processing Host: "zldv4297.vci.att.com" Service: "Filespace_STAR-var"
2017-12-26 09:54:54 [78788] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:55:14 [79066] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:55:14 [79066] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:55:14 [79066] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:55:14 [79066] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300099.perfdata.service-PID-79066 deleted
2017-12-26 09:55:14 [79066] [0] *** Timeout while processing Host: "hltv0173.hydc.sbc.com" Service: "Filespace_STAR-usr"
2017-12-26 09:55:14 [79066] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:55:14 [79064] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:55:14 [79064] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:55:14 [79064] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:55:14 [79064] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300091.perfdata.service-PID-79064 deleted
2017-12-26 09:55:14 [79064] [0] *** Timeout while processing Host: "pldv0075.pedc.sbc.com" Service: "Filespace_STAR-boot"
2017-12-26 09:55:14 [79064] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:55:34 [79252] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:55:34 [79252] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:55:34 [79252] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:55:34 [79252] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300112.perfdata.service-PID-79252 deleted
2017-12-26 09:55:34 [79252] [0] *** Timeout while processing Host: "zlt10805.vci.att.com" Service: "Filespace_STAR-usr"
2017-12-26 09:55:34 [79252] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:56:14 [79576] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:56:14 [79576] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:56:14 [79576] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:56:14 [79576] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300142.perfdata.service-PID-79576 deleted
2017-12-26 09:56:14 [79576] [0] *** Timeout while processing Host: "zlt12893.vci.att.com" Service: "Filespace_STAR-home"
2017-12-26 09:56:14 [79576] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:56:14 [79577] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:56:14 [79577] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:56:14 [79577] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:56:14 [79577] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300157.perfdata.service-PID-79577 deleted
2017-12-26 09:56:14 [79577] [0] *** Timeout while processing Host: "zltv4437.vci.att.com" Service: "Filespace_STAR-home"
2017-12-26 09:56:14 [79577] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:56:34 [79892] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:56:34 [79892] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:56:34 [79892] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:56:34 [79892] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300173.perfdata.service-PID-79892 deleted
2017-12-26 09:56:34 [79892] [0] *** Timeout while processing Host: "cltv0309.sldc.sbc.com" Service: "Filespace_STAR-boot"
2017-12-26 09:56:34 [79892] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:56:54 [80035] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:56:54 [80035] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:56:54 [80035] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:56:54 [80035] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300187.perfdata.service-PID-80035 deleted
2017-12-26 09:56:54 [80035] [0] *** Timeout while processing Host: "flsv0074.ffdc.sbc.com" Service: "CPU_STAR-Composite"
2017-12-26 09:56:54 [80035] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:57:34 [80406] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:57:34 [80406] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:57:34 [80406] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:57:34 [80406] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300232.perfdata.service-PID-80406 deleted
2017-12-26 09:57:34 [80406] [0] *** Timeout while processing Host: "zlp17969.vci.att.com" Service: "Filespace_STAR-home"
2017-12-26 09:57:34 [80406] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:58:10 [80752] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:58:10 [80752] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:58:10 [80752] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:58:10 [80752] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300262.perfdata.service-PID-80752 deleted
2017-12-26 09:58:10 [80752] [0] *** Timeout while processing Host: "cltv0322.sldc.sbc.com" Service: "Filespace_STAR-usr"
2017-12-26 09:58:10 [80752] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:58:10 [80753] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:58:10 [80753] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:58:10 [80753] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:58:10 [80753] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300277.perfdata.service-PID-80753 deleted
2017-12-26 09:58:10 [80753] [0] *** Timeout while processing Host: "pldi064.pedc.sbc.com" Service: "Filespace_STAR-var-crash"
2017-12-26 09:58:10 [80753] [0] *** process_perfdata.pl terminated on signal ALRM
2017-12-26 09:58:30 [81604] [0] *** TIMEOUT: Timeout after 5 secs. ***
2017-12-26 09:58:30 [81604] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2017-12-26 09:58:30 [81604] [0] *** TIMEOUT: Please check your npcd.cfg
2017-12-26 09:58:30 [81604] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1514300293.perfdata.service-PID-81604 deleted
2017-12-26 09:58:30 [81604] [0] *** Timeout while processing Host: "ulpd475.madc.att.com" Service: "Filespace_STAR-root"
2017-12-26 09:58:30 [81604] [0] *** process_perfdata.pl terminated on signal ALRM

[kyang]

Thanks for sharing this information.

I shared your profile with the other techs!

Could you try increasing the TIMEOUT value in the process_perfdata.pl file?

Code: Select all

/usr/local/nagios/etc/pnp/process_perfdata.pl

Code: Select all

TIMEOUT = 5

Change it to

TIMEOUT = 50

Code: Select all

service npcd restart

Let us know how that goes.

[sandeepatil]

Will configure as suggested and update you if facing any issue.