Issues with graphs caused by nagios process memory leak

[WillemDH]

Hello,

So it seems every time I'm on a holiday something is going wrong with the performance graphs, see attached screenshot. This is very annoying, as this has an impact om data such as averages etc. Please advise me how to
1) detect the reason why our graphs stop working sometimes
2) let me know how I could best monitor this, so in case this issue re-appears we can act asap

The issue was solved both times by rebooting the Nagios server.

Code: Select all

ls -l /usr/local/nagios/var/
total 116796
drwxrwxr-x. 2 nagios nagios    40960 Sep  2 23:59 archives
-rw-r--r--  1 nagios users   9095370 Oct  8  2014 check_fujitsu_eternus_dx.log
-rw-r--r--  1 nagios users  10239895 Oct  8  2014 check_fujitsu_eternus_dx.log.1
-rw-r--r--  1 nagios users  10239935 Oct  8  2014 check_fujitsu_eternus_dx.log.2
-rw-r--r--  1 nagios users  10239875 Oct  8  2014 check_fujitsu_eternus_dx.log.3
-rw-r--r--  1 nagios users  10239889 Oct  8  2014 check_fujitsu_eternus_dx.log.4
-rw-r--r--  1 nagios users  10239978 Oct  8  2014 check_fujitsu_eternus_dx.log.5
-rw-r--r--  1 apache apache     7907 Sep 30  2013 graphapi.log
-rw-r--r--  1 nagios nagios     1218 Apr 22 10:29 host-perfdata
-rw-r--r--  1 nagios nagios       34 Sep  3 09:00 nagios.configtest
-rw-r--r--  1 nagios nagios    55275 Jun 18 17:19 nagios.debug
-rw-r--r--  1 nagios nagios  1000706 Jun 18 17:19 nagios.debug.old
-rw-r--r--  1 nagios nagios        6 Sep  3 09:00 nagios.lock
-rw-rw-r--  1 nagios nagios  2588186 Sep  3 09:59 nagios.log
-rw-r--r--  1 nagios nagios        5 Aug 31 09:37 ndo2db.lock
-rw-r--r--  1 nagios nagios        0 Sep  3 09:00 ndomod.tmp
srwxr-xr-x  1 nagios nagios        0 Aug 31 09:37 ndo.sock
-rw-r--r--. 1 nagios nagios  2595018 Aug 31 09:45 npcd.log
-rw-r--r--. 1 nagios nagios  9953257 Apr 13 13:08 objects.cache
-rw-r--r--  1 nagios nagios 15204315 Sep  3 09:00 objects.precache
-rw-rw-rw-  1 nagios nagios  4723086 Aug 31 09:45 perfdata.log
-rw-------  1 nagios nagios 22933768 Sep  3 09:00 retention.dat
drwxrwsr-x. 2 nagios nagcmd     4096 Sep  3 09:00 rw
-rw-r--r--  1 nagios nagios    86962 Apr 22 10:29 service-perfdata
drwxr-xr-x. 5 root   root       4096 Jun  6  2012 spool
drwxr-xr-x. 2 nagios nagios     4096 Dec 23  2014 stats
-rw-r--r--  1 nagios users     15683 Oct 13  2014 test.txt

Code: Select all

ls -l /usr/local/nagios/var/spool
total 648
drwxrwsr-x. 2 nagios nagcmd  77824 Apr 13 13:13 checkresults
drwxr-xr-x. 2 nagios nagios 548864 Apr 22 10:27 perfdata
drwxr-xr-x. 2 nagios nagios  24576 Apr 22 10:29 xidpe

Code: Select all

lsof | grep "^nagios" | wc -l
250

Code: Select all

cat /proc/`cat /usr/local/nagios/var/nagios.lock`/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             86983                86983                processes
Max open files            4096                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       86983                86983                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Code: Select all

grep perfdata /usr/local/nagios/etc/nagios.cfg
service_perfdata_file=/var/nagiosramdisk/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
host_perfdata_file=/var/nagiosramdisk/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tHOSTOUTPUT::$HOSTOUTPUT$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file-bulk
perfdata_timeout=5

Grtz

Willem

[Box293]

Create some localhost services to monitor the important bits. I created these feature requests which I think would be useful in an XI system, the requests explain how to setup the services:

NPCD Service:
http://tracker.nagios.com/view.php?id=641

File Counts:
http://tracker.nagios.com/view.php?id=635
http://tracker.nagios.com/view.php?id=636

I suspect your npcd load limit is being reached and npcd is stopping. If it is, it should have been logged in /usr/local/nagios/var/npcd.log or /usr/local/nagios/var/perfdata.log

[WillemDH]

Hey Troy,

Thanks for the quick reply. So I indeed seem to find some data related to my issues in the /usr/local/nagios/var/npcd.log

Code: Select all

[07-04-2015 02:01:28] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1435968062.perfdata.service'
[07-27-2015 14:56:11] NPCD: Caught Termination Signal - Hasta la vista... baby
[07-27-2015 14:56:59] NPCD: npcd Daemon (0.4.14) started with PID=2358
[07-27-2015 14:56:59] NPCD: Please have a look at 'npcd -V' to get license information
[07-27-2015 14:56:59] NPCD: HINT: load_threshold is enabled - ('10.000000')
[07-27-2015 14:57:39] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:57:39] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001837.perfdata.service'
[07-27-2015 14:58:04] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:58:04] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001852.perfdata.service'
[07-27-2015 14:58:04] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:58:04] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001867.perfdata.service'
[07-27-2015 14:58:30] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:58:30] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001883.perfdata.service'
[07-27-2015 14:58:55] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:58:55] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001913.perfdata.service'
[07-27-2015 14:58:55] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:58:55] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001897.perfdata.service'
[07-27-2015 14:59:20] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:59:20] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001927.perfdata.service'
[07-27-2015 14:59:45] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:59:45] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001957.perfdata.service'
[07-27-2015 14:59:45] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:59:45] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001943.perfdata.service'
[07-27-2015 15:00:10] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:00:10] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001973.perfdata.service'
[07-27-2015 15:00:10] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:00:10] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001987.perfdata.service'
[07-27-2015 15:00:35] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:00:35] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002003.perfdata.service'
[07-27-2015 15:00:35] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:00:35] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002017.perfdata.service'
[07-27-2015 15:01:00] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:01:00] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002033.perfdata.service'
[07-27-2015 15:01:25] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:01:25] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002047.perfdata.service'
[07-27-2015 15:01:25] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:01:25] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002063.perfdata.service'
[07-27-2015 15:02:57] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:02:57] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002153.perfdata.service'
[07-27-2015 15:03:23] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:03:23] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002183.perfdata.service'
[07-27-2015 15:03:23] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:03:23] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002167.perfdata.service'
[07-27-2015 15:03:48] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:03:48] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002197.perfdata.service'
[07-27-2015 15:04:13] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:04:13] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002227.perfdata.service'
[07-27-2015 15:04:13] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:04:13] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002213.perfdata.service'
[07-27-2015 15:04:38] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:04:38] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002243.perfdata.service'
[07-27-2015 15:04:38] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:04:38] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002257.perfdata.service'
[07-27-2015 15:05:03] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:05:03] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002273.perfdata.service'
[07-27-2015 15:05:03] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:05:03] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002287.perfdata.service'
[07-27-2015 15:05:28] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:05:28] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002303.perfdata.service'
[07-27-2015 15:05:53] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:05:53] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002317.perfdata.service'
[08-01-2015 00:16:25] NPCD: ERROR: Executed command exits with return code '1'
[08-01-2015 00:16:25] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438380952.perfdata.service'
[08-31-2015 08:51:18] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 08:51:18] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 08:51:18] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441003853.perfdata.host'
[08-31-2015 08:51:18] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441003853.perfdata.service'
[08-31-2015 09:35:05] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:35:05] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006480.perfdata.service'
[08-31-2015 09:35:30] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:35:30] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006510.perfdata.service'
[08-31-2015 09:35:30] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:35:30] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006496.perfdata.service'
[08-31-2015 09:35:55] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:35:55] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006540.perfdata.service'
[08-31-2015 09:35:55] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:35:55] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006526.perfdata.service'
[08-31-2015 09:36:20] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:36:20] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006556.perfdata.service'
[08-31-2015 09:36:45] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:36:45] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006586.perfdata.service'
[08-31-2015 09:36:45] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:36:45] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006570.perfdata.service'
[08-31-2015 09:36:56] NPCD: Caught Termination Signal - Hasta la vista... baby
[08-31-2015 09:37:46] NPCD: npcd Daemon (0.4.14) started with PID=2379
[08-31-2015 09:37:46] NPCD: Please have a look at 'npcd -V' to get license information
[08-31-2015 09:37:46] NPCD: HINT: load_threshold is enabled - ('10.000000')
[08-31-2015 09:38:26] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:38:26] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006684.perfdata.service'
[08-31-2015 09:38:51] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:38:51] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006699.perfdata.service'
[08-31-2015 09:39:16] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:39:16] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006729.perfdata.service'
[08-31-2015 09:39:16] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:39:16] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006714.perfdata.service'
[08-31-2015 09:39:42] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:39:42] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006744.perfdata.service'
[08-31-2015 09:39:42] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:39:42] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006759.perfdata.service'
[08-31-2015 09:40:07] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:40:07] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006774.perfdata.service'
[08-31-2015 09:40:07] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:40:07] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006790.perfdata.service'
[08-31-2015 09:40:32] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:40:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006804.perfdata.service'
[08-31-2015 09:40:32] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:40:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006820.perfdata.service'
[08-31-2015 09:40:57] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:40:57] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006834.perfdata.service'
[08-31-2015 09:41:22] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:41:22] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006850.perfdata.service'
[08-31-2015 09:41:22] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:41:22] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006864.perfdata.service'
[08-31-2015 09:41:47] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:41:47] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006880.perfdata.service'
[08-31-2015 09:42:12] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:42:12] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006899.perfdata.service'
[08-31-2015 09:42:59] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:42:59] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006959.perfdata.service'
[08-31-2015 09:43:48] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:43:48] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441007005.perfdata.service'
[08-31-2015 09:44:13] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:44:13] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441007035.perfdata.service'
[08-31-2015 09:44:13] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:44:13] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441007019.perfdata.service'
[08-31-2015 09:44:38] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:44:38] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441007049.perfdata.service'
[08-31-2015 09:45:24] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:45:24] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441007109.perfdata.service'

And also in the /usr/local/nagios/var/perfdata.log

Code: Select all

 tail -250 /usr/local/nagios/var/perfdata.log
2015-07-27 15:00:10 [8786] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:00:10 [8786] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:00:10 [8786] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:00:10 [8786] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438001987.perfdata.service-PID-8786 deleted
2015-07-27 15:00:10 [8786] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:00:35 [9595] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:00:35 [9595] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:00:35 [9595] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:00:35 [9595] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002003.perfdata.service-PID-9595 deleted
2015-07-27 15:00:35 [9595] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:00:35 [9598] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:00:35 [9598] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:00:35 [9598] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:00:35 [9598] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002017.perfdata.service-PID-9598 deleted
2015-07-27 15:00:35 [9598] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:01:00 [10200] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:01:00 [10200] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:01:00 [10200] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:01:00 [10200] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002033.perfdata.service-PID-10200 deleted
2015-07-27 15:01:00 [10200] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:01:25 [10656] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:01:25 [10656] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:01:25 [10656] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:01:25 [10656] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002047.perfdata.service-PID-10656 deleted
2015-07-27 15:01:25 [10656] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:01:25 [10658] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:01:25 [10658] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:01:25 [10658] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:01:25 [10658] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002063.perfdata.service-PID-10658 deleted
2015-07-27 15:01:25 [10658] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:02:57 [13130] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:02:57 [13130] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:02:57 [13130] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:02:57 [13130] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002153.perfdata.service-PID-13130 deleted
2015-07-27 15:02:57 [13130] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:03:23 [14102] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:03:23 [14102] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:03:23 [14102] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:03:23 [14102] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002183.perfdata.service-PID-14102 deleted
2015-07-27 15:03:23 [14102] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:03:23 [14099] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:03:23 [14099] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:03:23 [14099] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:03:23 [14099] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002167.perfdata.service-PID-14099 deleted
2015-07-27 15:03:23 [14099] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:03:48 [14926] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:03:48 [14926] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:03:48 [14926] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:03:48 [14926] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002197.perfdata.service-PID-14926 deleted
2015-07-27 15:03:48 [14926] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:04:13 [15704] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:04:13 [15704] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:04:13 [15704] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:04:13 [15704] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002227.perfdata.service-PID-15704 deleted
2015-07-27 15:04:13 [15704] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:04:13 [15701] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:04:13 [15701] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:04:13 [15701] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:04:13 [15701] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002213.perfdata.service-PID-15701 deleted
2015-07-27 15:04:13 [15701] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:04:38 [16690] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:04:38 [16690] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:04:38 [16690] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:04:38 [16690] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002243.perfdata.service-PID-16690 deleted
2015-07-27 15:04:38 [16690] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:04:38 [16694] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:04:38 [16694] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:04:38 [16694] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:04:38 [16694] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002257.perfdata.service-PID-16694 deleted
2015-07-27 15:04:38 [16694] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:05:03 [17519] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:05:03 [17519] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:05:03 [17519] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:05:03 [17519] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002273.perfdata.service-PID-17519 deleted
2015-07-27 15:05:03 [17519] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:05:03 [17521] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:05:03 [17521] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:05:03 [17521] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:05:03 [17521] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002287.perfdata.service-PID-17521 deleted
2015-07-27 15:05:03 [17521] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:05:28 [18332] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:05:28 [18332] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:05:28 [18332] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:05:28 [18332] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002303.perfdata.service-PID-18332 deleted
2015-07-27 15:05:28 [18332] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:05:53 [18937] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:05:53 [18937] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:05:53 [18937] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:05:53 [18937] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002317.perfdata.service-PID-18937 deleted
2015-07-27 15:05:53 [18937] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-01 00:16:25 [22064] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-01 00:16:25 [22064] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-01 00:16:25 [22064] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-01 00:16:25 [22064] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438380952.perfdata.service-PID-22064 deleted
2015-08-01 00:16:25 [22064] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 08:51:18 [59148] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 08:51:18 [59147] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 08:51:18 [59148] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 08:51:18 [59147] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 08:51:18 [59147] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 08:51:18 [59148] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 08:51:18 [59147] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441003853.perfdata.service-PID-59147 deleted
2015-08-31 08:51:18 [59148] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441003853.perfdata.host-PID-59148 deleted
2015-08-31 08:51:18 [59147] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 08:51:18 [59148] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:35:05 [10482] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:35:05 [10482] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:35:05 [10482] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:35:05 [10482] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006480.perfdata.service-PID-10482 deleted
2015-08-31 09:35:05 [10482] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:35:30 [11422] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:35:30 [11422] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:35:30 [11422] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:35:30 [11422] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006510.perfdata.service-PID-11422 deleted
2015-08-31 09:35:30 [11422] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:35:30 [11420] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:35:30 [11420] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:35:30 [11420] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:35:30 [11420] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006496.perfdata.service-PID-11420 deleted
2015-08-31 09:35:30 [11420] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:35:55 [12211] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:35:55 [12211] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:35:55 [12211] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:35:55 [12211] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006540.perfdata.service-PID-12211 deleted
2015-08-31 09:35:55 [12211] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:35:55 [12208] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:35:55 [12208] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:35:55 [12208] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:35:55 [12208] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006526.perfdata.service-PID-12208 deleted
2015-08-31 09:35:55 [12208] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:36:20 [13278] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:36:20 [13278] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:36:20 [13278] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:36:20 [13278] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006556.perfdata.service-PID-13278 deleted
2015-08-31 09:36:20 [13278] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:36:45 [14225] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:36:45 [14225] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:36:45 [14225] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:36:45 [14225] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006586.perfdata.service-PID-14225 deleted
2015-08-31 09:36:45 [14225] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:36:45 [14222] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:36:45 [14222] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:36:45 [14222] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:36:45 [14222] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006570.perfdata.service-PID-14222 deleted
2015-08-31 09:36:45 [14222] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:38:26 [3532] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:38:26 [3532] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:38:26 [3532] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:38:26 [3532] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006684.perfdata.service-PID-3532 deleted
2015-08-31 09:38:26 [3532] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:38:51 [4727] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:38:51 [4727] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:38:51 [4727] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:38:51 [4727] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006699.perfdata.service-PID-4727 deleted
2015-08-31 09:38:51 [4727] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:39:16 [5570] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:39:16 [5570] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:39:16 [5570] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:39:16 [5570] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006729.perfdata.service-PID-5570 deleted
2015-08-31 09:39:16 [5570] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:39:16 [5566] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:39:16 [5566] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:39:16 [5566] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:39:16 [5566] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006714.perfdata.service-PID-5566 deleted
2015-08-31 09:39:16 [5566] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:39:42 [6551] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:39:42 [6551] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:39:42 [6551] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:39:42 [6551] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006744.perfdata.service-PID-6551 deleted
2015-08-31 09:39:42 [6551] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:39:42 [6554] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:39:42 [6554] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:39:42 [6554] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:39:42 [6554] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006759.perfdata.service-PID-6554 deleted
2015-08-31 09:39:42 [6554] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:40:07 [7446] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:40:07 [7446] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:40:07 [7446] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:40:07 [7446] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006774.perfdata.service-PID-7446 deleted
2015-08-31 09:40:07 [7446] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:40:07 [7449] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:40:07 [7449] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:40:07 [7449] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:40:07 [7449] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006790.perfdata.service-PID-7449 deleted
2015-08-31 09:40:07 [7449] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:40:32 [8601] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:40:32 [8601] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:40:32 [8601] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:40:32 [8601] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006804.perfdata.service-PID-8601 deleted
2015-08-31 09:40:32 [8601] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:40:32 [8604] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:40:32 [8604] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:40:32 [8604] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:40:32 [8604] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006820.perfdata.service-PID-8604 deleted
2015-08-31 09:40:32 [8604] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:40:57 [9781] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:40:57 [9781] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:40:57 [9781] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:40:57 [9781] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006834.perfdata.service-PID-9781 deleted
2015-08-31 09:40:57 [9781] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:41:22 [11037] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:41:22 [11037] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:41:22 [11037] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:41:22 [11037] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006850.perfdata.service-PID-11037 deleted
2015-08-31 09:41:22 [11037] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:41:22 [11040] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:41:22 [11040] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:41:22 [11040] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:41:22 [11040] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006864.perfdata.service-PID-11040 deleted
2015-08-31 09:41:22 [11040] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:41:47 [11728] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:41:47 [11728] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:41:47 [11728] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:41:47 [11728] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006880.perfdata.service-PID-11728 deleted
2015-08-31 09:41:47 [11728] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:42:12 [12342] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:42:12 [12342] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:42:12 [12342] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:42:12 [12342] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006899.perfdata.service-PID-12342 deleted
2015-08-31 09:42:12 [12342] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:42:59 [13445] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:42:59 [13445] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:42:59 [13445] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:42:59 [13445] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006959.perfdata.service-PID-13445 deleted
2015-08-31 09:42:59 [13445] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:43:48 [15247] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:43:48 [15247] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:43:48 [15247] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:43:48 [15247] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441007005.perfdata.service-PID-15247 deleted
2015-08-31 09:43:48 [15247] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:44:13 [16112] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:44:13 [16112] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:44:13 [16112] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:44:13 [16112] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441007035.perfdata.service-PID-16112 deleted
2015-08-31 09:44:13 [16112] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:44:13 [16111] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:44:13 [16111] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:44:13 [16111] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:44:13 [16111] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441007019.perfdata.service-PID-16111 deleted
2015-08-31 09:44:13 [16111] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:44:38 [16946] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:44:38 [16946] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:44:38 [16946] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:44:38 [16946] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441007049.perfdata.service-PID-16946 deleted
2015-08-31 09:44:38 [16946] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:45:24 [18622] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:45:24 [18622] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:45:24 [18622] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:45:24 [18622] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441007109.perfdata.service-PID-18622 deleted
2015-08-31 09:45:24 [18622] [0] *** process_perfdata.pl terminated on signal ALRM

Does these logs confirm your suspicion that my npcd load limit is reached? I also found that the RAM / Swap Used during the periods I'm experiencing issues seems to suddenly rise until the server is rebooted. The question is why this only happens during my holiday's. I'll monitor this more closely in the coming weeks. i'm not seeing high cpu spikes during the issue.
Is the NPCD load limit memory or cpu related? Where can I change this?

Grtz

Willem

[ssax]

I does look like you're hitting the TIMEOUT, you can read more on it (and load_threshold setting) and how to adjust them here:

https://support.nagios.com/wiki/index.p ... ta_Timeout

[WillemDH]

Sean,

I changed the load threshold and the timeout. Please leave this thread open for a while. Let's hope this issue is solved now. I implemented the tips of Troy also, so I should be able to act more quickly when the issue would re-appear.

Grtz and thanks.

Willem

[hsmith]

Sean,

I changed the load threshold and the timeout. Please leave this thread open for a while. Let's hope this issue is solved now. I implemented the tips of Troy also, so I should be able to act more quickly when the issue would re-appear.

Grtz and thanks.

Willem

Thanks Willem, let us know.

[WillemDH]

Hello,

First of all best wishes to all of the Nagios support team for 2016!

Again during my holiday our graphs stopped working. Aparently the things I changed to prevent this didn't solve the problem.

I have been troubleshooting this a bit and it seems the Nagios process has some sort of memory leak, causing it to continue using more and more % memory of the total available memory untill it starts causing issues. Please check my attached screenshot. This is a big problem imho, as it causes blanks in our graphs. Check the attached screenshot and please pay close attention to the memory usage of the Nagios process. The server was rebooted 28/10 and started having issues with graphs 03/01 10:00 untill Monday morning where I rebooted the server.

After a reboot the Nagios process memory usage is stable for some time and then starts using more and more memory, as is clearly visible in the 7 days graph. The moment the process start leaking memory again the last time was after work hours, so noone was doing anything in the gui. Every 1:45 the nagios processes consume 1 % more memory.

Please let me know how I can further troubleshoot and solve this memory leak and graphing problem.

Willem

[ssax]

Are you running gearman? If so, please post the output of these commands:

Code: Select all

rpm -qa | grep gearman
grep -v ^# /etc/mod_gearman/mod_gearman_worker.conf | sort -u
grep -v ^# /etc/mod_gearman2/worker.conf | sort -u

Thank you

[ssax]

In addition to my post above, are you seeing anything in your nagios.log around that time?

Also, what is the output of this:

Code: Select all

grep "Core Worker" /usr/local/nagios/var/nagios.log

[scottwilkerson]

Willem,

Could you also post the command you are using to produce the "nagios" item in this graph

https://support.nagios.com/forum/downlo ... &mode=view

Thanks