Page 1 of 3
Issues with graphs caused by nagios process memory leak
Posted: Thu Sep 03, 2015 2:18 am
by WillemDH
Hello,
So it seems every time I'm on a holiday something is going wrong with the performance graphs, see attached screenshot. This is very annoying, as this has an impact om data such as averages etc. Please advise me how to
1) detect the reason why our graphs stop working sometimes
2) let me know how I could best monitor this, so in case this issue re-appears we can act asap
The issue was solved both times by rebooting the Nagios server.
Code: Select all
ls -l /usr/local/nagios/var/
total 116796
drwxrwxr-x. 2 nagios nagios 40960 Sep 2 23:59 archives
-rw-r--r-- 1 nagios users 9095370 Oct 8 2014 check_fujitsu_eternus_dx.log
-rw-r--r-- 1 nagios users 10239895 Oct 8 2014 check_fujitsu_eternus_dx.log.1
-rw-r--r-- 1 nagios users 10239935 Oct 8 2014 check_fujitsu_eternus_dx.log.2
-rw-r--r-- 1 nagios users 10239875 Oct 8 2014 check_fujitsu_eternus_dx.log.3
-rw-r--r-- 1 nagios users 10239889 Oct 8 2014 check_fujitsu_eternus_dx.log.4
-rw-r--r-- 1 nagios users 10239978 Oct 8 2014 check_fujitsu_eternus_dx.log.5
-rw-r--r-- 1 apache apache 7907 Sep 30 2013 graphapi.log
-rw-r--r-- 1 nagios nagios 1218 Apr 22 10:29 host-perfdata
-rw-r--r-- 1 nagios nagios 34 Sep 3 09:00 nagios.configtest
-rw-r--r-- 1 nagios nagios 55275 Jun 18 17:19 nagios.debug
-rw-r--r-- 1 nagios nagios 1000706 Jun 18 17:19 nagios.debug.old
-rw-r--r-- 1 nagios nagios 6 Sep 3 09:00 nagios.lock
-rw-rw-r-- 1 nagios nagios 2588186 Sep 3 09:59 nagios.log
-rw-r--r-- 1 nagios nagios 5 Aug 31 09:37 ndo2db.lock
-rw-r--r-- 1 nagios nagios 0 Sep 3 09:00 ndomod.tmp
srwxr-xr-x 1 nagios nagios 0 Aug 31 09:37 ndo.sock
-rw-r--r--. 1 nagios nagios 2595018 Aug 31 09:45 npcd.log
-rw-r--r--. 1 nagios nagios 9953257 Apr 13 13:08 objects.cache
-rw-r--r-- 1 nagios nagios 15204315 Sep 3 09:00 objects.precache
-rw-rw-rw- 1 nagios nagios 4723086 Aug 31 09:45 perfdata.log
-rw------- 1 nagios nagios 22933768 Sep 3 09:00 retention.dat
drwxrwsr-x. 2 nagios nagcmd 4096 Sep 3 09:00 rw
-rw-r--r-- 1 nagios nagios 86962 Apr 22 10:29 service-perfdata
drwxr-xr-x. 5 root root 4096 Jun 6 2012 spool
drwxr-xr-x. 2 nagios nagios 4096 Dec 23 2014 stats
-rw-r--r-- 1 nagios users 15683 Oct 13 2014 test.txt
Code: Select all
ls -l /usr/local/nagios/var/spool
total 648
drwxrwsr-x. 2 nagios nagcmd 77824 Apr 13 13:13 checkresults
drwxr-xr-x. 2 nagios nagios 548864 Apr 22 10:27 perfdata
drwxr-xr-x. 2 nagios nagios 24576 Apr 22 10:29 xidpe
Code: Select all
cat /proc/`cat /usr/local/nagios/var/nagios.lock`/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 86983 86983 processes
Max open files 4096 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 86983 86983 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Code: Select all
grep perfdata /usr/local/nagios/etc/nagios.cfg
service_perfdata_file=/var/nagiosramdisk/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
host_perfdata_file=/var/nagiosramdisk/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tHOSTOUTPUT::$HOSTOUTPUT$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file-bulk
perfdata_timeout=5
Grtz
Willem
Re: Issues with graphs
Posted: Thu Sep 03, 2015 2:27 am
by Box293
Create some localhost services to monitor the important bits. I created these feature requests which I think would be useful in an XI system, the requests explain how to setup the services:
NPCD Service:
http://tracker.nagios.com/view.php?id=641
File Counts:
http://tracker.nagios.com/view.php?id=635
http://tracker.nagios.com/view.php?id=636
I suspect your npcd load limit is being reached and npcd is stopping. If it is, it should have been logged in
/usr/local/nagios/var/npcd.log or
/usr/local/nagios/var/perfdata.log
Re: Issues with graphs
Posted: Thu Sep 03, 2015 3:34 am
by WillemDH
Hey Troy,
Thanks for the quick reply. So I indeed seem to find some data related to my issues in the /usr/local/nagios/var/npcd.log
Code: Select all
[07-04-2015 02:01:28] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1435968062.perfdata.service'
[07-27-2015 14:56:11] NPCD: Caught Termination Signal - Hasta la vista... baby
[07-27-2015 14:56:59] NPCD: npcd Daemon (0.4.14) started with PID=2358
[07-27-2015 14:56:59] NPCD: Please have a look at 'npcd -V' to get license information
[07-27-2015 14:56:59] NPCD: HINT: load_threshold is enabled - ('10.000000')
[07-27-2015 14:57:39] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:57:39] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001837.perfdata.service'
[07-27-2015 14:58:04] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:58:04] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001852.perfdata.service'
[07-27-2015 14:58:04] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:58:04] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001867.perfdata.service'
[07-27-2015 14:58:30] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:58:30] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001883.perfdata.service'
[07-27-2015 14:58:55] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:58:55] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001913.perfdata.service'
[07-27-2015 14:58:55] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:58:55] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001897.perfdata.service'
[07-27-2015 14:59:20] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:59:20] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001927.perfdata.service'
[07-27-2015 14:59:45] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:59:45] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001957.perfdata.service'
[07-27-2015 14:59:45] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 14:59:45] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001943.perfdata.service'
[07-27-2015 15:00:10] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:00:10] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001973.perfdata.service'
[07-27-2015 15:00:10] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:00:10] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438001987.perfdata.service'
[07-27-2015 15:00:35] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:00:35] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002003.perfdata.service'
[07-27-2015 15:00:35] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:00:35] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002017.perfdata.service'
[07-27-2015 15:01:00] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:01:00] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002033.perfdata.service'
[07-27-2015 15:01:25] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:01:25] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002047.perfdata.service'
[07-27-2015 15:01:25] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:01:25] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002063.perfdata.service'
[07-27-2015 15:02:57] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:02:57] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002153.perfdata.service'
[07-27-2015 15:03:23] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:03:23] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002183.perfdata.service'
[07-27-2015 15:03:23] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:03:23] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002167.perfdata.service'
[07-27-2015 15:03:48] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:03:48] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002197.perfdata.service'
[07-27-2015 15:04:13] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:04:13] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002227.perfdata.service'
[07-27-2015 15:04:13] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:04:13] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002213.perfdata.service'
[07-27-2015 15:04:38] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:04:38] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002243.perfdata.service'
[07-27-2015 15:04:38] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:04:38] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002257.perfdata.service'
[07-27-2015 15:05:03] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:05:03] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002273.perfdata.service'
[07-27-2015 15:05:03] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:05:03] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002287.perfdata.service'
[07-27-2015 15:05:28] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:05:28] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002303.perfdata.service'
[07-27-2015 15:05:53] NPCD: ERROR: Executed command exits with return code '1'
[07-27-2015 15:05:53] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438002317.perfdata.service'
[08-01-2015 00:16:25] NPCD: ERROR: Executed command exits with return code '1'
[08-01-2015 00:16:25] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1438380952.perfdata.service'
[08-31-2015 08:51:18] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 08:51:18] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 08:51:18] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441003853.perfdata.host'
[08-31-2015 08:51:18] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441003853.perfdata.service'
[08-31-2015 09:35:05] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:35:05] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006480.perfdata.service'
[08-31-2015 09:35:30] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:35:30] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006510.perfdata.service'
[08-31-2015 09:35:30] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:35:30] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006496.perfdata.service'
[08-31-2015 09:35:55] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:35:55] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006540.perfdata.service'
[08-31-2015 09:35:55] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:35:55] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006526.perfdata.service'
[08-31-2015 09:36:20] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:36:20] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006556.perfdata.service'
[08-31-2015 09:36:45] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:36:45] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006586.perfdata.service'
[08-31-2015 09:36:45] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:36:45] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006570.perfdata.service'
[08-31-2015 09:36:56] NPCD: Caught Termination Signal - Hasta la vista... baby
[08-31-2015 09:37:46] NPCD: npcd Daemon (0.4.14) started with PID=2379
[08-31-2015 09:37:46] NPCD: Please have a look at 'npcd -V' to get license information
[08-31-2015 09:37:46] NPCD: HINT: load_threshold is enabled - ('10.000000')
[08-31-2015 09:38:26] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:38:26] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006684.perfdata.service'
[08-31-2015 09:38:51] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:38:51] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006699.perfdata.service'
[08-31-2015 09:39:16] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:39:16] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006729.perfdata.service'
[08-31-2015 09:39:16] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:39:16] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006714.perfdata.service'
[08-31-2015 09:39:42] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:39:42] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006744.perfdata.service'
[08-31-2015 09:39:42] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:39:42] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006759.perfdata.service'
[08-31-2015 09:40:07] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:40:07] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006774.perfdata.service'
[08-31-2015 09:40:07] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:40:07] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006790.perfdata.service'
[08-31-2015 09:40:32] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:40:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006804.perfdata.service'
[08-31-2015 09:40:32] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:40:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006820.perfdata.service'
[08-31-2015 09:40:57] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:40:57] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006834.perfdata.service'
[08-31-2015 09:41:22] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:41:22] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006850.perfdata.service'
[08-31-2015 09:41:22] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:41:22] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006864.perfdata.service'
[08-31-2015 09:41:47] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:41:47] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006880.perfdata.service'
[08-31-2015 09:42:12] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:42:12] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006899.perfdata.service'
[08-31-2015 09:42:59] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:42:59] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441006959.perfdata.service'
[08-31-2015 09:43:48] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:43:48] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441007005.perfdata.service'
[08-31-2015 09:44:13] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:44:13] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441007035.perfdata.service'
[08-31-2015 09:44:13] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:44:13] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441007019.perfdata.service'
[08-31-2015 09:44:38] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:44:38] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441007049.perfdata.service'
[08-31-2015 09:45:24] NPCD: ERROR: Executed command exits with return code '1'
[08-31-2015 09:45:24] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1441007109.perfdata.service'
And also in the /usr/local/nagios/var/perfdata.log
Code: Select all
tail -250 /usr/local/nagios/var/perfdata.log
2015-07-27 15:00:10 [8786] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:00:10 [8786] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:00:10 [8786] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:00:10 [8786] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438001987.perfdata.service-PID-8786 deleted
2015-07-27 15:00:10 [8786] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:00:35 [9595] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:00:35 [9595] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:00:35 [9595] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:00:35 [9595] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002003.perfdata.service-PID-9595 deleted
2015-07-27 15:00:35 [9595] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:00:35 [9598] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:00:35 [9598] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:00:35 [9598] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:00:35 [9598] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002017.perfdata.service-PID-9598 deleted
2015-07-27 15:00:35 [9598] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:01:00 [10200] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:01:00 [10200] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:01:00 [10200] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:01:00 [10200] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002033.perfdata.service-PID-10200 deleted
2015-07-27 15:01:00 [10200] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:01:25 [10656] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:01:25 [10656] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:01:25 [10656] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:01:25 [10656] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002047.perfdata.service-PID-10656 deleted
2015-07-27 15:01:25 [10656] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:01:25 [10658] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:01:25 [10658] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:01:25 [10658] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:01:25 [10658] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002063.perfdata.service-PID-10658 deleted
2015-07-27 15:01:25 [10658] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:02:57 [13130] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:02:57 [13130] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:02:57 [13130] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:02:57 [13130] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002153.perfdata.service-PID-13130 deleted
2015-07-27 15:02:57 [13130] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:03:23 [14102] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:03:23 [14102] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:03:23 [14102] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:03:23 [14102] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002183.perfdata.service-PID-14102 deleted
2015-07-27 15:03:23 [14102] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:03:23 [14099] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:03:23 [14099] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:03:23 [14099] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:03:23 [14099] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002167.perfdata.service-PID-14099 deleted
2015-07-27 15:03:23 [14099] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:03:48 [14926] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:03:48 [14926] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:03:48 [14926] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:03:48 [14926] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002197.perfdata.service-PID-14926 deleted
2015-07-27 15:03:48 [14926] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:04:13 [15704] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:04:13 [15704] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:04:13 [15704] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:04:13 [15704] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002227.perfdata.service-PID-15704 deleted
2015-07-27 15:04:13 [15704] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:04:13 [15701] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:04:13 [15701] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:04:13 [15701] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:04:13 [15701] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002213.perfdata.service-PID-15701 deleted
2015-07-27 15:04:13 [15701] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:04:38 [16690] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:04:38 [16690] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:04:38 [16690] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:04:38 [16690] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002243.perfdata.service-PID-16690 deleted
2015-07-27 15:04:38 [16690] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:04:38 [16694] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:04:38 [16694] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:04:38 [16694] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:04:38 [16694] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002257.perfdata.service-PID-16694 deleted
2015-07-27 15:04:38 [16694] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:05:03 [17519] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:05:03 [17519] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:05:03 [17519] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:05:03 [17519] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002273.perfdata.service-PID-17519 deleted
2015-07-27 15:05:03 [17519] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:05:03 [17521] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:05:03 [17521] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:05:03 [17521] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:05:03 [17521] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002287.perfdata.service-PID-17521 deleted
2015-07-27 15:05:03 [17521] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:05:28 [18332] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:05:28 [18332] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:05:28 [18332] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:05:28 [18332] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002303.perfdata.service-PID-18332 deleted
2015-07-27 15:05:28 [18332] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-27 15:05:53 [18937] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-07-27 15:05:53 [18937] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-07-27 15:05:53 [18937] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-07-27 15:05:53 [18937] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438002317.perfdata.service-PID-18937 deleted
2015-07-27 15:05:53 [18937] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-01 00:16:25 [22064] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-01 00:16:25 [22064] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-01 00:16:25 [22064] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-01 00:16:25 [22064] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1438380952.perfdata.service-PID-22064 deleted
2015-08-01 00:16:25 [22064] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 08:51:18 [59148] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 08:51:18 [59147] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 08:51:18 [59148] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 08:51:18 [59147] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 08:51:18 [59147] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 08:51:18 [59148] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 08:51:18 [59147] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441003853.perfdata.service-PID-59147 deleted
2015-08-31 08:51:18 [59148] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441003853.perfdata.host-PID-59148 deleted
2015-08-31 08:51:18 [59147] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 08:51:18 [59148] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:35:05 [10482] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:35:05 [10482] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:35:05 [10482] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:35:05 [10482] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006480.perfdata.service-PID-10482 deleted
2015-08-31 09:35:05 [10482] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:35:30 [11422] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:35:30 [11422] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:35:30 [11422] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:35:30 [11422] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006510.perfdata.service-PID-11422 deleted
2015-08-31 09:35:30 [11422] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:35:30 [11420] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:35:30 [11420] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:35:30 [11420] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:35:30 [11420] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006496.perfdata.service-PID-11420 deleted
2015-08-31 09:35:30 [11420] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:35:55 [12211] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:35:55 [12211] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:35:55 [12211] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:35:55 [12211] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006540.perfdata.service-PID-12211 deleted
2015-08-31 09:35:55 [12211] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:35:55 [12208] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:35:55 [12208] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:35:55 [12208] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:35:55 [12208] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006526.perfdata.service-PID-12208 deleted
2015-08-31 09:35:55 [12208] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:36:20 [13278] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:36:20 [13278] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:36:20 [13278] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:36:20 [13278] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006556.perfdata.service-PID-13278 deleted
2015-08-31 09:36:20 [13278] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:36:45 [14225] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:36:45 [14225] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:36:45 [14225] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:36:45 [14225] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006586.perfdata.service-PID-14225 deleted
2015-08-31 09:36:45 [14225] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:36:45 [14222] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:36:45 [14222] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:36:45 [14222] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:36:45 [14222] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006570.perfdata.service-PID-14222 deleted
2015-08-31 09:36:45 [14222] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:38:26 [3532] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:38:26 [3532] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:38:26 [3532] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:38:26 [3532] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006684.perfdata.service-PID-3532 deleted
2015-08-31 09:38:26 [3532] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:38:51 [4727] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:38:51 [4727] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:38:51 [4727] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:38:51 [4727] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006699.perfdata.service-PID-4727 deleted
2015-08-31 09:38:51 [4727] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:39:16 [5570] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:39:16 [5570] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:39:16 [5570] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:39:16 [5570] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006729.perfdata.service-PID-5570 deleted
2015-08-31 09:39:16 [5570] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:39:16 [5566] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:39:16 [5566] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:39:16 [5566] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:39:16 [5566] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006714.perfdata.service-PID-5566 deleted
2015-08-31 09:39:16 [5566] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:39:42 [6551] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:39:42 [6551] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:39:42 [6551] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:39:42 [6551] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006744.perfdata.service-PID-6551 deleted
2015-08-31 09:39:42 [6551] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:39:42 [6554] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:39:42 [6554] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:39:42 [6554] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:39:42 [6554] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006759.perfdata.service-PID-6554 deleted
2015-08-31 09:39:42 [6554] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:40:07 [7446] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:40:07 [7446] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:40:07 [7446] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:40:07 [7446] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006774.perfdata.service-PID-7446 deleted
2015-08-31 09:40:07 [7446] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:40:07 [7449] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:40:07 [7449] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:40:07 [7449] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:40:07 [7449] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006790.perfdata.service-PID-7449 deleted
2015-08-31 09:40:07 [7449] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:40:32 [8601] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:40:32 [8601] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:40:32 [8601] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:40:32 [8601] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006804.perfdata.service-PID-8601 deleted
2015-08-31 09:40:32 [8601] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:40:32 [8604] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:40:32 [8604] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:40:32 [8604] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:40:32 [8604] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006820.perfdata.service-PID-8604 deleted
2015-08-31 09:40:32 [8604] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:40:57 [9781] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:40:57 [9781] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:40:57 [9781] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:40:57 [9781] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006834.perfdata.service-PID-9781 deleted
2015-08-31 09:40:57 [9781] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:41:22 [11037] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:41:22 [11037] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:41:22 [11037] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:41:22 [11037] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006850.perfdata.service-PID-11037 deleted
2015-08-31 09:41:22 [11037] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:41:22 [11040] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:41:22 [11040] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:41:22 [11040] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:41:22 [11040] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006864.perfdata.service-PID-11040 deleted
2015-08-31 09:41:22 [11040] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:41:47 [11728] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:41:47 [11728] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:41:47 [11728] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:41:47 [11728] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006880.perfdata.service-PID-11728 deleted
2015-08-31 09:41:47 [11728] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:42:12 [12342] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:42:12 [12342] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:42:12 [12342] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:42:12 [12342] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006899.perfdata.service-PID-12342 deleted
2015-08-31 09:42:12 [12342] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:42:59 [13445] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:42:59 [13445] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:42:59 [13445] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:42:59 [13445] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441006959.perfdata.service-PID-13445 deleted
2015-08-31 09:42:59 [13445] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:43:48 [15247] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:43:48 [15247] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:43:48 [15247] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:43:48 [15247] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441007005.perfdata.service-PID-15247 deleted
2015-08-31 09:43:48 [15247] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:44:13 [16112] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:44:13 [16112] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:44:13 [16112] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:44:13 [16112] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441007035.perfdata.service-PID-16112 deleted
2015-08-31 09:44:13 [16112] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:44:13 [16111] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:44:13 [16111] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:44:13 [16111] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:44:13 [16111] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441007019.perfdata.service-PID-16111 deleted
2015-08-31 09:44:13 [16111] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:44:38 [16946] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:44:38 [16946] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:44:38 [16946] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:44:38 [16946] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441007049.perfdata.service-PID-16946 deleted
2015-08-31 09:44:38 [16946] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-31 09:45:24 [18622] [0] *** TIMEOUT: Timeout after 10 Sec. ****
2015-08-31 09:45:24 [18622] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-31 09:45:24 [18622] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-08-31 09:45:24 [18622] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1441007109.perfdata.service-PID-18622 deleted
2015-08-31 09:45:24 [18622] [0] *** process_perfdata.pl terminated on signal ALRM
Does these logs confirm your suspicion that my npcd load limit is reached? I also found that the RAM / Swap Used during the periods I'm experiencing issues seems to suddenly rise until the server is rebooted. The question is why this only happens during my holiday's. I'll monitor this more closely in the coming weeks. i'm not seeing high cpu spikes during the issue.
Is the NPCD load limit memory or cpu related? Where can I change this?
Grtz
Willem
Re: Issues with graphs
Posted: Thu Sep 03, 2015 9:14 am
by ssax
I does look like you're hitting the TIMEOUT, you can read more on it (and load_threshold setting) and how to adjust them here:
https://support.nagios.com/wiki/index.p ... ta_Timeout
Re: Issues with graphs
Posted: Fri Sep 04, 2015 2:27 am
by WillemDH
Sean,
I changed the load threshold and the timeout. Please leave this thread open for a while. Let's hope this issue is solved now. I implemented the tips of Troy also, so I should be able to act more quickly when the issue would re-appear.
Grtz and thanks.
Willem
Re: Issues with graphs
Posted: Fri Sep 04, 2015 9:02 am
by hsmith
WillemDH wrote:Sean,
I changed the load threshold and the timeout. Please leave this thread open for a while. Let's hope this issue is solved now. I implemented the tips of Troy also, so I should be able to act more quickly when the issue would re-appear.
Grtz and thanks.
Willem
Thanks Willem, let us know.
Re: Issues with graphs
Posted: Wed Jan 06, 2016 6:45 am
by WillemDH
Hello,
First of all best wishes to all of the Nagios support team for 2016!
Again during my holiday our graphs stopped working. Aparently the things I changed to prevent this didn't solve the problem.
I have been troubleshooting this a bit and it seems the Nagios process has some sort of memory leak, causing it to continue using more and more % memory of the total available memory untill it starts causing issues. Please check my attached screenshot. This is a big problem imho, as it causes blanks in our graphs. Check the attached screenshot and please pay close attention to the memory usage of the Nagios process. The server was rebooted 28/10 and started having issues with graphs 03/01 10:00 untill Monday morning where I rebooted the server.
After a reboot the Nagios process memory usage is stable for some time and then starts using more and more memory, as is clearly visible in the 7 days graph. The moment the process start leaking memory again the last time was after work hours, so noone was doing anything in the gui. Every 1:45 the nagios processes consume 1 % more memory.
Please let me know how I can further troubleshoot and solve this memory leak and graphing problem.
Willem
Re: Issues with graphs caused by nagios process memory leak
Posted: Wed Jan 06, 2016 10:24 am
by ssax
Are you running gearman? If so, please post the output of these commands:
Code: Select all
rpm -qa | grep gearman
grep -v ^# /etc/mod_gearman/mod_gearman_worker.conf | sort -u
grep -v ^# /etc/mod_gearman2/worker.conf | sort -u
Thank you
Re: Issues with graphs caused by nagios process memory leak
Posted: Wed Jan 06, 2016 10:50 am
by ssax
In addition to my post above, are you seeing anything in your nagios.log around that time?
Also, what is the output of this:
Code: Select all
grep "Core Worker" /usr/local/nagios/var/nagios.log
Re: Issues with graphs caused by nagios process memory leak
Posted: Wed Jan 06, 2016 11:49 am
by scottwilkerson
Willem,
Could you also post the command you are using to produce the "nagios" item in this graph
https://support.nagios.com/forum/downlo ... &mode=view
Thanks