performance data for disk check is empty

[MichielvM]

I have on disk check that shows up empty. Looking around on this forum, I found a similar post. So I did my homework!

Home/service detail - Advanced tab shows:
Performance Data: 'C: %'=35%;89;94 'C:'=65.306G;10;5;0;99.9

ll /usr/local/nagios/share/perfdata/

shows this particular host.

Code: Select all

drwxrwxr-x 2 nagios nagios 4096 Feb  3 17:41 HOST01

Output for

tail -25 /usr/local/nagios/var/perfdata.log

Code: Select all

2014-03-16 12:41:34 [26164] [0] *** TIMEOUT: Timeout after 5 secs. ***
2014-03-16 12:41:34 [26164] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2014-03-16 12:41:34 [26164] [0] *** TIMEOUT: Please check your npcd.cfg
2014-03-16 12:41:34 [26164] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1394969926.perfdata.service-PID-26164 deleted
2014-03-16 12:41:34 [26163] [0] *** TIMEOUT: Timeout after 5 secs. ***
2014-03-16 12:41:34 [26163] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2014-03-16 12:41:34 [26163] [0] *** TIMEOUT: Please check your npcd.cfg
2014-03-16 12:41:37 [26164] [0] *** Timeout while processing Host: "spozei-veeam01" Service: "CPU_Usage"
2014-03-16 12:41:37 [26164] [0] *** process_perfdata.pl terminated on signal ALRM
2014-03-16 12:41:57 [26163] [0] *** TIMEOUT: Could not delete /usr/local/nagios/var/spool/perfdata//1394969926.perfdata.host-PID-26163:No such file or directory
2014-03-16 12:41:59 [26163] [0] *** Timeout while processing Host: "" Service: ""
2014-03-16 12:41:59 [26163] [0] *** process_perfdata.pl terminated on signal ALRM
2014-07-24 13:16:24 [31070] [0] *** TIMEOUT: Timeout after 5 secs. ***
2014-07-24 13:16:24 [31070] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2014-07-24 13:16:24 [31070] [0] *** TIMEOUT: Please check your npcd.cfg
2014-07-24 13:16:24 [31069] [0] *** TIMEOUT: Timeout after 5 secs. ***
2014-07-24 13:16:24 [31070] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1406200371.perfdata.host-PID-31070 deleted
2014-07-24 13:16:24 [31069] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2014-07-24 13:16:24 [31069] [0] *** TIMEOUT: Please check your npcd.cfg
2014-07-24 13:16:24 [31069] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1406200371.perfdata.service-PID-31069 deleted
2014-07-24 13:16:24 [31070] [0] *** Timeout while processing Host: "" Service: ""
2014-07-24 13:16:24 [31069] [0] *** Timeout while processing Host: "" Service: ""
2014-07-24 13:16:24 [31070] [0] *** process_perfdata.pl terminated on signal ALRM
2014-07-24 13:16:24 [31069] [0] *** process_perfdata.pl terminated on signal ALRM

and

tail -25 /usr/local/nagios/var/npcd.log

Code: Select all

[01-26-2015 01:02:57] NPCD: WARN: MAX load reached: load 78.610000/10.000000 at i=1
[01-26-2015 01:03:12] NPCD: WARN: MAX load reached: load 100.290000/10.000000 at i=1
[01-26-2015 01:03:27] NPCD: WARN: MAX load reached: load 100.260000/10.000000 at i=1
[01-26-2015 01:03:42] NPCD: WARN: MAX load reached: load 78.460000/10.000000 at i=1
[01-26-2015 01:03:57] NPCD: WARN: MAX load reached: load 61.220000/10.000000 at i=1
[01-26-2015 01:04:12] NPCD: WARN: MAX load reached: load 48.200000/10.000000 at i=1
[01-26-2015 01:04:27] NPCD: WARN: MAX load reached: load 37.600000/10.000000 at i=1
[01-26-2015 01:04:42] NPCD: WARN: MAX load reached: load 29.350000/10.000000 at i=1
[01-26-2015 01:04:57] NPCD: WARN: MAX load reached: load 22.850000/10.000000 at i=1
[01-26-2015 01:05:12] NPCD: WARN: MAX load reached: load 18.530000/10.000000 at i=1
[01-26-2015 01:05:27] NPCD: WARN: MAX load reached: load 14.490000/10.000000 at i=1
[01-26-2015 01:05:42] NPCD: WARN: MAX load reached: load 11.280000/10.000000 at i=1
[01-28-2015 09:32:09] NPCD: WARN: MAX load reached: load 11.290000/10.000000 at i=0
[01-28-2015 09:32:24] NPCD: WARN: MAX load reached: load 12.240000/10.000000 at i=1
[01-28-2015 09:32:39] NPCD: WARN: MAX load reached: load 17.050000/10.000000 at i=1
[01-28-2015 09:32:54] NPCD: WARN: MAX load reached: load 15.550000/10.000000 at i=1
[01-28-2015 09:33:09] NPCD: WARN: MAX load reached: load 13.570000/10.000000 at i=1
[01-28-2015 09:33:24] NPCD: WARN: MAX load reached: load 10.640000/10.000000 at i=1
[01-29-2015 10:23:58] NPCD: npcd Daemon (0.4.14) started with PID=7664
[01-29-2015 10:23:58] NPCD: Please have a look at 'npcd -V' to get license information
[01-29-2015 10:23:58] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-29-2015 11:00:47] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-29-2015 11:01:40] NPCD: npcd Daemon (0.4.14) started with PID=2249
[01-29-2015 11:01:40] NPCD: Please have a look at 'npcd -V' to get license information
[01-29-2015 11:01:40] NPCD: HINT: load_threshold is enabled - ('10.000000')

Check intervals are the same as other (working) checks.

Any pointers?

[lmiltchev]

Do you have any files (RRDs and XMLs) in the "HOST01" directory?

Code: Select all

ll /usr/local/nagios/share/perfdata/HOST01

Is npcd running at the moment?

Code: Select all

service npcd status

It seems that the load on your system went above the default load threshold of 10, specified in the npcd.cfg...

Have you tried restarting it?

Code: Select all

service npcd restart

Run the following commands and show us the output:

Code: Select all

ls /usr/local/nagios/var/spool/xidpe | wc -l
ls /usr/local/nagios/var/spool/perfdata | wc -l
ls /usr/local/nagios/var/spool/checkresults | wc -l

[MichielvM]

The HOST01 has both RRD and XML for this particular check.
NPCD is running
NPCD is defaul nistall. No modifications on our part. (checked against other Nagios server)
I have restarted, no difference.

ls /usr/local/nagios/var/spool/xidpe | wc -l
0
ls /usr/local/nagios/var/spool/perfdata | wc -l
0
ls /usr/local/nagios/var/spool/checkresults | wc -l
0

[abrist]

Follow the FAQ below and increase the logging verbosity for both perfdata.log and npcd.log. Wait 15 minutes and then check those logs again:
http://support.nagios.com/wiki/index.ph ... h_Problems

Code: Select all

tail -50 /usr/local/nagios/var/perfdata.log
tail -50 /usr/local/nagios/var/npcd.log

[MichielvM]

Problem solved.

It looks like the .RRD and the .XML files were locked in some way.
We did an edit and save without actually changing something.
That got the train rolling again.

[abrist]

Great! Locking up.