Nagios Support Forum • All Performance Graphs Blank

Page 1 of 2

All Performance Graphs Blank

Posted: Wed Aug 12, 2015 11:19 am

by CFT6Server

Just noticed this morning that all our performance graphs stopped working an all checks with graphs are showing up blank. Looking for some assistance on where to start looking.

I checked "/usr/local/nagios/var/spool/perfdata" and looks like files are still being modified.

Re: All Performance Graphs Blank

Posted: Wed Aug 12, 2015 11:56 am

by CFT6Server

I restarted the Nagios services and the graphs are coming back, but I am definitely missing all performance data during the time when this was stuck. Would love to know what happened and how to prevent this.

Re: All Performance Graphs Blank

Posted: Wed Aug 12, 2015 12:22 pm

by tgriep

In this folder are where the log files are kept.

Code: Select all

/usr/local/nagios/var/

They are called

Code: Select all

npcd.log
perfdata.log

Take a look at them to see what happened.
Also, take a look at this link to help you trouble shoot this.
https://support.nagios.com/wiki/index.p ... h_Problems

Re: All Performance Graphs Blank

Posted: Wed Aug 12, 2015 2:09 pm

by CFT6Server

I am seeing a bit of these in the perfdata logs... coming from various host/services. There seems to be a few ever couple minutes.

Code: Select all

2015-08-12 11:45:51 [20232] [0] *** TIMEOUT: Timeout after 5 secs. ***
2015-08-12 11:45:51 [20232] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-12 11:45:51 [20232] [0] *** TIMEOUT: Please check your npcd.cfg
2015-08-12 11:45:51 [20232] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1439405123.perfdata.service-PID-20232 deleted
2015-08-12 11:45:51 [20232] [0] *** Timeout while processing Host: "kdcbchccvbd1" Service: "PageFile"
2015-08-12 11:45:51 [20232] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-12 11:46:11 [26295] [0] *** TIMEOUT: Timeout after 5 secs. ***
2015-08-12 11:46:11 [26295] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-12 11:46:11 [26295] [0] *** TIMEOUT: Please check your npcd.cfg
2015-08-12 11:46:11 [26295] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1439405153.perfdata.service-PID-26295 deleted
2015-08-12 11:46:11 [26295] [0] *** Timeout while processing Host: "kdcbchucs03-a_network" Service: "server_2_1_VHBA_fc0_Bandwidth"
2015-08-12 11:46:11 [26295] [0] *** process_perfdata.pl terminated on signal ALRM

Re: All Performance Graphs Blank

Posted: Wed Aug 12, 2015 2:36 pm

by tgriep

Go through the Perfdata Timeout section in the link I provided, that should fix that for you.

Re: All Performance Graphs Blank

Posted: Wed Aug 12, 2015 2:42 pm

by lmiltchev

You can increase the "default" timeout value in the "/usr/local/nagios/etc/pnp/process_perfdata.cfg" from this:

Code: Select all

TIMEOUT = 5

to this:

Code: Select all

TIMEOUT = 20

and restart npcd:

Code: Select all

service npcd restart

The npcd stops if the "load_threshold" value, defined in the "/usr/local/nagios/etc/pnp/npcd.cfg" has been exceeded. Have you noticed big load spikes on this system?

Do you have files that are piled up in the "xidpe", "perfdata" or "checkresults" directory? What is the output of the following commands?

Code: Select all

ls /usr/local/nagios/var/spool/xidpe | wc -l
ls /usr/local/nagios/var/spool/perfdata | wc -l
ls /usr/local/nagios/var/spool/checkresults | wc -l

Re: All Performance Graphs Blank

Posted: Tue Aug 25, 2015 9:53 am

by CFT6Server

So I am continuing to see blank performance graphs at night. It is triggered by something, looks like this is due to CPU load. During the day, the load is ok, but it spikes at night and then a few hours later, the performance graphs stops working.

here are the errors in the npcd log:

Code: Select all

]# tail -200 /usr/local/nagios/var/npcd.log
[08-23-2015 21:36:01] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 21:36:01] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440390913.perfdata.service'
[08-23-2015 21:36:01] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 21:36:01] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440390928.perfdata.service'
[08-23-2015 21:36:36] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 21:36:36] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440390959.perfdata.service'
[08-23-2015 21:36:36] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 21:36:36] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440390944.perfdata.service'
[08-23-2015 21:37:11] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 21:37:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440390988.perfdata.service'
[08-23-2015 21:37:11] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 21:37:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440390973.perfdata.service'
[08-23-2015 21:37:26] NPCD: WARN: MAX load reached: load 11.490000/10.000000 at i=0
[08-23-2015 21:37:41] NPCD: WARN: MAX load reached: load 12.450000/10.000000 at i=1
[08-23-2015 21:37:56] NPCD: WARN: MAX load reached: load 13.470000/10.000000 at i=1
[08-23-2015 21:38:11] NPCD: WARN: MAX load reached: load 14.280000/10.000000 at i=1
[08-23-2015 21:38:26] NPCD: WARN: MAX load reached: load 15.630000/10.000000 at i=1
[08-23-2015 21:38:41] NPCD: WARN: MAX load reached: load 13.840000/10.000000 at i=1
[08-23-2015 21:38:56] NPCD: WARN: MAX load reached: load 13.810000/10.000000 at i=1
[08-23-2015 21:39:11] NPCD: WARN: MAX load reached: load 12.490000/10.000000 at i=1
[08-23-2015 21:39:26] NPCD: WARN: MAX load reached: load 15.830000/10.000000 at i=1
[08-23-2015 21:39:41] NPCD: WARN: MAX load reached: load 20.720000/10.000000 at i=1
[08-23-2015 21:39:56] NPCD: WARN: MAX load reached: load 18.470000/10.000000 at i=1
[08-23-2015 21:40:11] NPCD: WARN: MAX load reached: load 15.290000/10.000000 at i=1
[08-23-2015 21:40:26] NPCD: WARN: MAX load reached: load 12.490000/10.000000 at i=1
[08-23-2015 21:40:41] NPCD: WARN: MAX load reached: load 10.550000/10.000000 at i=1
[08-23-2015 21:40:56] NPCD: WARN: MAX load reached: load 11.390000/10.000000 at i=1
[08-23-2015 21:41:11] NPCD: WARN: MAX load reached: load 11.510000/10.000000 at i=1
[08-23-2015 21:41:26] NPCD: WARN: MAX load reached: load 10.430000/10.000000 at i=1
[08-23-2015 21:42:01] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 21:42:01] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440391033.perfdata.service'
[08-23-2015 21:42:01] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 21:42:01] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440391004.perfdata.service'
[08-23-2015 21:42:39] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 21:42:39] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440391063.perfdata.service'
[08-23-2015 21:42:59] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 21:42:59] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440391094.perfdata.service'
[08-23-2015 21:43:38] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 21:43:38] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440391169.perfdata.service'
[08-23-2015 21:43:38] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 21:43:38] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440391183.perfdata.service'
[08-23-2015 22:27:16] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 22:27:17] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440394003.perfdata.service'
[08-23-2015 22:51:31] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 22:51:31] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440395443.perfdata.service'
[08-23-2015 23:01:38] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:01:38] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440396044.perfdata.service'
[08-23-2015 23:02:13] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:02:13] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440396074.perfdata.service'
[08-23-2015 23:26:56] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:26:56] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440397574.perfdata.service'
[08-23-2015 23:27:32] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:27:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440397604.perfdata.service'
[08-23-2015 23:28:07] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:28:07] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440397633.perfdata.service'
[08-23-2015 23:36:56] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:36:56] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440398173.perfdata.service'
[08-23-2015 23:50:37] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:50:37] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:50:37] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440398998.perfdata.host'
[08-23-2015 23:50:37] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:50:37] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440398983.perfdata.host'
[08-23-2015 23:50:37] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440398999.perfdata.service'
[08-23-2015 23:50:37] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:50:37] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440398984.perfdata.service'
[08-23-2015 23:50:52] NPCD: WARN: MAX load reached: load 11.260000/10.000000 at i=0
[08-23-2015 23:51:07] NPCD: WARN: MAX load reached: load 13.410000/10.000000 at i=1
[08-23-2015 23:51:22] NPCD: WARN: MAX load reached: load 17.410000/10.000000 at i=1
[08-23-2015 23:51:37] NPCD: WARN: MAX load reached: load 15.410000/10.000000 at i=1
[08-23-2015 23:51:52] NPCD: WARN: MAX load reached: load 14.810000/10.000000 at i=1
[08-23-2015 23:52:07] NPCD: WARN: MAX load reached: load 13.760000/10.000000 at i=1
[08-23-2015 23:52:22] NPCD: WARN: MAX load reached: load 12.890000/10.000000 at i=1
[08-23-2015 23:52:37] NPCD: WARN: MAX load reached: load 11.190000/10.000000 at i=1
[08-23-2015 23:53:14] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:53:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440399019.perfdata.service'
[08-23-2015 23:53:14] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:53:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440399013.perfdata.host'
[08-23-2015 23:53:14] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:53:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440399029.perfdata.service'
[08-23-2015 23:53:15] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:53:15] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440399028.perfdata.host'
[08-23-2015 23:53:30] NPCD: WARN: MAX load reached: load 34.510000/10.000000 at i=0
[08-23-2015 23:53:45] NPCD: WARN: MAX load reached: load 41.630000/10.000000 at i=1
[08-23-2015 23:54:00] NPCD: WARN: MAX load reached: load 41.780000/10.000000 at i=1
[08-23-2015 23:54:15] NPCD: WARN: MAX load reached: load 36.730000/10.000000 at i=1
[08-23-2015 23:54:30] NPCD: WARN: MAX load reached: load 33.410000/10.000000 at i=1
[08-23-2015 23:54:45] NPCD: WARN: MAX load reached: load 27.260000/10.000000 at i=1
[08-23-2015 23:55:00] NPCD: WARN: MAX load reached: load 24.090000/10.000000 at i=1
[08-23-2015 23:55:15] NPCD: WARN: MAX load reached: load 20.350000/10.000000 at i=1
[08-23-2015 23:55:30] NPCD: WARN: MAX load reached: load 17.040000/10.000000 at i=1
[08-23-2015 23:55:45] NPCD: WARN: MAX load reached: load 15.270000/10.000000 at i=1
[08-23-2015 23:56:00] NPCD: WARN: MAX load reached: load 13.070000/10.000000 at i=1
[08-23-2015 23:56:15] NPCD: WARN: MAX load reached: load 12.300000/10.000000 at i=1
[08-23-2015 23:56:30] NPCD: WARN: MAX load reached: load 10.690000/10.000000 at i=1
[08-23-2015 23:57:05] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:57:05] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440399063.perfdata.service'
[08-23-2015 23:57:05] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:57:05] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440399047.perfdata.service'
[08-23-2015 23:57:25] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:57:25] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440399074.perfdata.service'
[08-23-2015 23:57:26] NPCD: ERROR: Executed command exits with return code '7'
[08-23-2015 23:57:26] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440399108.perfdata.service'
[08-23-2015 23:57:26] NPCD: WARN: MAX load reached: load 10.790000/10.000000 at i=12
[08-23-2015 23:57:41] NPCD: WARN: MAX load reached: load 11.220000/10.000000 at i=12
[08-23-2015 23:57:56] NPCD: WARN: MAX load reached: load 11.250000/10.000000 at i=12
[08-24-2015 00:00:27] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 00:00:27] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 00:00:27] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440399553.perfdata.service'
[08-24-2015 00:00:27] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440399539.perfdata.host'
[08-24-2015 00:00:28] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 00:00:28] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440399568.perfdata.service'
[08-24-2015 00:00:28] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 00:00:28] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440399538.perfdata.service'
[08-24-2015 00:00:28] NPCD: WARN: MAX load reached: load 10.080000/10.000000 at i=7
[08-24-2015 00:11:50] NPCD: WARN: MAX load reached: load 10.650000/10.000000 at i=0
[08-24-2015 00:12:05] NPCD: WARN: MAX load reached: load 14.310000/10.000000 at i=1
[08-24-2015 00:12:20] NPCD: WARN: MAX load reached: load 13.200000/10.000000 at i=1
[08-24-2015 00:12:35] NPCD: WARN: MAX load reached: load 15.620000/10.000000 at i=1
[08-24-2015 00:12:50] NPCD: WARN: MAX load reached: load 13.190000/10.000000 at i=1
[08-24-2015 00:13:05] NPCD: WARN: MAX load reached: load 12.800000/10.000000 at i=1
[08-24-2015 00:13:20] NPCD: WARN: MAX load reached: load 10.550000/10.000000 at i=1
[08-24-2015 00:13:35] NPCD: WARN: MAX load reached: load 10.400000/10.000000 at i=1
[08-24-2015 00:22:35] NPCD: WARN: MAX load reached: load 11.020000/10.000000 at i=0
[08-24-2015 00:23:06] NPCD: WARN: MAX load reached: load 12.270000/10.000000 at i=0
[08-24-2015 00:23:21] NPCD: WARN: MAX load reached: load 10.080000/10.000000 at i=1
[08-24-2015 00:23:36] NPCD: WARN: MAX load reached: load 14.470000/10.000000 at i=1
[08-24-2015 00:23:51] NPCD: WARN: MAX load reached: load 11.630000/10.000000 at i=1
[08-24-2015 00:24:06] NPCD: WARN: MAX load reached: load 14.000000/10.000000 at i=1
[08-24-2015 00:24:21] NPCD: WARN: MAX load reached: load 11.280000/10.000000 at i=1
[08-24-2015 00:24:36] NPCD: WARN: MAX load reached: load 12.020000/10.000000 at i=1
[08-24-2015 00:25:07] NPCD: WARN: MAX load reached: load 10.460000/10.000000 at i=0
[08-24-2015 07:11:40] NPCD: WARN: MAX load reached: load 11.010000/10.000000 at i=0
[08-24-2015 07:11:55] NPCD: WARN: MAX load reached: load 10.190000/10.000000 at i=1
[08-24-2015 07:12:10] NPCD: WARN: MAX load reached: load 10.170000/10.000000 at i=1
[08-24-2015 07:12:44] NPCD: WARN: MAX load reached: load 13.320000/10.000000 at i=0
[08-24-2015 07:12:59] NPCD: WARN: MAX load reached: load 11.050000/10.000000 at i=1
[08-24-2015 18:01:52] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 18:01:52] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440464489.perfdata.service'
[08-24-2015 18:07:13] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 18:07:13] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440464788.perfdata.service'
[08-24-2015 18:07:49] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 18:07:49] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440464818.perfdata.service'
[08-24-2015 18:08:24] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 18:08:24] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440464878.perfdata.service'
[08-24-2015 18:17:28] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 18:17:28] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440465419.perfdata.service'
[08-24-2015 19:01:09] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 19:01:09] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440468028.perfdata.service'
[08-24-2015 19:30:14] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 19:30:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440469784.perfdata.service'
[08-24-2015 19:39:21] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 19:39:21] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440470338.perfdata.service'
[08-24-2015 20:29:40] NPCD: WARN: MAX load reached: load 11.030000/10.000000 at i=0
[08-24-2015 20:32:34] NPCD: WARN: MAX load reached: load 10.620000/10.000000 at i=0
[08-24-2015 21:06:01] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 21:06:01] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440475529.perfdata.service'
[08-24-2015 21:06:01] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 21:06:01] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440475514.perfdata.service'
[08-24-2015 21:07:09] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 21:07:09] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440475573.perfdata.service'
[08-24-2015 21:07:10] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 21:07:10] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440475589.perfdata.service'
[08-24-2015 21:07:25] NPCD: WARN: MAX load reached: load 10.190000/10.000000 at i=0
[08-24-2015 21:08:00] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 21:08:00] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440475619.perfdata.service'
[08-24-2015 21:08:00] NPCD: WARN: MAX load reached: load 10.640000/10.000000 at i=7
[08-24-2015 21:08:15] NPCD: WARN: MAX load reached: load 10.060000/10.000000 at i=7
[08-24-2015 21:08:30] NPCD: WARN: MAX load reached: load 11.310000/10.000000 at i=7
[08-24-2015 21:08:45] NPCD: WARN: MAX load reached: load 12.040000/10.000000 at i=7
[08-24-2015 21:09:00] NPCD: WARN: MAX load reached: load 10.890000/10.000000 at i=7
[08-24-2015 21:11:04] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 21:11:04] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440475799.perfdata.service'
[08-24-2015 21:11:24] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 21:11:24] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440475814.perfdata.service'
[08-24-2015 21:11:24] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 21:11:24] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440475828.perfdata.service'
[08-24-2015 21:11:59] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 21:11:59] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440475859.perfdata.service'
[08-24-2015 21:21:29] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 21:21:29] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440476459.perfdata.service'
[08-24-2015 21:22:04] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 21:22:04] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440476489.perfdata.service'
[08-24-2015 21:26:01] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 21:26:01] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440476729.perfdata.service'
[08-24-2015 23:27:01] NPCD: ERROR: Executed command exits with return code '7'
[08-24-2015 23:27:01] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440483988.perfdata.service'
[08-25-2015 00:22:29] NPCD: ERROR: Executed command exits with return code '7'
[08-25-2015 00:22:29] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440487318.perfdata.service'
[08-25-2015 00:28:55] NPCD: ERROR: Executed command exits with return code '7'
[08-25-2015 00:28:55] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440487694.perfdata.service'
[08-25-2015 01:32:14] NPCD: ERROR: Executed command exits with return code '7'
[08-25-2015 01:32:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440491504.perfdata.service'
[08-25-2015 01:37:24] NPCD: ERROR: Executed command exits with return code '7'
[08-25-2015 01:37:24] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440491804.perfdata.service'
[08-25-2015 01:47:13] NPCD: ERROR: Executed command exits with return code '7'
[08-25-2015 01:47:13] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440492403.perfdata.service'
[08-25-2015 02:39:25] NPCD: ERROR: Executed command exits with return code '7'
[08-25-2015 02:39:25] NPCD: ERROR: Executed command exits with return code '7'
[08-25-2015 02:39:25] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440495538.perfdata.host'
[08-25-2015 02:39:25] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1440495539.perfdata.service'

Looks like load issue and it can't recover afterwards?

Code: Select all

# ls /usr/local/nagios/var/spool/xidpe | wc -l
0
# ls /usr/local/nagios/var/spool/perfdata | wc -l
0
# ls /usr/local/nagios/var/spool/checkresults | wc -l
4

The graphs seems to have stopped after the 2:39 entry where there's no longer lines in the log.

I am also still seeing some timeouts, again, log stopped after 2:39

Code: Select all

2015-08-25 00:22:29 [28481] [0] *** TIMEOUT: Timeout after 20 secs. ***
2015-08-25 00:22:29 [28481] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-25 00:22:29 [28481] [0] *** TIMEOUT: Please check your npcd.cfg
2015-08-25 00:22:29 [28481] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1440487318.perfdata.service-PID-28481 deleted
2015-08-25 00:22:29 [28481] [0] *** Timeout while processing Host: "vmhost" Service: "Datastore_usage_for_VMHost"
2015-08-25 00:22:29 [28481] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-25 00:28:55 [23108] [0] *** TIMEOUT: Timeout after 20 secs. ***
2015-08-25 00:28:55 [23108] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-25 00:28:55 [23108] [0] *** TIMEOUT: Please check your npcd.cfg
2015-08-25 00:28:55 [23108] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1440487694.perfdata.service-PID-23108 deleted
2015-08-25 00:28:55 [23108] [0] *** Timeout while processing Host: "NetScaler_Prod_Secondary" Service: "vserver_-_lb_vserver_citrix_bchydro_com_ng_Connections"
2015-08-25 00:28:55 [23108] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-25 01:32:14 [22692] [0] *** TIMEOUT: Timeout after 20 secs. ***
2015-08-25 01:32:14 [22692] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-25 01:32:14 [22692] [0] *** TIMEOUT: Please check your npcd.cfg
2015-08-25 01:32:14 [22692] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1440491504.perfdata.service-PID-22692 deleted
2015-08-25 01:32:14 [22692] [0] *** Timeout while processing Host: "ZGW-INT-B01" Service: "visualsource_Bandwidth"
2015-08-25 01:32:14 [22692] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-25 01:37:24 [11213] [0] *** TIMEOUT: Timeout after 20 secs. ***
2015-08-25 01:37:24 [11213] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-25 01:37:24 [11213] [0] *** TIMEOUT: Please check your npcd.cfg
2015-08-25 01:37:24 [11213] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1440491804.perfdata.service-PID-11213 deleted
2015-08-25 01:37:24 [11213] [0] *** Timeout while processing Host: "L2E-LAN-A06" Service: "Bay7_Port1_Bandwidth"
2015-08-25 01:37:24 [11213] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-25 01:47:13 [32565] [0] *** TIMEOUT: Timeout after 20 secs. ***
2015-08-25 01:47:13 [32565] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-25 01:47:13 [32565] [0] *** TIMEOUT: Please check your npcd.cfg
2015-08-25 01:47:13 [32565] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1440492403.perfdata.service-PID-32565 deleted
2015-08-25 01:47:13 [32565] [0] *** Timeout while processing Host: "vm host" Service: "Distributed-Virtual-VMware-switch_-DvsPortset-0_Bandwidth"
2015-08-25 01:47:13 [32565] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-25 02:39:25 [27324] [0] *** TIMEOUT: Timeout after 20 secs. ***
2015-08-25 02:39:25 [27324] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-25 02:39:25 [27324] [0] *** TIMEOUT: Please check your npcd.cfg
2015-08-25 02:39:25 [27323] [0] *** TIMEOUT: Timeout after 20 secs. ***
2015-08-25 02:39:25 [27324] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1440495539.perfdata.service-PID-27324 deleted
2015-08-25 02:39:25 [27323] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-08-25 02:39:25 [27323] [0] *** TIMEOUT: Please check your npcd.cfg
2015-08-25 02:39:25 [27324] [0] *** Timeout while processing Host: "vm host"Service: "Distributed-Virtual-VMware-switch_-DvsPortset-2_Bandwidth"
2015-08-25 02:39:25 [27324] [0] *** process_perfdata.pl terminated on signal ALRM
2015-08-25 02:39:25 [27323] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1440495538.perfdata.host-PID-27323 deleted
2015-08-25 02:39:25 [27323] [0] *** Timeout while processing Host: "host" Service: "_HOST_"
2015-08-25 02:39:25 [27323] [0] *** process_perfdata.pl terminated on signal ALRM

Re: All Performance Graphs Blank

Posted: Tue Aug 25, 2015 10:15 am

by CFT6Server

Some more additional information. Reviewing the logs and looks like gearmand is erroring also.... Not sure if this is related to the picture overall.

Code: Select all

# tail -100 /var/log/gearmand.log
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 01:37:27.000000 [  main ] accept(Too many open files) -> libgearman-server/gearmand.cc:851
  ERROR 2015-08-24 06:23:59.000000 [     2 ] lost connection to client during send(EPIPE || ECONNRESET || EHOSTDOWN)(Connection reset by peer) -> libgearman-server/io.cc:287
  ERROR 2015-08-24 13:25:04.000000 [     3 ] lost connection to client during send(EPIPE || ECONNRESET || EHOSTDOWN)(Connection reset by peer) -> libgearman-server/io.cc:287
  ERROR 2015-08-24 15:57:56.000000 [     1 ] lost connection to client during send(EPIPE || ECONNRESET || EHOSTDOWN)(Connection reset by peer) -> libgearman-server/io.cc:287
  ERROR 2015-08-25 00:33:31.000000 [     3 ] lost connection to client during send(EPIPE || ECONNRESET || EHOSTDOWN)(Connection reset by peer) -> libgearman-server/io.cc:287
  ERROR 2015-08-25 01:49:34.000000 [     1 ] lost connection to client during send(EPIPE || ECONNRESET || EHOSTDOWN)(Connection reset by peer) -> libgearman-server/io.cc:287
  ERROR 2015-08-25 02:13:13.000000 [     2 ] lost connection to client during send(EPIPE || ECONNRESET || EHOSTDOWN)(Connection reset by peer) -> libgearman-server/io.cc:287

Re: All Performance Graphs Blank

Posted: Tue Aug 25, 2015 12:52 pm

by tgriep

Do you have any backups running or anything scheduled on the system during the time the performance graphs are failing?
Can you check the log files in /var/log folder for any clues?

Try to increase load threshold by editing this file:

Code: Select all

/usr/local/nagios/etc/pnp/npcd.cfg

Change:

Code: Select all

load_threshold = 10.0

To:

Code: Select all

load_threshold = 20.0

Save the file and restart NPCD:

Code: Select all

service npcd restart

I found this on the the gearman errors.
https://groups.google.com/forum/#!topic ... nGlRdYQeDM
It doesn't say much but I would guess the gearman workers were overloading the gearman server while the performance issue was happening. Can you check the gearman workers log files?

Re: All Performance Graphs Blank

Posted: Tue Aug 25, 2015 12:54 pm

by CFT6Server

I will increase the load threshold.

Also the scheduled backups happens at 9pm and this doesn't happen until hours later. but I do see load spike when backups are running, but it clears eventually.