Graphs don't work every night

[bosecorp]

I just noticed that most of my graphs don;t work every night, around midnight all graphs stop working. everything goes back to normal early in the morning.

this morning I noticed that it does not happen for all the devices every day

I am running backups every night. the backups start at mid-night and they are SSH backups. normally the backups take 2-3 hours. I am able to tell by going to Admin==>Load Bakcups. the time stamp is about 2-3 hours after I start my backups

could this be caused by backups or by some of other DB maintenance

[tmcdonald]

My guess is a combination of a long-running backup, and a difference between your system time and your PHP time. The backup causes your load to spike, which stops NPCD from processing perfdata. Then your time being different makes it look like it appeared later than it did.

In /usr/local/nagios/etc/pnp/npcd.cfg what is your load_threshold set to? You might try doubling it and restarting npcd if it is set to the default of 10.0.

[bosecorp]

here is my setting. I though the same. last night while this has happening I change the setting from 40 to 60, then restarted NCPD. I made the change around 2 am last night but everything recover between 3-4 am. in addition, the devices seem to recover in different times


# EXPERIMENTAL
#
# load_threshold - npcd won't start new threads
# if your system load is over this threshold
#
# load_threshold = <float value> (default: 10.0)
#
# Hint: Do not use "," as decimal delimeter
#

# Changed from 10.0 to 40.0 by JJ per Nagios support 9/11/2014

load_threshold = 60.0


also one more thing, the DB does not run on the XI server

[lmiltchev]

The default setting:

Code: Select all

load_threshold = 10.0

is for a single CPU. With dual core, you can change it to 20.0 (x2), quad core - to 40.0 (x4), etc. I am not sure how many CPUs you have and how high the load on your system can go, but you bumped the value up to 60 already, so I would recommend watching the load on the system for a while to see if it is going to exceed this value. Let us know if changing the load_threshold solved your problem.

[bosecorp]

my XI server is running on a VM

here is what I have

[tgriep]

Could you post your perfdata log file so we can see what the load was when the backup was running?
Here is where the file is located at on the Nagios server.

Code: Select all

/usr/local/nagios/var/perfdata.log

[bosecorp]

here you go

[tgriep]

Can you post this file too?

Code: Select all

/usr/local/nagios/var/npcd.log

[bosecorp]

here you go

[jolson]

It looks like the assumption that your load spikes in the early morning is correct. This could account for the discrepancy as tmcdonald has described:

Code: Select all

Line 10352: [05-06-2015 05:50:29] NPCD: WARN: MAX load reached: load 161.460000/40.000000 at i=1
Line 10353: [05-06-2015 05:50:44] NPCD: WARN: MAX load reached: load 127.900000/40.000000 at i=1
Line 10354: [05-06-2015 05:50:59] NPCD: WARN: MAX load reached: load 100.300000/40.000000 at i=1
Line 10355: [05-06-2015 05:51:14] NPCD: WARN: MAX load reached: load 203.380000/40.000000 at i=1
Line 10356: [05-06-2015 05:51:29] NPCD: WARN: MAX load reached: load 291.110000/40.000000 at i=1
Line 10357: [05-06-2015 05:51:44] NPCD: WARN: MAX load reached: load 314.510000/40.000000 at i=1
Line 10358: [05-06-2015 05:51:59] NPCD: WARN: MAX load reached: load 309.810000/40.000000 at i=1
Line 10363: [05-06-2015 05:52:29] NPCD: WARN: MAX load reached: load 188.390000/40.000000 at i=0
Line 10364: [05-06-2015 05:52:44] NPCD: WARN: MAX load reached: load 148.250000/40.000000 at i=1
Line 10365: [05-06-2015 05:52:59] NPCD: WARN: MAX load reached: load 187.170000/40.000000 at i=1
Line 10366: [05-06-2015 05:53:14] NPCD: WARN: MAX load reached: load 214.090000/40.000000 at i=1
Line 10367: [05-06-2015 05:53:29] NPCD: WARN: MAX load reached: load 167.100000/40.000000 at i=1
Line 10368: [05-06-2015 05:53:44] NPCD: WARN: MAX load reached: load 131.040000/40.000000 at i=1
Line 10369: [05-06-2015 05:53:59] NPCD: WARN: MAX load reached: load 103.340000/40.000000 at i=1
Line 10370: [05-06-2015 05:54:14] NPCD: WARN: MAX load reached: load 81.500000/40.000000 at i=1
Line 10371: [05-06-2015 05:54:29] NPCD: WARN: MAX load reached: load 63.670000/40.000000 at i=1
Line 10372: [05-06-2015 05:54:44] NPCD: WARN: MAX load reached: load 50.580000/40.000000 at i=1
Line 10373: [05-06-2015 05:54:59] NPCD: WARN: MAX load reached: load 41.920000/40.000000 at i=1
Line 11452: [05-07-2015 00:55:23] NPCD: WARN: MAX load reached: load 55.760000/40.000000 at i=0
Line 11453: [05-07-2015 00:55:39] NPCD: WARN: MAX load reached: load 62.390000/40.000000 at i=1
Line 11454: [05-07-2015 00:55:54] NPCD: WARN: MAX load reached: load 64.320000/40.000000 at i=1
Line 11455: [05-07-2015 00:56:10] NPCD: WARN: MAX load reached: load 59.800000/40.000000 at i=1
Line 11456: [05-07-2015 00:56:25] NPCD: WARN: MAX load reached: load 51.840000/40.000000 at i=1
Line 11457: [05-07-2015 00:56:40] NPCD: WARN: MAX load reached: load 43.290000/40.000000 at i=1
Line 11660: [05-07-2015 01:20:18] NPCD: WARN: MAX load reached: load 53.250000/40.000000 at i=0
Line 11661: [05-07-2015 01:20:33] NPCD: WARN: MAX load reached: load 63.030000/40.000000 at i=1
Line 11662: [05-07-2015 01:20:48] NPCD: WARN: MAX load reached: load 74.360000/40.000000 at i=1
Line 11663: [05-07-2015 01:21:03] NPCD: WARN: MAX load reached: load 88.290000/40.000000 at i=1
Line 11664: [05-07-2015 01:21:18] NPCD: WARN: MAX load reached: load 101.510000/40.000000 at i=1
Line 11665: [05-07-2015 01:21:33] NPCD: WARN: MAX load reached: load 101.960000/40.000000 at i=1
Line 11666: [05-07-2015 01:21:48] NPCD: WARN: MAX load reached: load 97.020000/40.000000 at i=1
Line 11667: [05-07-2015 01:22:03] NPCD: WARN: MAX load reached: load 84.170000/40.000000 at i=1
Line 11668: [05-07-2015 01:22:18] NPCD: WARN: MAX load reached: load 70.000000/40.000000 at i=1
Line 11669: [05-07-2015 01:22:33] NPCD: WARN: MAX load reached: load 54.930000/40.000000 at i=1
Line 11670: [05-07-2015 01:22:48] NPCD: WARN: MAX load reached: load 43.370000/40.000000 at i=1
Line 11699: [05-07-2015 01:25:38] NPCD: WARN: MAX load reached: load 56.180000/40.000000 at i=0
Line 11700: [05-07-2015 01:25:54] NPCD: WARN: MAX load reached: load 74.080000/40.000000 at i=1
Line 11701: [05-07-2015 01:26:09] NPCD: WARN: MAX load reached: load 85.060000/40.000000 at i=1
Line 11702: [05-07-2015 01:26:24] NPCD: WARN: MAX load reached: load 84.630000/40.000000 at i=1
Line 11703: [05-07-2015 01:26:39] NPCD: WARN: MAX load reached: load 80.050000/40.000000 at i=1
Line 11704: [05-07-2015 01:26:54] NPCD: WARN: MAX load reached: load 80.350000/40.000000 at i=1
Line 11705: [05-07-2015 01:27:09] NPCD: WARN: MAX load reached: load 71.570000/40.000000 at i=1
Line 11706: [05-07-2015 01:27:24] NPCD: WARN: MAX load reached: load 61.160000/40.000000 at i=1
Line 11707: [05-07-2015 01:27:39] NPCD: WARN: MAX load reached: load 51.450000/40.000000 at i=1
Line 11708: [05-07-2015 01:27:54] NPCD: WARN: MAX load reached: load 41.090000/40.000000 at i=1
Line 11741: [05-07-2015 01:30:39] NPCD: WARN: MAX load reached: load 46.060000/40.000000 at i=0
Line 11742: [05-07-2015 01:30:54] NPCD: WARN: MAX load reached: load 41.940000/40.000000 at i=1
Line 11743: [05-07-2015 01:31:09] NPCD: WARN: MAX load reached: load 41.400000/40.000000 at i=1

You may need to restart nagios and npcd for the new threshold of '60' to take place:

Code: Select all

service nagios restart
service npcd restart