Page 2 of 4
Re: Graphs don't work every night
Posted: Thu May 07, 2015 1:30 pm
by bosecorp
I did that this morning. I changed that to 60 and restarted the services. that was this morning around 2 am. But still everything went back to normal around 3-4 am.
I could always stop backups tonight. but that is not ideal
Re: Graphs don't work every night
Posted: Thu May 07, 2015 1:46 pm
by abrist
Are there any nightly backups that you run or anything else that would cause load to spike into the 100s?
Until you solve your load problem, you will probably not be able to get graphing under control. What is load at currently?
Re: Graphs don't work every night
Posted: Thu May 07, 2015 1:50 pm
by lmiltchev
Your load went through the roof...
Code: Select all
Line 10357: [05-06-2015 05:51:44] NPCD: WARN: MAX load reached: load 314.510000/40.000000 at i=1
The "load_threshold" set to 60 won't help with such a high load values... Have you checked the I/O when the load is high? Have you considered moving your Nagios XI instance to a dedicated physical server?
Re: Graphs don't work every night
Posted: Thu May 07, 2015 2:05 pm
by bosecorp
so can increase to 300 or more. I included a screenshoot with the amount of CPU i have assigned to this VM
Re: Graphs don't work every night
Posted: Thu May 07, 2015 2:47 pm
by jdalrymple
Does your host have enough resources to allocate at this time of day? You may be exhausting the resources on the parent host and experience trickle down into the VM. One way to be sure would be to reserve CPU for that host, but your other guests may suffer as a result.
Re: Graphs don't work every night
Posted: Thu May 07, 2015 2:55 pm
by bosecorp
Yes, I do have enough resources on the host
Re: Graphs don't work every night
Posted: Thu May 07, 2015 3:00 pm
by jdalrymple
bosecorp wrote:Yes, I do have enough resources on the host
Clearly the guest does NOT have enough resources otherwise the load would not spike the way it did. Either we will have to determine what is running at that time to cause the high utilization or we will have to add more resources to the guest.
Do you have staff that can monitor the Nagios host's processes during the time when the graphs fail populate?
Re: Graphs don't work every night
Posted: Thu May 07, 2015 3:14 pm
by bosecorp
this is the load for the last week
one is CPU and the other is memory
Re: Graphs don't work every night
Posted: Thu May 07, 2015 3:17 pm
by lmiltchev
Hi I/O can cause high load as well. If you are sure that you have plenty of resources, you can increase the "load_threshold" to give it a try. I would also recommend increasing the timeout value in the "/usr/local/nagios/etc/pnp/process_perfdata.cfg" as you had bunch of timeouts in the perdata.log.
Restart npcd.
Let us know if this fixed your issue.
Re: Graphs don't work every night
Posted: Thu May 07, 2015 3:20 pm
by bosecorp
Hi dmiltchev
done
JR Dalrymple
please look at the performance graphs for this VM. took them from vCenter