Page 1 of 3

Performance graphs stopped updating

Posted: Wed Feb 09, 2011 10:17 am
by FLCUISIT
We have a situation where our performance graphs stopped updating over the weekend. It is happening to all of our graphs. We are on the current release of the XI R1.4B and would need help figuring out a solution.

Attached is a sample graph
performance.png
Thanks,

Kirk

Re: Performance graphs stopped updating

Posted: Wed Feb 09, 2011 10:35 am
by tonyyarusso
The first thing I'd try is going under Admin and select the "Action" button for "Performance Grapher", and restart that service.

Re: Performance graphs stopped updating

Posted: Wed Feb 09, 2011 11:13 am
by FLCUISIT
That did not change anything. We are running XI under the VMWare image on an ESX box. Most of the performance graphs seem to stop on around the 4th, but we have some that were working until about yesterday afternoon.

Re: Performance graphs stopped updating

Posted: Wed Feb 09, 2011 11:40 am
by mguthrie
Try the Admin->Reset Security Credentials and reset the credentials for the backend components.

Re: Performance graphs stopped updating

Posted: Wed Feb 09, 2011 1:15 pm
by FLCUISIT
That may have done something. It looks like though not current, that some of the graphs are building. Is there a process that I can see where it is reading from the database and writing to the performance graphs?

Re: Performance graphs stopped updating

Posted: Thu Feb 10, 2011 11:24 am
by FLCUISIT
The performance graphs still are not up to date, while very slowly updating. System status shows load in the 15-20 percent based on the 1-min, 5-min, and 15-min, while the memory is consistently near 2GB of usage (with no swap usage of the 1GB assigned). Any thoughts? Is there some performance tweaking that can be done?

Re: Performance graphs stopped updating

Posted: Thu Feb 10, 2011 1:50 pm
by tonyyarusso
System status shows load in the 15-20 percent based on the 1-min, 5-min, and 15-min
I believe this is your problem. Load is not given as a percentage, but rather a value relative to the number of CPU cores available. On most systems, values of 15-20 would be rather concerning. I would suggest shutting down the VM temporarily and assigning more CPU cores to it in the VMware settings (this solved a similar issue for another customer in December).

Re: Performance graphs stopped updating

Posted: Fri Feb 11, 2011 9:35 am
by FLCUISIT
Adding the CPU's helped for a while, but the load numbers have actually gone up since increasing the number of CPU's for the VM from 2 to 4, now sitting with values of 45-50 in the metrics, and is consuming around 8 GHz of processing on my ESX server. It still appears to stop producing occasionally, and based of other similar issues, restarting the following 3 processes (ndo2db, npcd, and nagios) helps restart the processing.

I can get you a sample of the running processes if that helps. One thing I notice is this process information occasionally:

nagios 1406 0.1 0.0 0 0 ? Z 09:46 0:00 [nagios] <defunct>

Any thoughts on what that means?

Re: Performance graphs stopped updating

Posted: Fri Feb 11, 2011 10:57 am
by rdedon
It may be service_perfdata_file_processing_command=process-service-perfdata-file-bulk in nagios.cfg


As actual name that is congigured - process-service-perfdata-file-pnp-bulk

Lets edit nagios.cfg -

Instead of:
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
host_perfdata_file_processing_command=process-host-perfdata-file-bulk

and change to:

service_perfdata_file_processing_command=process-service-perfdata-file-pnp-bulk
host_perfdata_file_processing_command=process-host-perfdata-file-pnp-bulk

Re: Performance graphs stopped updating

Posted: Fri Feb 11, 2011 11:00 am
by mguthrie
What kind of check load do you have on your system -> How many hosts and services are you monitoring on your server? Are you running mostly active or passive checks?

What kind of output do you get from running "top"?