Performance graphs stopped updating

[FLCUISIT]

We have a situation where our performance graphs stopped updating over the weekend. It is happening to all of our graphs. We are on the current release of the XI R1.4B and would need help figuring out a solution.

Attached is a sample graph

performance.png

Thanks,

Kirk

[tonyyarusso]

The first thing I'd try is going under Admin and select the "Action" button for "Performance Grapher", and restart that service.

[FLCUISIT]

That did not change anything. We are running XI under the VMWare image on an ESX box. Most of the performance graphs seem to stop on around the 4th, but we have some that were working until about yesterday afternoon.

[mguthrie]

Try the Admin->Reset Security Credentials and reset the credentials for the backend components.

[FLCUISIT]

That may have done something. It looks like though not current, that some of the graphs are building. Is there a process that I can see where it is reading from the database and writing to the performance graphs?

[FLCUISIT]

The performance graphs still are not up to date, while very slowly updating. System status shows load in the 15-20 percent based on the 1-min, 5-min, and 15-min, while the memory is consistently near 2GB of usage (with no swap usage of the 1GB assigned). Any thoughts? Is there some performance tweaking that can be done?

[tonyyarusso]

System status shows load in the 15-20 percent based on the 1-min, 5-min, and 15-min

I believe this is your problem. Load is not given as a percentage, but rather a value relative to the number of CPU cores available. On most systems, values of 15-20 would be rather concerning. I would suggest shutting down the VM temporarily and assigning more CPU cores to it in the VMware settings (this solved a similar issue for another customer in December).

[FLCUISIT]

Adding the CPU's helped for a while, but the load numbers have actually gone up since increasing the number of CPU's for the VM from 2 to 4, now sitting with values of 45-50 in the metrics, and is consuming around 8 GHz of processing on my ESX server. It still appears to stop producing occasionally, and based of other similar issues, restarting the following 3 processes (ndo2db, npcd, and nagios) helps restart the processing.

I can get you a sample of the running processes if that helps. One thing I notice is this process information occasionally:

nagios 1406 0.1 0.0 0 0 ? Z 09:46 0:00 [nagios] <defunct>

Any thoughts on what that means?

[rdedon]

It may be service_perfdata_file_processing_command=process-service-perfdata-file-bulk in nagios.cfg


As actual name that is congigured - process-service-perfdata-file-pnp-bulk

Lets edit nagios.cfg -

Instead of:
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
host_perfdata_file_processing_command=process-host-perfdata-file-bulk

and change to:

service_perfdata_file_processing_command=process-service-perfdata-file-pnp-bulk
host_perfdata_file_processing_command=process-host-perfdata-file-pnp-bulk

[mguthrie]

What kind of check load do you have on your system -> How many hosts and services are you monitoring on your server? Are you running mostly active or passive checks?

What kind of output do you get from running "top"?