Perfdata Stop Working!

[lb2cons]

Yesterday my Perfdata stop working.

When I checked some graphs I've discovered every graph stopped plotting after yesterday (2015-23-08) 05:00 PM. So I simple restarted the Performance Grapher and everything worked again. Another interesting fact is that every XI System Component was with status OK.

Screen Shot 2015-08-24 at 09.29.30.png

Screen Shot 2015-08-24 at 09.34.21.png

My question is how to further investigate what happen. What logs should I check?

Performance graphs working it's crucial for me because I've send graphs reports for customers and I don't want to find that my Perf. isn't working after 15 days or more.

My second question is how can I prevent this?

[tgriep]

We would have to look at these logs to determine the failure. Can you run the following and post back the results?

Code: Select all

tail -50 /usr/local/nagios/var/perfdata.log
tail -50 /usr/local/nagios/var/npcd.log

[lb2cons]

I've attached the log from the time performance data stopped working.

Note that on both logs there's a gap of information from day 23 17:27 to day 24 08:56 that is the time I've restarted monitoring engine.

perfdata.log

npcd.log

The log doesn't tell nothing for me. Please report if you need more information.

[jolson]

One thing to take a look at is the load of your Nagios XI server versus the threshold that you have set up for performance data processing:

Load:

Code: Select all

sar

Threshold:

Code: Select all

grep threshold /usr/local/nagios/etc/pnp/npcd.cfg

By default, performance data will not be processed if your load is sustained at greater than 10 percent. Is it possible that your load was higher than the threshold during the period that your perfdata stopped processing?

You will want to take a look at this FAQ entry: https://support.nagios.com/wiki/index.p ... h_Problems

[ssax]

If you look at this FAQ entry here, what is your TIMEOUT and load_threshold set to?

https://support.nagios.com/wiki/index.p ... ta_Timeout

[lb2cons]

Jolson, the load was above "3" and the threshold was 10.

I've read the docs that both of you sent and updated load_threshold and timeout just in case.

[jolson]

Are your perfdata graphs working now? It will be worth watching your perfdata.log for any additional timeouts that might occur.

You will want to see how many files have piled up in the xidpe directory:

Code: Select all

ls -l /usr/local/nagios/var/spool/xidpe | wc -l

Is the number of files in that directory decreasing?

[lb2cons]

Jolson, yes, it's decreasing!

[jolson]

That's good news! You should see your performance graphs populating appropriately at this point. Let us know if you experience any other difficulties!