Performance Graphs Showing Data during Host Down Time Periods

Why are performance graphs displaying data during periods when the host was down?

Overview

The data that makes up the graphs are stored in rrd files found under /usr/local/nagios/share/perfdata/<hostname>/. These files also hold information about how the data is stored and ultimately displayed, including the heartbeat setting "minimal_heartbeat" which by default holds the value 8460(seconds). If a host or service becomes unavailable for a period that does not exceed the heartbeat limit then a value is calculated for the time intervals that went missing. If the heartbeat value is exceeded, then no values will be stored and the graph will be blank for that time period.

Viewing the Heartbeat Value

To see the values and change them we can use the rrdtool command. For example:

rrdtool dump /usr/local/nagios/share/perfdata/localhost/PING.rrd | more

This will display something like:

<ds>
<name> 1 </name>
<type> GAUGE </type>
<minimal_heartbeat>8460</minimal_heartbeat>
<min>NaN</min>
<max>NaN</max>

<!-- PDP Status -->
<last_ds>0.145000</last_ds>
<value>1.7400000000e+00</value>
<unknown_sec> 0 </unknown_sec>
</ds>

<ds>
<name> 2 </name>
<type> GAUGE </type>
<minimal_heartbeat>8460</minimal_heartbeat>
<min>NaN</min>
<max>NaN</max>


Note that there are two data sources in the PING.rrd file - 1 and 2. The number of data sources and names can vary.

Changing the Heartbeat Value

Option 1

To change the heartbeat on the above you would run:

rrdtool tune --heartbeat 1:60
rrdtool tune --heartbeat 2:60

This would change the heartbeat for both data the data sources named "1" and "2" which you can then see by running the 'rrdtool dump' command again.

You would need to run these commands for each existing RRD file.

Option 2

Another option would be to modify /usr/local/nagios/etc/pnp/process_perfdata.cfg which includes this line:

RRD_HEARTBEAT = 8460

Change the value to the desired heartbeat length and then delete all the existing RRD files(or the ones you want to have the default value changed in). This method does remove all the old data and then creates new RRD files with the new default value. New hosts and services going forward would get this default value as well.