rrdcached Problem

[mikew]

Nagios graphing has a problem which just started. It looks like graphs are now building about an hour behind. The server has plenty of CPU (16 cores), 32 GB of RAM, resources do not look like an issue.

/var/log/messages
Jun 11 10:02:44 nag rrdcached[5586]: queue_thread_main: rrd_update_r (/app01/perfdata/extn/Check_Disk.rrd) failed with status -1. (/app01/perfdata/extn/Check_Disk.rrd: expected 45 data source readings (got 36) from 1402497763)
Jun 11 10:03:45 nag rrdcached[5586]: queue_thread_main: rrd_update_r (/app01/perfdata/cmin/Check_Disk.rrd) failed with status -1. (/app01/perfdata/cmin/Check_Disk.rrd: found extra data on update argument: 445185)

Early this morning I saw a huge spike in the ramdisk size which probably relates to this problem. That has been resolved and returned to normal.



CentOs 6.4 64_bit
rrdcached is used
rrdcached has been restarted with no errors
nagios restarted
npcd restarted (nothing significant in log)

[lmiltchev]

Mike, run the following commands and show us the output:

Code: Select all

grep rrdcached /usr/local/nagios/etc/pnp/process_perfdata.cfg
grep date.timezone /etc/php.ini
cat /etc/sysconfig/clock
date

[mikew]

Here is the output.


grep rrdcached /usr/local/nagios/etc/pnp/process_perfdata.cfg
# EXPERIMENTAL rrdcached Support
# RRD_DAEMON_OPTS = unix:/tmp/rrdcached.sock
RRD_DAEMON_OPTS = unix:/var/rrdtool/rrdcached/rrdcached.sock
RRD_DAEMON_OPTS = unix:/var/rrdtool/rrdcached/rrdcached.sock

grep date.timezone /etc/php.ini
; http://www.php.net/manual/en/datetime.c ... e.timezone
date.timezone = America/Chicago

cat /etc/sysconfig/clock
ZONE="America/Chicago"

Wed Jun 11 19:35:34 CDT 2014


Ludmil...good to see your picture on the new Nagios. I tell every customer what a great guy you are to have as a representative of the company...

[slansing]

Was this specifically after you built rrdcached in? Or after an upgrade to XI 2014? There were changes made between the version of plugins that was present throughout 2012, and the one released with 2014. Some of these changes may have altered the amount of performance data coming in, and it may be complaining about not having a place to put that data in the already created RRD's, or it may have less data.

[lmiltchev]

Thanks, Mike! I am honored to be on the Nagios XI's Home page, even though it makes me a bit uncomfortable. I always change the "default" home page on my Nagios XI test boxes to something else, so I won't have to look at myself all the time.

Have you done any changes to the system, prior to noticing the error? Have you updated the plugins? This seems to be an issue with the # of data sources. What is the version of the plugin that you are using in your command?