Performance Graph broken

[vmesquita]

no, now I am getting:

Code: Select all

2013-09-16 12:47:58 [19017] [1] RRDs::update ERROR Unable to connect to rrdcached: No such file or directory
2013-09-16 12:47:58 [19017] [1] Found Performance Data for ******/ GigabitEthernet2_0_36_Bandwidth (in=0Mb/s;2;5 out=0Mb/s;2;5)
2013-09-16 12:47:58 [19017] [1] RRDs::update ERROR Unable to connect to rrdcached: No such file or directory
2013-09-16 12:47:58 [19017] [1] Found Performance Data for ****** / GigabitEthernet2_0_15_Bandwidth (in=0Mb/s;2;5 out=0Mb/s;2;5)
2013-09-16 12:47:58 [19017] [1] RRDs::update ERROR Unable to connect to rrdcached: No such file or directory
2013-09-16 12:47:58 [19017] [1] Found Performance Data for ****** / GigabitEthernet1_0_26_Bandwidth (in=0Mb/s;2;5 out=0Mb/s;2;5)
2013-09-16 12:47:58 [19017] [1] RRDs::update ERROR Unable to connect to rrdcached: No such file or directory
2013-09-16 12:47:58 [19017] [1] 59 lines processed
2013-09-16 12:47:58 [19017] [1] /usr/local/nagios/var/spool/perfdata//1379346463.perfdata.service-PID-19017 deleted
2013-09-16 12:47:58 [19017] [1] PNP exiting (runtime 0.114276s) ...

and

Code: Select all

[09-16-2013 12:48:13] NPCD: Processing file '1379346478.perfdata.service'
[09-16-2013 12:48:13] NPCD: Processing file '1379346492.perfdata.host'
[09-16-2013 12:48:14] NPCD: No more files to process... waiting for 15 seconds
[09-16-2013 12:48:29] NPCD: Processing file '1379346493.perfdata.service'
[09-16-2013 12:48:29] NPCD: No more files to process... waiting for 15 seconds
[09-16-2013 12:48:44] NPCD: Processing file '1379346507.perfdata.host'
[09-16-2013 12:48:44] NPCD: Processing file '1379346509.perfdata.service'
[09-16-2013 12:48:44] NPCD: Processing file '1379346522.perfdata.host'
[09-16-2013 12:48:44] NPCD: Processing file '1379346523.perfdata.service'
[09-16-2013 12:48:45] NPCD: No more files to process... waiting for 15 seconds

Aparently there's a file missing, but I can't figure which...

[abrist]

Is your rrdcached journals on /tmp? If so, do you have /tmp on a separate mount point? If so, what is the disk usage for the mount point? (You can get this error if you are out of space)

[vmesquita]

The rrd.journal files are on /tmp, in the same mouting point as /. Disk Usage is only 42%. But we have a separated mouting point for /usr/local which ran out of space 2 weeks ago (we were not monitoring this partition) and caused some issues.

[abrist]

What are the contents of:

Code: Select all

ls -la /var/rrdtool/rrdcached

And:

Code: Select all

cat /etc/sysconfig/rrdcached

[vmesquita]

Code: Select all

[root@nagios ~]# ls -la /var/rrdtool/rrdcached
total 20
drwxrwxr-x 2 nagios rrdcached 4096 Sep 16 11:45 .
drwxr-xr-x 3 root   root      4096 Mar 20  2011 ..
-rw-r--r-- 1 nagios users        6 Sep 16 11:45 rrdcached.pid
srwxr-xr-x 1 nagios users        0 Sep 16 11:45 rrdcached.sock

Code: Select all

# Settings for rrdcached
#OPTIONS="-l unix:/var/rrdtool/rrdcached/rrdcached.sock -s rrdcached -m 664 -b /var/rrdtool/rrdcached"
#RRDC_USER=rrdcached
OPTIONS="-l unix:/var/rrdtool/rrdcached/rrdcached.sock -F -s nagios -m 0660 -w 900 -z 90 -j /tmp/ -b /var/rrdtool/rrdcached -P FLUSH,PENDING"
RRDC_USER=nagios

[abrist]

Those look fine. Lets check /tmp:

Code: Select all

ls -lad /tmp
ls -la /tmp | grep rrd

[vmesquita]

ok:

Code: Select all

[root@nagios ~]# ls -lad /tmp
drwxrwxrwt 9 root root 90112 Sep 16 15:54 /tmp
[root@nagios ~]# ls -la /tmp | grep rrd
-rw-r--r--  1 nagios   users        0 Sep 16 14:45 rrd.journal.1379353534.183086
-rw-r--r--  1 nagios   users        0 Sep 16 15:45 rrd.journal.1379357135.205812
-rwxr-xr-x  1 root     root      3270 Jan 18  2013 xi-rrdcached.sh

[abrist]

You really should only have 1 rrdcached journal. Lets remove both, and then restart rrdcached:

Code: Select all

rm /tmp/rrd.journal.*
service rrdcached restart

[vmesquita]

ok, done. Also, are this files supposed to be empty?

[abrist]

No, they should fill up and empty out. It may take 5 - 15 minutes before you see graphs.