Performance Graph broken

[vmesquita]

I changed the threshold to 80, since we have a virtual machine with 8 CPUs. This machine has 3Gb. And the virtual machine is currently alone in the host machine. The host machine is a Intel Xeon CPU E5320 @ 1.86GHz with 4 Gb memory.

I tried this commands, but still it doesn't work. However now I have a different result in the previous logfiles:

Code: Select all

==> /usr/local/nagios/var/perfdata.log <==
2013-09-13 11:41:07 [25931] [1] RRDs::update ERROR Unable to connect to rrdcached: No such file or directory
2013-09-13 11:41:07 [25931] [1] Found Performance Data for **** / GigabitEthernet2_0_22_Bandwidth (in=0Mb/s;2;5 out=0Mb/s;2;5)
2013-09-13 11:41:07 [25931] [1] RRDs::update ERROR Unable to connect to rrdcached: No such file or directory
2013-09-13 11:41:07 [25931] [1] Found Performance Data for *** / GigabitEthernet2_0_23_Bandwidth (in=0Mb/s;200;500 out=0Mb/s;200;500)
2013-09-13 11:41:07 [25931] [1] RRDs::update ERROR Unable to connect to rrdcached: No such file or directory
2013-09-13 11:41:07 [25931] [1] Found Performance Data for **** / CPU_Stats (user=0.00% system=0.03% iowait=0.00%;85;95 idle=99.97%)
2013-09-13 11:41:07 [25931] [1] RRDs::update ERROR Unable to connect to rrdcached: No such file or directory
2013-09-13 11:41:07 [25931] [1] 88 lines processed
2013-09-13 11:41:07 [25931] [1] /usr/local/nagios/var/spool/perfdata//1379083257.perfdata.service-PID-25931 deleted
2013-09-13 11:41:07 [25931] [1] PNP exiting (runtime 0.191066s) ...

Code: Select all

==> /usr/local/nagios/var/npcd.log <==
[09-13-2013 11:41:07] NPCD: No more files to process... waiting for 15 seconds
[09-13-2013 11:41:22] NPCD: Processing file '1379083272.perfdata.host'
[09-13-2013 11:41:22] NPCD: Processing file '1379083272.perfdata.service'
[09-13-2013 11:41:22] NPCD: No more files to process... waiting for 15 seconds
[09-13-2013 11:41:37] NPCD: Processing file '1379083287.perfdata.host'
[09-13-2013 11:41:37] NPCD: Processing file '1379083287.perfdata.service'
[09-13-2013 11:41:37] NPCD: No more files to process... waiting for 15 seconds
[09-13-2013 11:41:52] NPCD: Processing file '1379083302.perfdata.host'
[09-13-2013 11:41:52] NPCD: Processing file '1379083302.perfdata.service'
[09-13-2013 11:41:53] NPCD: No more files to process... waiting for 15 seconds

[abrist]

You may have a half-baked rrdcached deployment. Did you follow the entirety of the rrdcached doc?
http://assets.nagios.com/downloads/nagi ... ios_XI.pdf

[vmesquita]

I disable rrdcached by modifying RRD_DAEMON_OPTS

And it's working again. Actually rrdcached was woking previously, but after the nagios server run out of space, something must have happened that it got broken. Also CPU usage seems to have dropped:

Code: Select all

top - 14:05:31 up 44 min,  1 user,  load average: 4.69, 10.11, 12.08
Tasks: 301 total,  24 running, 275 sleeping,   0 stopped,   2 zombie
Cpu(s): 23.4%us, 32.7%sy,  0.0%ni, 41.7%id,  2.1%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   3115156k total,  1625048k used,  1490108k free,    66224k buffers
Swap:  4194296k total,        0k used,  4194296k free,   915400k cached

[sreinhardt]

Great! Are you planning on resolving the rrdcached lockup, or leaving as is?

[vmesquita]

I would like to fix it. I reenabled again, and the graphs stopped bring updated. How can I investigate this:

[sreinhardt]

First steps would be to ll your journaling directory and see how recent updates are, and what permissions are. Our documentation goes off of /tmp, but you could put this anywhere.

[vmesquita]

The updates are quite recent:

Code: Select all

-rwxrwxrwx 1 nagios users 0 Sep 13 16:17 rrd.journal.1379099852.140868
-rw-r--r-- 1 nagios users 0 Sep 13 17:17 rrd.journal.1379103453.155199

Note: premission on the first file was changed manually in an attempt to fix the issue.

[scottwilkerson]

Did you follow the entirety of the rrdcached doc?
http://assets.nagios.com/downloads/nagi ... ios_XI.pdf

Can you verify each step?

Did you deviate from this setup at all?

[vmesquita]

Since I am on RHEL 5, I also needed to edit the file /etc/sysconfig/rrdcached

Code: Select all

# Settings for rrdcached
#OPTIONS="-l unix:/var/rrdtool/rrdcached/rrdcached.sock -s rrdcached -m 664 -b /var/rrdtool/rrdcached"
#RRDC_USER=rrdcached
OPTIONS="-l unix:/var/rrdtool/rrdcached/rrdcached.sock -F -s nagios -m 0660 -w 900 -z 90 -j /tmp/ -b /var/rrdtool/rrdcached -P FLUSH,PENDING"
RRDC_USER=nagios

Before I could get this to work, in the past. But right now, of couse, it doesn't any more.

[abrist]

Are you currently receiving the same errors in npcd.log/perfdata.log as your previous post?