Broke Charting again...

[jlmoldan]

It is a combination of things. We recently Yum updated the box - which looks to be around the time that charting stopped. We also updated froom 2014R1.3 to 2.3 yesterday, noticed that they weren't working, and then saw that 2.4 was released so upgraded this morning to 2.4 to see if that helps, but it didn't. Both Test and Prod are not working now.

Both are CentOS 6.6/64bit, running 2014R2.4 XI (attached prod system profile) Test was a deployed VM, and prod was a custom install to match test.. Or close to it.)

Looks like /usr/local/nagios/share/perfdata RRDs are not updating.

Ran basic troubleshooting from http://support.nagios.com/wiki/index.ph ... leshooting

cat /usr/local/nagios/etc/pnp/process_perfdata.cfg |grep LOG_LEVEL
LOG_LEVEL = 2

ATTACHED /usr/local/nagios/var/perfdata.log (from test and prod)

cat /usr/local/nagios/etc/pnp/npcd.cfg |grep log_level
log_level = -1

ATTACHED /usr/local/nagios/var/npcd.log from both


tail -50 /usr/local/nagios/var/perfdata.log | grep TIMEOUT --> NONE

tail -50 /usr/local/nagios/var/npcd.log | grep "MAX load reached" --> NONE

Section: Performance Graphs Are Missing Or Not Displayed
Perfdata Permissions drwxrwxr-x. 492 nagios nagios 24576 Jan 20 11:29 perfdata

Ran /usr/local/nagiosxi/scripts/reset_config_perms

touched /var/lock/mrtg for the fun of it. nothing.

[root@ust-nagios1p /]# ls /usr/local/nagios/var/spool/xidpe | wc -l
2

[root@ust-nagios1p /]# ls /usr/local/nagios/var/spool/perfdata | wc -l
2

ls /usr/local/nagios/var/spool/checkresults | wc -l
0

tried disabling HighCharts for perfdata as per another random charting topic in the forum and that didn't work.

EDIT: removed prod-profile.zip

[lmiltchev]

Can you run the following commands and show us the output?

Code: Select all

which rrdtool
whereis rrdtool
rrdtool --version
grep rrdcached /usr/local/nagios/etc/pnp/process_perfdata.cfg

Have you tried deleting the RRDs and XMLs for one of your hosts/services to see if they are going to be recreated and the graphs will appear? You can also set up a new host, wait for 15-20 min and check to see if perf graphs are available.

[jlmoldan]

When I remove the .rrd from /usr/local/nagios/share/perfdata - it will recreate them new, but then never update them again. I removed a couple from the test system, and then added a whole new server and it had the same results. Here are your other commands. Thx.


[root@ust-nagios1t ~]# which rrdtool
/usr/bin/rrdtool
[root@ust-nagios1t ~]# whereis rrdtool
rrdtool: /usr/bin/rrdtool /usr/share/rrdtool /usr/share/man/man1/rrdtool.1.gz
[root@ust-nagios1t ~]# rrdtool --version
RRDtool 1.3.8 Copyright 1997-2009 by Tobias Oetiker <[email protected]>
Compiled Apr 3 2014 13:07:03

Usage: rrdtool [options] command command_options

Valid commands: create, update, updatev, graph, graphv, dump, restore,
last, lastupdate, first, info, fetch, tune,
resize, xport

RRDtool is distributed under the Terms of the GNU General
Public License Version 2. (www.gnu.org/copyleft/gpl.html)

For more information read the RRD manpages

[root@ust-nagios1t ~]# grep rrdcached /usr/local/nagios/etc/pnp/process_perfdata.cfg
# EXPERIMENTAL rrdcached Support
# RRD_DAEMON_OPTS = unix:/tmp/rrdcached.sock
RRD_DAEMON_OPTS = unix:/var/rrdtool/rrdcached/rrdcached.sock

[scottwilkerson]

Ok, being you are using rrdcached, what are the settings for the flush time?

Code: Select all

cat /etc/sysconfig/rrdcached

The graphs will not update until the rrdcached flush time is reached.

[jlmoldan]

[root@ust-nagios1t ~]# cat /etc/sysconfig/rrdcached
OPTIONS="-s nagios -m 0660 -l unix:/var/rrdtool/rrdcached/rrdcached.sock -F -w 900 -z 90 -j /tmp/ -b /var/rrdtool/rrdcached -P FLUSH,PENDING"
RRDC_USER=nagios

[lmiltchev]

Do you have a "rrd.journal" file in the "/tmp" directory and what is the timestamp on it? Is it updating every 15 min?

Code: Select all

ll /tmp | grep journal

[jlmoldan]

looks like they are updating hourly.

[root@ust-nagios1t ~]# ll /tmp | grep journal
-rw-r--r-- 1 nagios users 0 Jan 27 12:06 rrd.journal.1422382000.586888
-rw-r--r-- 1 nagios users 0 Jan 27 13:06 rrd.journal.1422385600.586905
[root@ust-nagios1t ~]#


[root@ust-nagios1p ~]# ll /tmp | grep journal
-rw-r--r-- 1 nagios users 0 Jan 27 12:48 rrd.journal.1422384501.504505
-rw-r--r-- 1 nagios users 0 Jan 27 13:48 rrd.journal.1422388101.504521

[cmerchant]

The journal file gets updated every 3600 seconds(default) so that you don't loose rrd data when the system is shutdown. Every 15 minutes the rrdcache gets flushed to disk (-w 900).

[scottwilkerson]

It is weird your journal files are empty...

A couple more things, lets run

Code: Select all

service rrdcached status
ls -la /var/rrdtool/rrdcached
ls -ld /tmp
cat /etc/mrtg/mrtg.cfg|grep LibAdd

[jlmoldan]

For test - and Prod. Both don't work. I do remember an upgrade or 2 before some charting issues and I ran some command to find the "actual" rrd cache as per some other Forum and changed the path in prod and that fixed it back then. (just a little historical info) Looks like the pid on both systems is when it stopped working.

[root@ust-nagios1t ~]# ls -la /var/rrdtool/rrdcached
total 12
drwxr-xr-x 2 nagios nagios 4096 Jan 15 10:06 .
drwxr-xr-x 3 nagios nagios 4096 Jun 25 2014 ..
-rw-r--r-- 1 nagios users 5 Jan 15 10:06 rrdcached.pid
srw-rw---- 1 nagios nagios 0 Jan 15 10:06 rrdcached.sock
[root@ust-nagios1t ~]# ls -ld /tmp
drwxrwxrwt. 17 root root 86016 Jan 28 09:26 /tmp
[root@ust-nagios1t ~]# cat /etc/mrtg/mrtg.cfg|grep LibAdd
LibAdd: /opt/rrdtool-1.4.4/lib/perl/5.10.1
[root@ust-nagios1t ~]#


[root@ust-nagios1p ~]# service rrdcached status
rrdcached (pid 22963) is running...
[root@ust-nagios1p ~]# ls -la /var/rrdtool/rrdcached
total 12
drwxr-xr-x 2 nagios users 4096 Jan 21 12:48 .
drwxr-xr-x 3 nagios nagios 4096 Sep 9 2013 ..
-rw-r--r-- 1 nagios users 6 Jan 21 12:48 rrdcached.pid
srw-rw---- 1 nagios nagios 0 Jan 21 12:48 rrdcached.sock
[root@ust-nagios1p ~]# ls -ld /tmp
drwxrwxrwt. 9 root root 135168 Jan 28 09:25 /tmp
[root@ust-nagios1p ~]# cat /etc/mrtg/mrtg.cfg|grep LibAdd
LibAdd: /opt/rrdtool-1.4.4/lib/perl/5.10.1/x86_64-linux-thread-multi/
#LibAdd: /opt/rrdtool-1.4.4/lib/perl/5.10.1
[root@ust-nagios1p ~]#