Post XI5 Upgrade Performance Charts Stopped Working

[chriscamm]

Since upgrading to XI 5 some of my performance charts stopped updating:

So I removed the RRD file from one of them and this meant it disappeared completely. Although the RRD file returned on the next check of the service it failed to update again and never appeared back in the performance chart window.

So I ran the RRD repair script to see if that would help. (https://support.nagios.com/wiki/index.p ... 14_Upgrade)

Now all my graphs have disappeared as per the attachment and none are coming back. I do not want to lose any historical data and can snapshot back to before the upgrade to XI 5 if needed and only lose 24 hours of data which is far better than losing 3 years.

I am running a VM with Centos6.7 and the mySql is on a separate server.

Here is a copy of the /usr/local/nagios/var/perfdata.log

Code: Select all

[root@xxxxxxxxx ~]# tail -25 /usr/local/nagios/var/perfdata.log 
2015-10-09 09:20:25 [9650] [2] Template is check_xi_service_wmiplus_xx.php
2015-10-09 09:20:25 [9650] [2] data2rrd called
2015-10-09 09:20:25 [9650] [2] RRDs::update --daemon=unix:/var/rrdtool/rrdcached/rrdcached.sock /usr/local/nagios/share/perfdata/xxxxx.xxxx.xx.xx/Exchange_Transport_Log_Search_Service.rrd 1444378819:1:1:0:0
2015-10-09 09:20:25 [9650] [1] RRDs::update ERROR unknown option '--daemon=unix:/var/rrdtool/rrdcached/rrdcached.sock'
2015-10-09 09:20:25 [9650] [2] Processing Line 136
2015-10-09 09:20:25 [9650] [2] Datatype set to 'SERVICEPERFDATA' 
2015-10-09 09:20:25 [9650] [1] Found Performance Data for xxxxxxxxxxxxx / Active_Directory_Domain_Service ('Total Service Count'=1; 'Service Count OK State'=1; 'Service Count Problem State'=0; 'Excluded Service Count'=0;) 
2015-10-09 09:20:25 [9650] [2] No Custom Template found for check_xi_service_wmiplus_xxxx_check_win_service (/usr/local/nagios/etc/pnp/check_commands/check_xi_service_wmiplus_xxxx_check_win_service.cfg) 
2015-10-09 09:20:25 [9650] [2] RRD Datatype is GAUGE
2015-10-09 09:20:25 [9650] [2] Template is check_xi_service_wmiplus_xxxx_check_win_service.php
2015-10-09 09:20:25 [9650] [2] No Custom Template found for check_xi_service_wmiplus_xxxx_check_win_service (/usr/local/nagios/etc/pnp/check_commands/check_xi_service_wmiplus_xxxx_check_win_service.cfg) 
2015-10-09 09:20:25 [9650] [2] RRD Datatype is GAUGE
2015-10-09 09:20:25 [9650] [2] Template is check_xi_service_wmiplus_xxxx_check_win_service.php
2015-10-09 09:20:25 [9650] [2] No Custom Template found for check_xi_service_wmiplus_xxxx_check_win_service (/usr/local/nagios/etc/pnp/check_commands/check_xi_service_wmiplus_xxxx_check_win_service.cfg) 
2015-10-09 09:20:25 [9650] [2] RRD Datatype is GAUGE
2015-10-09 09:20:25 [9650] [2] Template is check_xi_service_wmiplus_xxxx_check_win_service.php
2015-10-09 09:20:25 [9650] [2] No Custom Template found for check_xi_service_wmiplus_xxxx_check_win_service (/usr/local/nagios/etc/pnp/check_commands/check_xi_service_wmiplus_xxxx_check_win_service.cfg) 
2015-10-09 09:20:25 [9650] [2] RRD Datatype is GAUGE
2015-10-09 09:20:25 [9650] [2] Template is check_xi_service_wmiplus_xxxx_check_win_service.php
2015-10-09 09:20:25 [9650] [2] data2rrd called
2015-10-09 09:20:25 [9650] [2] RRDs::update --daemon=unix:/var/rrdtool/rrdcached/rrdcached.sock /usr/local/nagios/share/perfdata/xxxxxxxxxx/Active_Directory_Domain_Service.rrd 1444378819:1:1:0:0
2015-10-09 09:20:25 [9650] [1] RRDs::update ERROR unknown option '--daemon=unix:/var/rrdtool/rrdcached/rrdcached.sock'
2015-10-09 09:20:25 [9650] [1] 136 Lines processed
2015-10-09 09:20:25 [9650] [1] /usr/local/nagios/var/spool/perfdata//1444378819.perfdata.service-PID-9650 deleted
2015-10-09 09:20:25 [9650] [1] PNP exiting (runtime 0.751278s) ...
[root@xxxxxx ~]# 

tail -25 /var/local/nagios/var/npcd.log

Code: Select all

[root@xxxxxx ~]# tail -25 /usr/local/nagios/var/npcd.log
[10-09-2015 09:23:55] NPCD: DEBUG: load 4.130000/60.000000
[10-09-2015 09:23:55] NPCD: ThreadCounter 0/10 File is 1444379014.perfdata.service
[10-09-2015 09:23:55] NPCD: Regular File: 1444379014.perfdata.service
[10-09-2015 09:23:55] NPCD: A thread was started on thread_counter = 0
[10-09-2015 09:23:55] NPCD: Processing file 1444379014.perfdata.service with ID 139820664858368 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1444379014.perfdata.service
[10-09-2015 09:23:55] NPCD: DEBUG: load 4.130000/60.000000
[10-09-2015 09:23:55] NPCD: Processing file '1444379014.perfdata.service'
[10-09-2015 09:23:55] NPCD: ThreadCounter 1/10 File is 1444379015.perfdata.host
[10-09-2015 09:23:55] NPCD: Regular File: 1444379015.perfdata.host
[10-09-2015 09:23:55] NPCD: A thread was started on thread_counter = 1
[10-09-2015 09:23:55] NPCD: Processing file 1444379015.perfdata.host with ID 139820654368512 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1444379015.perfdata.host
[10-09-2015 09:23:55] NPCD: Processing file '1444379015.perfdata.host'
[10-09-2015 09:23:55] NPCD: DEBUG: load 4.130000/60.000000
[10-09-2015 09:23:55] NPCD: ThreadCounter 2/10 File is 1444379029.perfdata.service
[10-09-2015 09:23:55] NPCD: Regular File: 1444379029.perfdata.service
[10-09-2015 09:23:55] NPCD: A thread was started on thread_counter = 2
[10-09-2015 09:23:55] NPCD: Processing file 1444379029.perfdata.service with ID 139820643878656 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1444379029.perfdata.service
[10-09-2015 09:23:55] NPCD: DEBUG: load 4.130000/60.000000
[10-09-2015 09:23:55] NPCD: Processing file '1444379029.perfdata.service'
[10-09-2015 09:23:55] NPCD: ThreadCounter 3/10 File is 1444379030.perfdata.host
[10-09-2015 09:23:55] NPCD: Regular File: 1444379030.perfdata.host
[10-09-2015 09:23:55] NPCD: A thread was started on thread_counter = 3
[10-09-2015 09:23:55] NPCD: Processing file 1444379030.perfdata.host with ID 139820633388800 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1444379030.perfdata.host
[10-09-2015 09:23:55] NPCD: Have to wait: Filecounter = 4 - thread_counter = 4
[10-09-2015 09:23:55] NPCD: Processing file '1444379030.perfdata.host'
[root@xxxxxx ~]# 

tail -25 /var/log/httpd/error_log

Code: Select all

[root@qualngs ~]# tail -25 /var/log/httpd/error_log
[Fri Oct 09 03:20:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 03:35:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 03:50:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 04:05:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 04:20:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 04:35:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 04:50:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 05:05:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 05:20:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 05:35:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 05:50:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 06:05:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 06:20:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 06:35:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 06:50:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 07:05:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 07:20:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 07:35:01 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 07:50:00 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 08:05:00 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 08:20:00 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 08:35:00 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 08:50:00 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 09:05:00 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[Fri Oct 09 09:20:00 2015] [error] [client xxxxx] PHP Notice:  Undefined variable: api_version in /usr/local/nrdp/server/plugins/nrdsconfigupdate/nrdsconfigupdate.inc.php on line 269
[root@qualngs ~]# 

If you need anything else please let me know

[tgriep]

Can you post your process_perfdata.cfg file so we can review it?
This is where it is located.

Code: Select all

/usr/local/nagios/etc/pnp/process_perfdata.cfg

[chriscamm]

Code: Select all

#
# Config File for process_perfdata.pl
#
# $Id: process_perfdata.cfg-sample.in 520 2008-09-16 12:50:10Z pitchfork $
#
# process_perfdata.pl Timout
#
TIMEOUT = 120
#
# Use RRDs Perl Module
#
USE_RRDs = 1
#
#
#
RRDPATH = /usr/local/nagios/share/perfdata
#
#
#
RRDTOOL = /usr/bin/rrdtool
#
#
#
CFG_DIR = /usr/local/nagios/etc/pnp
#
#
#
RRD_HEARTBEAT = 8460
#
#
#
RRA_CFG = /usr/local/nagios/etc/pnp/rra.cfg
#
#
#
RRA_STEP = 60
#
#
#
LOG_FILE = /usr/local/nagios/var/perfdata.log
#
# Loglevel 0=silent 1=normal 2=debug
#
LOG_LEVEL = 2
#
# XML encoding
# The supported encodings are ISO-8859-1, UTF-8 and US-ASCII.
# http://www.php.net/xml-parser-create
XML_ENC = UTF-8
#
# EXPERIMENTAL rrdcached Support
# Use only with rrdtool svn revision 1511+
#
# RRD_DAEMON_OPTS = unix:/tmp/rrdcached.sock

RRD_DAEMON_OPTS = unix:/var/rrdtool/rrdcached/rrdcached.sock

Let me know if you need anything else.

Thanks

Chris

[tmcdonald]

Please run the following and post the results:

Code: Select all

service npcd status
ls -l /usr/local/nagios/var/spool/xidpe
ls -l /usr/local/nagios/var/spool/perfdata

[chriscamm]

Please see the output

Code: Select all

service npcd status
NPCD running (pid 8394).
ls -l /usr/local/nagios/var/spool/xidpe/
total 0
ls -l /usr/local/nagios/var/spool/perfdata
total 0 

[tgriep]

Try and restart the rrdcached daemon to see if that fixes it for you. Run this to do that.

Code: Select all

service rrdcached restart

[chriscamm]

Thanks for the suggestion but this has no affect

Many thanks

Chris

[tmcdonald]

What XI version were you running before the upgrade?

[Box293]

I suspect rrdtool is no longer at version 1.4.4 which is required for rrdcached.

What is the output of:

Code: Select all

rrdtool

It's it's less than 1.4.4 then follow this document:
http://assets.nagios.com/downloads/nagi ... ios_XI.pdf

If after running this rrdtool is still not 1.4.4 then for some reason rrdtool-1.4.4.tar.gz is not being downloaded/upgraded.

Line 114 in the xi-rrdcached.sh script is downloading this file:
wget http://oss.oetiker.ch/rrdtool/pub/rrdtool-1.4.4.tar.gz

If for some reason the server cannot download this file, download it and transfer it to the XI server into the same directory as the xi-rrdcached.sh script. Comment out line 114 and then re-run the script and it should fix the problem.

[chriscamm]

Hi,

I was at the latest version of 2014 prior to the upgrade. I always try to upgrade within a week or two of the latest patches.

my rrdtool is version 1.4.4

Thanks

Chris