Performance data not showing up

[itbilling]

About 4 weeks ago, performance data has stopped showing up for any services. Host checks are showing performance data, but any service is not. One thing that catches my eye is that service-perfdata and perfdata.log do not exist.

Code: Select all

[root@localhost libexec]# which rrdtool
/usr/bin/rrdtool

Code: Select all

[root@localhost libexec]# ls /usr/local/nagios/var/rrd
argobr90.argointl.com       atcwebapp1.geips-amer.com   esxi-fl-12.geips-amer.com   fgs1.geips-amer.com           njladaptive.geips-amer.com    sql1a.geips-amer.com
atbelfs1.geips-amer.com     atcwebapp2.geips-amer.com   esxi-gar-7.geips-amer.com   flaerofs.geips-amer.com       njlbiztalk.geips-amer.com     sql1b.geips-amer.com
atcbackup.geips-amer.com    atcwebapp3.geips-amer.com   esxi-gar-8.geips-amer.com   fs01.geips-amer.com           njlcforms.geips-amer.com      sql2a.geips-amer.com

Code: Select all

# PNP settings - bulk mode with NCPD
process_performance_data=1
# service performance data
service_perfdata_file=/usr/local/nagios/var/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
# host performance data
host_perfdata_file=/usr/local/nagios/var/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tHOSTOUTPUT::$HOSTOUTPUT$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file-bulk

[jdalrymple]

Keep an eye on

Code: Select all

tail -f /usr/local/nagiosxi/var/perfdataproc.log

for about 2-3 minutes and see what you see.

[itbilling]

Keep an eye on

Code: Select all

tail -f /usr/local/nagiosxi/var/perfdataproc.log

for about 2-3 minutes and see what you see.

Code: Select all

[root@localhost /]# tail -f /usr/local/nagiosxi/var/perfdataproc.log
Outbound data DISABLED Tue, 21 Jul 2015 16:38:02 -0400
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
tail: /usr/local/nagiosxi/var/perfdataproc.log: file truncated
Outbound data DISABLED Tue, 21 Jul 2015 16:39:02 -0400
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
/spool/xidpe/*': No such file or directory
tail: /usr/local/nagiosxi/var/perfdataproc.log: file truncated
Outbound data DISABLED Tue, 21 Jul 2015 16:42:02 -0400
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory

[jdalrymple]

Nothing haywire there, let's grep perfdata from nagios.cfg

Code: Select all

grep perfdata /usr/local/nagios/etc/nagios.cfg

[itbilling]

Nothing haywire there, let's grep perfdata from nagios.cfg

Code: Select all

grep perfdata /usr/local/nagios/etc/nagios.cfg

Code: Select all

service_perfdata_file=/usr/local/nagios/var/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
host_perfdata_file=/usr/local/nagios/var/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tHOSTOUTPUT::$HOSTOUTPUT$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file-bulk
perfdata_timeout=5
service_perfdata_file=/usr/local/nagios/var/perfdata.log
service_perfdata_file_template=$LASTSERVICECHECK$||$HOSTNAME$||$SERVICEDESC$||$SERVICEOUTPUT$||$SERVICEPERFDATA$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=30
service_perfdata_file_processing_command=process-service-perfdata-for-nagiosgraph

[jdalrymple]

Where did this 2nd set of service perfdata defs come from?

Code: Select all

service_perfdata_file=/usr/local/nagios/var/perfdata.log
service_perfdata_file_template=$LASTSERVICECHECK$||$HOSTNAME$||$SERVICEDESC$||$SERVICEOUTPUT$||$SERVICEPERFDATA$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=30
service_perfdata_file_processing_command=process-service-perfdata-for-nagiosgraph

I suspect they're the problem. I would rid myself of them if that were my machine. From a stock 2014r2.7 install:

Code: Select all

[jdalrymple@localhost libexec]$ grep perfdata /usr/local/nagios/etc/nagios.cfg  | grep service
service_perfdata_file=/usr/local/nagios/var/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file-bulk

[itbilling]

Where did this 2nd set of service perfdata defs come from?

Code: Select all

service_perfdata_file=/usr/local/nagios/var/perfdata.log
service_perfdata_file_template=$LASTSERVICECHECK$||$HOSTNAME$||$SERVICEDESC$||$SERVICEOUTPUT$||$SERVICEPERFDATA$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=30
service_perfdata_file_processing_command=process-service-perfdata-for-nagiosgraph

It came from a third party application called NagioGraph, they were both working together at one point, but since I'm having issues I'll remove that block for now.

[lmiltchev]

It came from a third party application called NagioGraph, they were both working together at one point, but since I'm having issues I'll remove that block for now.

Let us know if this fixed your issue. I will keep this thread open for a while.

[itbilling]

Yes, I am now receiving performance data again, is there a way to get the perfdata that didn't process already? I see a lot of RRD files, but I'm not sure how I would add it to Nagios.

It came from a third party application called NagioGraph, they were both working together at one point, but since I'm having issues I'll remove that block for now.

Let us know if this fixed your issue. I will keep this thread open for a while.

[tgriep]

The RRD files is where the data is stored after they are processed but with the issues you were having, the data maybe lost.
Lets check to see if the data is being processed now by running the following and post back the results.

Code: Select all

ls /usr/local/nagios/var/spool/xidpe | wc -l
ls /usr/local/nagios/var/spool/perfdata | wc -l
ls /usr/local/nagios/var/spool/checkresults | wc -l