Page 1 of 2
Performance data not showing up
Posted: Tue Jul 21, 2015 3:25 pm
by itbilling
About 4 weeks ago, performance data has stopped showing up for any services. Host checks are showing performance data, but any service is not. One thing that catches my eye is that service-perfdata and perfdata.log do not exist.
Code: Select all
[root@localhost libexec]# which rrdtool
/usr/bin/rrdtool
Code: Select all
[root@localhost libexec]# ls /usr/local/nagios/var/rrd
argobr90.argointl.com atcwebapp1.geips-amer.com esxi-fl-12.geips-amer.com fgs1.geips-amer.com njladaptive.geips-amer.com sql1a.geips-amer.com
atbelfs1.geips-amer.com atcwebapp2.geips-amer.com esxi-gar-7.geips-amer.com flaerofs.geips-amer.com njlbiztalk.geips-amer.com sql1b.geips-amer.com
atcbackup.geips-amer.com atcwebapp3.geips-amer.com esxi-gar-8.geips-amer.com fs01.geips-amer.com njlcforms.geips-amer.com sql2a.geips-amer.com
Code: Select all
# PNP settings - bulk mode with NCPD
process_performance_data=1
# service performance data
service_perfdata_file=/usr/local/nagios/var/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
# host performance data
host_perfdata_file=/usr/local/nagios/var/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tHOSTOUTPUT::$HOSTOUTPUT$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file-bulk
Re: Performance data not showing up
Posted: Tue Jul 21, 2015 3:30 pm
by jdalrymple
Keep an eye on
Code: Select all
tail -f /usr/local/nagiosxi/var/perfdataproc.log
for about 2-3 minutes and see what you see.
Re: Performance data not showing up
Posted: Tue Jul 21, 2015 3:43 pm
by itbilling
jdalrymple wrote:Keep an eye on
Code: Select all
tail -f /usr/local/nagiosxi/var/perfdataproc.log
for about 2-3 minutes and see what you see.
Code: Select all
[root@localhost /]# tail -f /usr/local/nagiosxi/var/perfdataproc.log
Outbound data DISABLED Tue, 21 Jul 2015 16:38:02 -0400
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
tail: /usr/local/nagiosxi/var/perfdataproc.log: file truncated
Outbound data DISABLED Tue, 21 Jul 2015 16:39:02 -0400
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
/spool/xidpe/*': No such file or directory
tail: /usr/local/nagiosxi/var/perfdataproc.log: file truncated
Outbound data DISABLED Tue, 21 Jul 2015 16:42:02 -0400
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
Re: Performance data not showing up
Posted: Tue Jul 21, 2015 3:54 pm
by jdalrymple
Nothing haywire there, let's grep perfdata from nagios.cfg
Code: Select all
grep perfdata /usr/local/nagios/etc/nagios.cfg
Re: Performance data not showing up
Posted: Tue Jul 21, 2015 4:02 pm
by itbilling
jdalrymple wrote:Nothing haywire there, let's grep perfdata from nagios.cfg
Code: Select all
grep perfdata /usr/local/nagios/etc/nagios.cfg
Code: Select all
service_perfdata_file=/usr/local/nagios/var/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
host_perfdata_file=/usr/local/nagios/var/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tHOSTOUTPUT::$HOSTOUTPUT$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file-bulk
perfdata_timeout=5
service_perfdata_file=/usr/local/nagios/var/perfdata.log
service_perfdata_file_template=$LASTSERVICECHECK$||$HOSTNAME$||$SERVICEDESC$||$SERVICEOUTPUT$||$SERVICEPERFDATA$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=30
service_perfdata_file_processing_command=process-service-perfdata-for-nagiosgraph
Re: Performance data not showing up
Posted: Wed Jul 22, 2015 9:06 am
by jdalrymple
Where did this 2nd set of service perfdata defs come from?
itbilling wrote:Code: Select all
service_perfdata_file=/usr/local/nagios/var/perfdata.log
service_perfdata_file_template=$LASTSERVICECHECK$||$HOSTNAME$||$SERVICEDESC$||$SERVICEOUTPUT$||$SERVICEPERFDATA$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=30
service_perfdata_file_processing_command=process-service-perfdata-for-nagiosgraph
I suspect they're the problem. I would rid myself of them if that were my machine. From a stock 2014r2.7 install:
Code: Select all
[jdalrymple@localhost libexec]$ grep perfdata /usr/local/nagios/etc/nagios.cfg | grep service
service_perfdata_file=/usr/local/nagios/var/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
Re: Performance data not showing up
Posted: Wed Jul 22, 2015 2:21 pm
by itbilling
jdalrymple wrote:Where did this 2nd set of service perfdata defs come from?
itbilling wrote:Code: Select all
service_perfdata_file=/usr/local/nagios/var/perfdata.log
service_perfdata_file_template=$LASTSERVICECHECK$||$HOSTNAME$||$SERVICEDESC$||$SERVICEOUTPUT$||$SERVICEPERFDATA$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=30
service_perfdata_file_processing_command=process-service-perfdata-for-nagiosgraph
It came from a third party application called NagioGraph, they were both working together at one point, but since I'm having issues I'll remove that block for now.
Re: Performance data not showing up
Posted: Wed Jul 22, 2015 2:30 pm
by lmiltchev
It came from a third party application called NagioGraph, they were both working together at one point, but since I'm having issues I'll remove that block for now.
Let us know if this fixed your issue. I will keep this thread open for a while.
Re: Performance data not showing up
Posted: Wed Jul 22, 2015 5:45 pm
by itbilling
Yes, I am now receiving performance data again, is there a way to get the perfdata that didn't process already? I see a lot of RRD files, but I'm not sure how I would add it to Nagios.
lmiltchev wrote:It came from a third party application called NagioGraph, they were both working together at one point, but since I'm having issues I'll remove that block for now.
Let us know if this fixed your issue. I will keep this thread open for a while.
Re: Performance data not showing up
Posted: Thu Jul 23, 2015 8:35 am
by tgriep
The RRD files is where the data is stored after they are processed but with the issues you were having, the data maybe lost.
Lets check to see if the data is being processed now by running the following and post back the results.
Code: Select all
ls /usr/local/nagios/var/spool/xidpe | wc -l
ls /usr/local/nagios/var/spool/perfdata | wc -l
ls /usr/local/nagios/var/spool/checkresults | wc -l