host & service perfdata increasing alarmingly

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
pmithil7
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Post by pmithil7 »

I have this in my commands.cfg:
'''
# SAMPLE PERFORMANCE DATA COMMANDS
#
# These are sample performance data commands that can be used to send performance
# data output to two text files (one for hosts, another for services). If you
# plan on simply writing performance data out to a file, consider using the
# host_perfdata_file and service_perfdata_file options in the main config file.
#
################################################################################
# 'process-host-perfdata' command definition
define command{
command_name process-host-perfdata
command_line /usr/bin/printf "%b" "$LASTHOSTCHECK$\t$HOSTNAME$\t$HOSTSTATE$\t$HOSTATTEMPT$\t$HOSTSTATETYPE$\t$HOSTEXECUTIONTIME$\t$HOSTOUTPUT$\t$HOSTPERFDATA$\n" >> /usr/local/nagios/var/host-perfdata.out
}

# 'process-service-perfdata' command definition
define command{
command_name process-service-perfdata
command_line /usr/bin/printf "%b" "$LASTSERVICECHECK$\t$HOSTNAME$\t$SERVICEDESC$\t$SERVICESTATE$\t$SERVICEATTEMPT$\t$SERVICESTATETYPE$\t$SERVICEEXECUTIONTIME$\t$SERVICELATENCY$\t$SERVICEOUTPUT$\t$SERVICEPERFDATA$\n" >> /usr/local/nagios/var/service-perfdata.out
}

define command{
command_name process-service-perfdata-file
command_line /bin/mv /usr/local/pnp4nagios/var/service-perfdata /usr/local/pnp4nagios/var/spool/service-perfdata.$TIMET$
}

define command{
command_name process-host-perfdata-file
command_line /bin/mv /usr/local/pnp4nagios/var/host-perfdata /usr/local/pnp4nagios/var/spool/host-perfdata.$TIMET$
}
'''

Also, an interesting find that might help:
'''
[root@nagios-eqx-01 local]# grep -R -i 'process-host-perfdata-file' .
./nagios/var/nagios.log:[1486632671] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486632707] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486663091] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486663173] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486663610] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486664528] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486667224] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486668876] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486668923] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486676061] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
grep: ./nagios/var/rw/nagios.qh: No such device or address
^C
[root@nagios-eqx-01 local]# grep -R -i 'process-service-perfdata-file' .
./nagios/var/nagios.log:[1486632671] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486632707] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486663091] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486663173] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486663610] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486664528] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486667224] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486668876] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486668923] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486676061] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
grep: ./nagios/var/rw/nagios.qh: No such device or address
'''
pmithil7
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Post by pmithil7 »

Any updates on this?
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: host & service perfdata increasing alarmingly

Post by rkennedy »

pmithil7 wrote:Any updates on this?
Please do not bump your thread. We work through a queue, and you only push yourself to the bottom once again.
./nagios/var/nagios.log:[1486676061] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
grep: ./nagios/var/rw/nagios.qh: No such device or address
^C
[root@nagios-eqx-01 local]# grep -R -i 'process-service-perfdata-file' .
./nagios/var/nagios.log:[1486632671] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
Can you PM over myself and @dwhitfield an entire copy of your /usr/local/nagios/etc directory? The error mentions that something isn't processing properly with your performance data so it'll help to see everything at play.
Former Nagios Employee
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: host & service perfdata increasing alarmingly

Post by rkennedy »

I was able to find this thread - https://support.nagios.com/forum/viewto ... =6&t=41487 - it appears this issue may have been going on for a bit here. From what I can tell in your file -

Code: Select all

host_perfdata_file_processing_command=process-host-perfdata-file
service_perfdata_file_processing_command=process-service-perfdata-file
Is executing this -

Code: Select all

define command{
       command_name    process-service-perfdata-file
       command_line    /bin/mv /usr/local/pnp4nagios/var/service-perfdata /usr/local/pnp4nagios/var/spool/service-perfdata.$TIMET$
}

define command{
       command_name    process-host-perfdata-file
       command_line    /bin/mv /usr/local/pnp4nagios/var/host-perfdata /usr/local/pnp4nagios/var/spool/host-perfdata.$TIMET$
}
Mine is a bit different over here -

Code: Select all

host_perfdata_file_processing_command=process-host-perfdata-file-bulk

define command {
       command_name                             process-host-perfdata-file-bulk
       command_line                             /bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.host
}
From there, it would really depend on what's processing the perf data to pick it up. What other addons are you using in conjunction with Nagios? I saw in the previous post mentions of livestatus / pnp4nagios. What versions?
Former Nagios Employee
pmithil7
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Post by pmithil7 »

Sorry for being a little impatient. I have sent a PM message as requested with the directory contents.

From the logs i feel like changes need to be made in the commands.cfg for the definitions of 'process-service-perfdata-file' and 'process-host-perfdata-file'. Let me know your thoughts on it.

Thanks,
Mithil
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: host & service perfdata increasing alarmingly

Post by rkennedy »

Can you address the question above?
rkennedy wrote:I was able to find this thread - https://support.nagios.com/forum/viewto ... =6&t=41487 - it appears this issue may have been going on for a bit here. From what I can tell in your file -
From there, it would really depend on what's processing the perf data to pick it up. What other addons are you using in conjunction with Nagios? I saw in the previous post mentions of livestatus / pnp4nagios. What versions?
It's hard to say what needs to be changed since we did not configure all of this. It might be revisiting what @ssax mentioned - https://support.nagios.com/forum/viewto ... 87#p205430
Former Nagios Employee
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: host & service perfdata increasing alarmingly

Post by tgriep »

If has to be one of the pnp4nagios config files that are putting the performance data in the wrong folder.
Take a look at the files in the following folder and see if they are correct.

Code: Select all

/usr/local/pnp4nagios/etc
Or reinstall / remove pnp4nagios to fix the issue.
Be sure to check out our Knowledgebase for helpful articles and solutions!
pmithil7
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Post by pmithil7 »

From the thread that you mentioned, my problem wasn't solved and hence i temporarily stopped processing performance data by putting process_performance_data = 0. This stopped data from accumulating and it after a month it again started accumulating data in those two files inspite of process_performance_data being 0. Hence, this grew up my concern about how data is still being gathered.

I have pnp4nagios of :

PKG_VERSION="0.6.25"
PKG_NAME="pnp4nagios"

Other then that i have nrpe, check_mk and wato as plugins on nagios.

I also checked /usr/local/pnp4nagios/etc and the configuration files seem to be ideal.

Any suggestion? Should i uninstall and re-install pnp4nagios again?
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: host & service perfdata increasing alarmingly

Post by tgriep »

Yes, try uninstalling pnp4nagios and reinstalling to see if that fixes the issue.
If you have any other brokers enabled in the nagios.cfg file, try commenting them out as well to see if that stops the collecting of the performance data.
Be sure to check out our Knowledgebase for helpful articles and solutions!
pmithil7
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Post by pmithil7 »

Thank you for your help. Any document or instructions on how to safely uninstall pnp4nagios without affecting nagios and other plugins? Any specifics that i should look forward to that won't crash the server?

I'm just being cautious here. Thanks
Locked