Page 2 of 5

Re: host & service perfdata increasing alarmingly

Posted: Thu Feb 09, 2017 6:57 pm
by pmithil7
I have this in my commands.cfg:
'''
# SAMPLE PERFORMANCE DATA COMMANDS
#
# These are sample performance data commands that can be used to send performance
# data output to two text files (one for hosts, another for services). If you
# plan on simply writing performance data out to a file, consider using the
# host_perfdata_file and service_perfdata_file options in the main config file.
#
################################################################################
# 'process-host-perfdata' command definition
define command{
command_name process-host-perfdata
command_line /usr/bin/printf "%b" "$LASTHOSTCHECK$\t$HOSTNAME$\t$HOSTSTATE$\t$HOSTATTEMPT$\t$HOSTSTATETYPE$\t$HOSTEXECUTIONTIME$\t$HOSTOUTPUT$\t$HOSTPERFDATA$\n" >> /usr/local/nagios/var/host-perfdata.out
}

# 'process-service-perfdata' command definition
define command{
command_name process-service-perfdata
command_line /usr/bin/printf "%b" "$LASTSERVICECHECK$\t$HOSTNAME$\t$SERVICEDESC$\t$SERVICESTATE$\t$SERVICEATTEMPT$\t$SERVICESTATETYPE$\t$SERVICEEXECUTIONTIME$\t$SERVICELATENCY$\t$SERVICEOUTPUT$\t$SERVICEPERFDATA$\n" >> /usr/local/nagios/var/service-perfdata.out
}

define command{
command_name process-service-perfdata-file
command_line /bin/mv /usr/local/pnp4nagios/var/service-perfdata /usr/local/pnp4nagios/var/spool/service-perfdata.$TIMET$
}

define command{
command_name process-host-perfdata-file
command_line /bin/mv /usr/local/pnp4nagios/var/host-perfdata /usr/local/pnp4nagios/var/spool/host-perfdata.$TIMET$
}
'''

Also, an interesting find that might help:
'''
[root@nagios-eqx-01 local]# grep -R -i 'process-host-perfdata-file' .
./nagios/var/nagios.log:[1486632671] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486632707] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486663091] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486663173] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486663610] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486664528] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486667224] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486668876] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486668923] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
./nagios/var/nagios.log:[1486676061] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
grep: ./nagios/var/rw/nagios.qh: No such device or address
^C
[root@nagios-eqx-01 local]# grep -R -i 'process-service-perfdata-file' .
./nagios/var/nagios.log:[1486632671] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486632707] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486663091] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486663173] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486663610] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486664528] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486667224] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486668876] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486668923] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
./nagios/var/nagios.log:[1486676061] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
grep: ./nagios/var/rw/nagios.qh: No such device or address
'''

Re: host & service perfdata increasing alarmingly

Posted: Fri Feb 10, 2017 1:17 pm
by pmithil7
Any updates on this?

Re: host & service perfdata increasing alarmingly

Posted: Fri Feb 10, 2017 1:32 pm
by rkennedy
pmithil7 wrote:Any updates on this?
Please do not bump your thread. We work through a queue, and you only push yourself to the bottom once again.
./nagios/var/nagios.log:[1486676061] Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
grep: ./nagios/var/rw/nagios.qh: No such device or address
^C
[root@nagios-eqx-01 local]# grep -R -i 'process-service-perfdata-file' .
./nagios/var/nagios.log:[1486632671] Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
Can you PM over myself and @dwhitfield an entire copy of your /usr/local/nagios/etc directory? The error mentions that something isn't processing properly with your performance data so it'll help to see everything at play.

Re: host & service perfdata increasing alarmingly

Posted: Fri Feb 10, 2017 2:57 pm
by rkennedy
I was able to find this thread - https://support.nagios.com/forum/viewto ... =6&t=41487 - it appears this issue may have been going on for a bit here. From what I can tell in your file -

Code: Select all

host_perfdata_file_processing_command=process-host-perfdata-file
service_perfdata_file_processing_command=process-service-perfdata-file
Is executing this -

Code: Select all

define command{
       command_name    process-service-perfdata-file
       command_line    /bin/mv /usr/local/pnp4nagios/var/service-perfdata /usr/local/pnp4nagios/var/spool/service-perfdata.$TIMET$
}

define command{
       command_name    process-host-perfdata-file
       command_line    /bin/mv /usr/local/pnp4nagios/var/host-perfdata /usr/local/pnp4nagios/var/spool/host-perfdata.$TIMET$
}
Mine is a bit different over here -

Code: Select all

host_perfdata_file_processing_command=process-host-perfdata-file-bulk

define command {
       command_name                             process-host-perfdata-file-bulk
       command_line                             /bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.host
}
From there, it would really depend on what's processing the perf data to pick it up. What other addons are you using in conjunction with Nagios? I saw in the previous post mentions of livestatus / pnp4nagios. What versions?

Re: host & service perfdata increasing alarmingly

Posted: Fri Feb 10, 2017 2:58 pm
by pmithil7
Sorry for being a little impatient. I have sent a PM message as requested with the directory contents.

From the logs i feel like changes need to be made in the commands.cfg for the definitions of 'process-service-perfdata-file' and 'process-host-perfdata-file'. Let me know your thoughts on it.

Thanks,
Mithil

Re: host & service perfdata increasing alarmingly

Posted: Fri Feb 10, 2017 3:16 pm
by rkennedy
Can you address the question above?
rkennedy wrote:I was able to find this thread - https://support.nagios.com/forum/viewto ... =6&t=41487 - it appears this issue may have been going on for a bit here. From what I can tell in your file -
From there, it would really depend on what's processing the perf data to pick it up. What other addons are you using in conjunction with Nagios? I saw in the previous post mentions of livestatus / pnp4nagios. What versions?
It's hard to say what needs to be changed since we did not configure all of this. It might be revisiting what @ssax mentioned - https://support.nagios.com/forum/viewto ... 87#p205430

Re: host & service perfdata increasing alarmingly

Posted: Fri Feb 10, 2017 3:32 pm
by tgriep
If has to be one of the pnp4nagios config files that are putting the performance data in the wrong folder.
Take a look at the files in the following folder and see if they are correct.

Code: Select all

/usr/local/pnp4nagios/etc
Or reinstall / remove pnp4nagios to fix the issue.

Re: host & service perfdata increasing alarmingly

Posted: Fri Feb 10, 2017 5:24 pm
by pmithil7
From the thread that you mentioned, my problem wasn't solved and hence i temporarily stopped processing performance data by putting process_performance_data = 0. This stopped data from accumulating and it after a month it again started accumulating data in those two files inspite of process_performance_data being 0. Hence, this grew up my concern about how data is still being gathered.

I have pnp4nagios of :

PKG_VERSION="0.6.25"
PKG_NAME="pnp4nagios"

Other then that i have nrpe, check_mk and wato as plugins on nagios.

I also checked /usr/local/pnp4nagios/etc and the configuration files seem to be ideal.

Any suggestion? Should i uninstall and re-install pnp4nagios again?

Re: host & service perfdata increasing alarmingly

Posted: Mon Feb 13, 2017 10:51 am
by tgriep
Yes, try uninstalling pnp4nagios and reinstalling to see if that fixes the issue.
If you have any other brokers enabled in the nagios.cfg file, try commenting them out as well to see if that stops the collecting of the performance data.

Re: host & service perfdata increasing alarmingly

Posted: Mon Feb 13, 2017 12:53 pm
by pmithil7
Thank you for your help. Any document or instructions on how to safely uninstall pnp4nagios without affecting nagios and other plugins? Any specifics that i should look forward to that won't crash the server?

I'm just being cautious here. Thanks