Page 1 of 1
host-perfdata continuously grows
Posted: Wed Apr 30, 2014 7:20 am
by pnewlon
I have a 750MB ramdisk that keeps filling because host-perfdata never shrinks. It took just short of seven days to fill the ramdisk. What do I have configured wrong that is causing this to keep growing?
tmpfs 750M 750M 160K 100% /var/nagiosramdisk
# cd /var/nagiosramdisk/
# ls -lt
-rw-rw-r-- 1 nagios nagios 0 Apr 30 07:48 service-perfdata
drwxrwxr-x 2 nagios nagios 360 Apr 30 07:48 tmp
-rw-rw-r-- 1 nagios nagios 780091489 Apr 29 01:04 host-perfdata
-rw-r--r-- 1 nagios nagios 4622171 Apr 22 13:38 objects.cache
drwxrwxr-x 5 nagios nagios 100 Apr 22 13:37 spool
First record: Tue, 22 Apr 2014 17:38:13 GMT
DATATYPE::HOSTPERFDATA TIMET::1398188293 HOSTNAME::00697_HTTP HOSTPERFDATA::time=0.308773s;;;0.000000 size=120B;;;0 HOSTCHECKCOMMAND::check_xi_service_http!-p 16992 -t 4!!!!!!! HOSTSTATE::UP HOSTSTATETYPE::HARD HOSTOUTPUT::HTTP OK - HTTP/1.1 303 See Other - 0.309 second response time
Last record: Tue, 29 Apr 2014 05:04:06 GMT
DATATYPE::HOSTPERFDATA TIMET::1398747846 HOSTNAME::08232_IB04-MP HOSTPERFDATA::rta=90.132004ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0 HOSTCHECKCOMMAND::check_ping!3000,80%!5000,100%!!!!!! HOSTSTATE::UP HOSTSTATETYPE::HARD HOSTOUTPUT::PING OK - Packet loss = 0%, RTA = 90.13 ms
Re: host-perfdata continuously grows
Posted: Wed Apr 30, 2014 10:49 am
by scottwilkerson
I have a feeling you might have missed the last step on page 3 of
http://assets.nagios.com/downloads/nagi ... giosXI.pdf
what do the following commands look like in the Core Config Manager
process-host-perfdata-file-bulk
process-service-perfdata-file-bulk
also what is the output of the following
Code: Select all
cat /usr/local/nagios/etc/nagios.cfg|grep perfdata_file_processing_command
Re: host-perfdata continuously grows
Posted: Wed Apr 30, 2014 1:51 pm
by pnewlon
Of course the last thing I check is the "what do the following commands look like in the Core Config Manager".
[root@lpnagv03 etc]# cat /usr/local/nagios/etc/nagios.cfg|grep perfdata_file_processing_command
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
host_perfdata_file_processing_command=process-host-perfdata-file-bulk
# host performance data
#host_perfdata_file=/usr/local/nagios/var/host-perfdata
host_perfdata_file=/var/nagiosramdisk/host-perfdata
# service performance data
#service_perfdata_file=/usr/local/nagios/var/service-perfdata
service_perfdata_file=/var/nagiosramdisk/service-perfdata
[root@lpnagv03 nagiosramdisk]# cat /usr/local/nagios/etc/nagios.cfg|grep ramd
service_perfdata_file=/var/nagiosramdisk/service-perfdata
host_perfdata_file=/var/nagiosramdisk/host-perfdata
check_result_path=/var/nagiosramdisk/spool/checkresults
object_cache_file=/var/nagiosramdisk/objects.cache
status_file=/var/nagiosramdisk/status.dat
temp_path=/var/nagiosramdisk/tmp
Interestingly, service perfdata is processing...
[root@lpnagv03 nagiosramdisk]# ls -lt
total 21988
drwxrwxr-x 2 nagios nagios 15500 Apr 30 14:34 tmp
-rw-r--r-- 1 nagios nagios 6702401 Apr 30 14:34 status.dat
-rw-rw-r-- 1 nagios nagios 22196 Apr 30 14:34 service-perfdata
-rw-rw-r-- 1 nagios nagios 11095999 Apr 30 14:34 host-perfdata
-rw-r--r-- 1 nagios nagios 4622171 Apr 30 08:37 objects.cache
drwxrwxr-x 5 nagios nagios 100 Apr 22 13:37 spool
Re: host-perfdata continuously grows
Posted: Wed Apr 30, 2014 2:07 pm
by abrist
What do you have set for the perfdata processing intervals?
Code: Select all
grep perfdata_file_processing_interval /usr/local/nagios/etc/nagios.cfg
Re: host-perfdata continuously grows
Posted: Wed Apr 30, 2014 2:11 pm
by pnewlon
[root@lpnagv03 nagiosramdisk]# grep perfdata_file_processing_interval /usr/local/nagios/etc/nagios.cfg
service_perfdata_file_processing_interval=15
#host_perfdata_file_processing_interval=15
host_perfdata_file_processing_interval=5
Re: host-perfdata continuously grows
Posted: Wed Apr 30, 2014 2:16 pm
by abrist
Any errors in the perfdata/npcd logs?
Code: Select all
tail -50 /usr/local/nagios/var/perfdata.log
tail -50 /usr/local/nagios/var/npcd.log
pnewlon wrote:host_perfdata_file_processing_interval=5
I am curious if this interval is too short, causing a timeout on processing as the new processing task could be started, killing the previous one before it finishes processing. It is just conjecture at this point though.
Re: host-perfdata continuously grows
Posted: Wed Apr 30, 2014 2:19 pm
by pnewlon
Actually all is well now - the "process-host-perfdata-file-bulk" command had "/bin/mv" in it twice. I must have gotten my cursor in the wrong place when I modified it while creating the ramdisk and doing the mods.
Code: Select all
[root@lpnagv03 xidpe]# tail -50 /usr/local/nagios/var/perfdata.log
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08161_IB02-MP / pctProcTime (INFORMANT-STD::cpuPercentProcessorTime._Total=70)
2014-04-30 15:18:25 [6017] [1] Found Performance Data for 03774_IB09-MP / _HOST_ (rta=72.228996ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08161_IB04-MP / pctProcTime (INFORMANT-STD::cpuPercentProcessorTime._Total=62)
2014-04-30 15:18:25 [6017] [1] 13 lines processed
2014-04-30 15:18:25 [6017] [1] /var/nagiosramdisk/spool/perfdata//1398885492.perfdata.host-PID-6017 deleted
2014-04-30 15:18:25 [6017] [1] PNP exiting (runtime 0.043764s) ...
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 06627_IB08-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=42301)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08596_IB04-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=33249)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 06627_IB01-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=39093)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08232_IB01-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=940)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08232_IB06-DSP / Ping (-6])
2014-04-30 15:18:25 [6018] [1] Invalid Perfdata detected
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08257_IB09-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=2302)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08257_IB08-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=2307)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08257_IB04-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=2336)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08596_ENV01 / InternalTempFarenheitX100 (ROOMALERT4E-MIB::internal-tempf.0=7232)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08257_IB03-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=2105)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08227_IB03-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=32729)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 06627_IB04-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=38979)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02592_IB08-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=51781)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08250_IB04-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=32661)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08498_IB03-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=39224)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08232_OB02-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=232851)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 10421_IB09-DSP / NECtemp2F (tempF=95.9)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02475_IB02-DSP / NECtemp2F (tempF=82.4)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02475_IB04-DSP / NECtemp2F (tempF=94.1)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08161_IB09-DSP / NECtemp2F (tempF=97.7)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02434_IB09-AMT / Ping (rta=69.853996ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02475_IB03-MP / Ping (rta=70.221001ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02293_IB04-MP / Ping (rta=74.642998ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02434_IB08-AMT / Ping (rta=66.317001ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02634_IB04-MP / Ping (rta=78.741997ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08596_IB04-AMT / Ping (rta=111.503998ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08498_IB03-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=2130)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08498_IB01-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=2098)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08596_IB01-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=1062)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 09635_ENV02 / InternalTempFarenheitX100 (ROOMALERT4E-MIB::internal-tempf.0=7228)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02495_IB08-MP / Ping (rta=424.644989ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02495_IB02-AMT / Ping (rta=444.026001ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] 138 lines processed
2014-04-30 15:18:25 [6018] [1] /var/nagiosramdisk/spool/perfdata//1398885492.perfdata.service-PID-6018 deleted
2014-04-30 15:18:25 [6018] [1] PNP exiting (runtime 0.153134s) ...
2014-04-30 15:18:25 [6020] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2014-04-30 15:18:25 [6020] [1] Found Performance Data for 02495_HTTP / _HOST_ (time=1.036304s;;;0.000000 size=120B;;;0)
2014-04-30 15:18:25 [6020] [1] Found Performance Data for 03774_IB08-AMT / _HOST_ (rta=85.990997ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6020] [1] Found Performance Data for 01168_RTR / _HOST_ (rta=88.190002ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6020] [1] Found Performance Data for 06607_IB01-DSP / _HOST_ (rta=135.042007ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6020] [1] 4 lines processed
2014-04-30 15:18:25 [6020] [1] /var/nagiosramdisk/spool/perfdata//1398885502.perfdata.host-PID-6020 deleted
2014-04-30 15:18:25 [6020] [1] PNP exiting (runtime 0.01096s) ...
Code: Select all
[root@lpnagv03 xidpe]# tail -50 /usr/local/nagios/var/npcd.log
[04-30-2014 15:18:25] NPCD: ThreadCounter 0/5 File is .
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 0/5 File is ..
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 0/5 File is 1398885482.perfdata.host
[04-30-2014 15:18:25] NPCD: Regular File: 1398885482.perfdata.host
[04-30-2014 15:18:25] NPCD: A thread was started on thread_counter = 0
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 1/5 File is 1398885487.perfdata.host
[04-30-2014 15:18:25] NPCD: Regular File: 1398885487.perfdata.host
[04-30-2014 15:18:25] NPCD: Processing file 1398885482.perfdata.host with ID 140645669369600 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1398885482.perfdata.host
[04-30-2014 15:18:25] NPCD: Processing file '1398885482.perfdata.host'
[04-30-2014 15:18:25] NPCD: A thread was started on thread_counter = 1
[04-30-2014 15:18:25] NPCD: Processing file 1398885487.perfdata.host with ID 140645648393984 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1398885487.perfdata.host
[04-30-2014 15:18:25] NPCD: Processing file '1398885487.perfdata.host'
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 2/5 File is 1398885492.perfdata.host
[04-30-2014 15:18:25] NPCD: Regular File: 1398885492.perfdata.host
[04-30-2014 15:18:25] NPCD: A thread was started on thread_counter = 2
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 3/5 File is 1398885492.perfdata.service
[04-30-2014 15:18:25] NPCD: Regular File: 1398885492.perfdata.service
[04-30-2014 15:18:25] NPCD: A thread was started on thread_counter = 3
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 4/5 File is 1398885497.perfdata.host
[04-30-2014 15:18:25] NPCD: Regular File: 1398885497.perfdata.host
[04-30-2014 15:18:25] NPCD: A thread was started on thread_counter = 4
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 5/5 File is 1398885502.perfdata.host
[04-30-2014 15:18:25] NPCD: Regular File: 1398885502.perfdata.host
[04-30-2014 15:18:25] NPCD: WARN: MAX Thread reached: 1398885502.perfdata.host comes later with ThreadCounter: 5
[04-30-2014 15:18:25] NPCD: DEBUG: Will wait for th['4']
[04-30-2014 15:18:25] NPCD: Processing file 1398885497.perfdata.host with ID 140645507258112 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1398885497.perfdata.host
[04-30-2014 15:18:25] NPCD: Processing file '1398885497.perfdata.host'
[04-30-2014 15:18:25] NPCD: Processing file 1398885492.perfdata.host with ID 140645549209344 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1398885492.perfdata.host
[04-30-2014 15:18:25] NPCD: Processing file '1398885492.perfdata.host'
[04-30-2014 15:18:25] NPCD: Processing file 1398885492.perfdata.service with ID 140645528233728 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1398885492.perfdata.service
[04-30-2014 15:18:25] NPCD: Processing file '1398885492.perfdata.service'
[04-30-2014 15:18:25] NPCD: DEBUG: Will wait for th['3']
[04-30-2014 15:18:25] NPCD: DEBUG: Will wait for th['2']
[04-30-2014 15:18:25] NPCD: DEBUG: Will wait for th['1']
[04-30-2014 15:18:25] NPCD: DEBUG: Will wait for th['0']
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.710000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 0/5 File is 1398885502.perfdata.host
[04-30-2014 15:18:25] NPCD: Regular File: 1398885502.perfdata.host
[04-30-2014 15:18:25] NPCD: A thread was started on thread_counter = 0
[04-30-2014 15:18:25] NPCD: Have to wait: Filecounter = 6 - thread_counter = 1
[04-30-2014 15:18:25] NPCD: Processing file 1398885502.perfdata.host with ID 140645669369600 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1398885502.perfdata.host
[04-30-2014 15:18:25] NPCD: Processing file '1398885502.perfdata.host'
[04-30-2014 15:18:25] NPCD: No more files to process... waiting for 15 seconds
[root@lpnagv03 xidpe]#
Re: host-perfdata continuously grows
Posted: Wed Apr 30, 2014 2:26 pm
by abrist
pnewlon wrote:Actually all is well now - the "process-host-perfdata-file-bulk" command had "/bin/mv" in it twice.
Great. Shall we lock this thread?
Re: host-perfdata continuously grows
Posted: Wed Apr 30, 2014 2:27 pm
by pnewlon
Yes please!