host-perfdata continuously grows

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
pnewlon
Posts: 86
Joined: Mon May 16, 2011 2:19 pm

host-perfdata continuously grows

Post by pnewlon »

I have a 750MB ramdisk that keeps filling because host-perfdata never shrinks. It took just short of seven days to fill the ramdisk. What do I have configured wrong that is causing this to keep growing?


tmpfs 750M 750M 160K 100% /var/nagiosramdisk

# cd /var/nagiosramdisk/
# ls -lt
-rw-rw-r-- 1 nagios nagios 0 Apr 30 07:48 service-perfdata
drwxrwxr-x 2 nagios nagios 360 Apr 30 07:48 tmp
-rw-rw-r-- 1 nagios nagios 780091489 Apr 29 01:04 host-perfdata
-rw-r--r-- 1 nagios nagios 4622171 Apr 22 13:38 objects.cache
drwxrwxr-x 5 nagios nagios 100 Apr 22 13:37 spool

First record: Tue, 22 Apr 2014 17:38:13 GMT
DATATYPE::HOSTPERFDATA TIMET::1398188293 HOSTNAME::00697_HTTP HOSTPERFDATA::time=0.308773s;;;0.000000 size=120B;;;0 HOSTCHECKCOMMAND::check_xi_service_http!-p 16992 -t 4!!!!!!! HOSTSTATE::UP HOSTSTATETYPE::HARD HOSTOUTPUT::HTTP OK - HTTP/1.1 303 See Other - 0.309 second response time

Last record: Tue, 29 Apr 2014 05:04:06 GMT
DATATYPE::HOSTPERFDATA TIMET::1398747846 HOSTNAME::08232_IB04-MP HOSTPERFDATA::rta=90.132004ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0 HOSTCHECKCOMMAND::check_ping!3000,80%!5000,100%!!!!!! HOSTSTATE::UP HOSTSTATETYPE::HARD HOSTOUTPUT::PING OK - Packet loss = 0%, RTA = 90.13 ms
Top
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:
Contact scottwilkerson

Re: host-perfdata continuously grows

Post by scottwilkerson »

I have a feeling you might have missed the last step on page 3 of
http://assets.nagios.com/downloads/nagi ... giosXI.pdf

what do the following commands look like in the Core Config Manager
process-host-perfdata-file-bulk
process-service-perfdata-file-bulk

also what is the output of the following

Code: Select all

cat /usr/local/nagios/etc/nagios.cfg|grep perfdata_file_processing_command
Former Nagios employee
Creator:
Human Design Website
Get Your Human Design Chart
Top
pnewlon
Posts: 86
Joined: Mon May 16, 2011 2:19 pm

Re: host-perfdata continuously grows

Post by pnewlon »

Of course the last thing I check is the "what do the following commands look like in the Core Config Manager".

[root@lpnagv03 etc]# cat /usr/local/nagios/etc/nagios.cfg|grep perfdata_file_processing_command
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
host_perfdata_file_processing_command=process-host-perfdata-file-bulk

# host performance data
#host_perfdata_file=/usr/local/nagios/var/host-perfdata
host_perfdata_file=/var/nagiosramdisk/host-perfdata

# service performance data
#service_perfdata_file=/usr/local/nagios/var/service-perfdata
service_perfdata_file=/var/nagiosramdisk/service-perfdata

[root@lpnagv03 nagiosramdisk]# cat /usr/local/nagios/etc/nagios.cfg|grep ramd
service_perfdata_file=/var/nagiosramdisk/service-perfdata
host_perfdata_file=/var/nagiosramdisk/host-perfdata
check_result_path=/var/nagiosramdisk/spool/checkresults
object_cache_file=/var/nagiosramdisk/objects.cache
status_file=/var/nagiosramdisk/status.dat
temp_path=/var/nagiosramdisk/tmp


Interestingly, service perfdata is processing...

[root@lpnagv03 nagiosramdisk]# ls -lt
total 21988
drwxrwxr-x 2 nagios nagios 15500 Apr 30 14:34 tmp
-rw-r--r-- 1 nagios nagios 6702401 Apr 30 14:34 status.dat
-rw-rw-r-- 1 nagios nagios 22196 Apr 30 14:34 service-perfdata
-rw-rw-r-- 1 nagios nagios 11095999 Apr 30 14:34 host-perfdata
-rw-r--r-- 1 nagios nagios 4622171 Apr 30 08:37 objects.cache
drwxrwxr-x 5 nagios nagios 100 Apr 22 13:37 spool
You do not have the required permissions to view the files attached to this post.
Top
abrist
Red Shirt
Posts: 8334
Joined: Thu Nov 15, 2012 1:20 pm

Re: host-perfdata continuously grows

Post by abrist »

What do you have set for the perfdata processing intervals?

Code: Select all

grep perfdata_file_processing_interval /usr/local/nagios/etc/nagios.cfg
Former Nagios employee
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
Top
pnewlon
Posts: 86
Joined: Mon May 16, 2011 2:19 pm

Re: host-perfdata continuously grows

Post by pnewlon »

[root@lpnagv03 nagiosramdisk]# grep perfdata_file_processing_interval /usr/local/nagios/etc/nagios.cfg
service_perfdata_file_processing_interval=15
#host_perfdata_file_processing_interval=15
host_perfdata_file_processing_interval=5
Top
abrist
Red Shirt
Posts: 8334
Joined: Thu Nov 15, 2012 1:20 pm

Re: host-perfdata continuously grows

Post by abrist »

Any errors in the perfdata/npcd logs?

Code: Select all

tail -50 /usr/local/nagios/var/perfdata.log
tail -50 /usr/local/nagios/var/npcd.log
pnewlon wrote:host_perfdata_file_processing_interval=5
I am curious if this interval is too short, causing a timeout on processing as the new processing task could be started, killing the previous one before it finishes processing. It is just conjecture at this point though.
Former Nagios employee
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
Top
pnewlon
Posts: 86
Joined: Mon May 16, 2011 2:19 pm

Re: host-perfdata continuously grows

Post by pnewlon »

Actually all is well now - the "process-host-perfdata-file-bulk" command had "/bin/mv" in it twice. I must have gotten my cursor in the wrong place when I modified it while creating the ramdisk and doing the mods.

Code: Select all

[root@lpnagv03 xidpe]# tail -50 /usr/local/nagios/var/perfdata.log
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08161_IB02-MP / pctProcTime (INFORMANT-STD::cpuPercentProcessorTime._Total=70)
2014-04-30 15:18:25 [6017] [1] Found Performance Data for 03774_IB09-MP / _HOST_ (rta=72.228996ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08161_IB04-MP / pctProcTime (INFORMANT-STD::cpuPercentProcessorTime._Total=62)
2014-04-30 15:18:25 [6017] [1] 13 lines processed
2014-04-30 15:18:25 [6017] [1] /var/nagiosramdisk/spool/perfdata//1398885492.perfdata.host-PID-6017 deleted
2014-04-30 15:18:25 [6017] [1] PNP exiting (runtime 0.043764s) ...
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 06627_IB08-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=42301)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08596_IB04-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=33249)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 06627_IB01-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=39093)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08232_IB01-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=940)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08232_IB06-DSP / Ping (-6])
2014-04-30 15:18:25 [6018] [1] Invalid Perfdata detected
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08257_IB09-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=2302)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08257_IB08-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=2307)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08257_IB04-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=2336)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08596_ENV01 / InternalTempFarenheitX100 (ROOMALERT4E-MIB::internal-tempf.0=7232)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08257_IB03-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=2105)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08227_IB03-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=32729)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 06627_IB04-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=38979)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02592_IB08-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=51781)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08250_IB04-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=32661)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08498_IB03-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=39224)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08232_OB02-MP / logicalDiskFreeSpace (INFORMANT-STD::lDiskFreeMegabytes._Total=232851)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 10421_IB09-DSP / NECtemp2F (tempF=95.9)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02475_IB02-DSP / NECtemp2F (tempF=82.4)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02475_IB04-DSP / NECtemp2F (tempF=94.1)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08161_IB09-DSP / NECtemp2F (tempF=97.7)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02434_IB09-AMT / Ping (rta=69.853996ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02475_IB03-MP / Ping (rta=70.221001ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02293_IB04-MP / Ping (rta=74.642998ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02434_IB08-AMT / Ping (rta=66.317001ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02634_IB04-MP / Ping (rta=78.741997ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08596_IB04-AMT / Ping (rta=111.503998ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08498_IB03-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=2130)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08498_IB01-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=2098)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 08596_IB01-MP / memAvailMB (INFORMANT-STD::memoryAvailableMBytes.0=1062)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 09635_ENV02 / InternalTempFarenheitX100 (ROOMALERT4E-MIB::internal-tempf.0=7228)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02495_IB08-MP / Ping (rta=424.644989ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] Found Performance Data for 02495_IB02-AMT / Ping (rta=444.026001ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6018] [1] 138 lines processed
2014-04-30 15:18:25 [6018] [1] /var/nagiosramdisk/spool/perfdata//1398885492.perfdata.service-PID-6018 deleted
2014-04-30 15:18:25 [6018] [1] PNP exiting (runtime 0.153134s) ...
2014-04-30 15:18:25 [6020] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2014-04-30 15:18:25 [6020] [1] Found Performance Data for 02495_HTTP / _HOST_ (time=1.036304s;;;0.000000 size=120B;;;0)
2014-04-30 15:18:25 [6020] [1] Found Performance Data for 03774_IB08-AMT / _HOST_ (rta=85.990997ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6020] [1] Found Performance Data for 01168_RTR / _HOST_ (rta=88.190002ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6020] [1] Found Performance Data for 06607_IB01-DSP / _HOST_ (rta=135.042007ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2014-04-30 15:18:25 [6020] [1] 4 lines processed
2014-04-30 15:18:25 [6020] [1] /var/nagiosramdisk/spool/perfdata//1398885502.perfdata.host-PID-6020 deleted
2014-04-30 15:18:25 [6020] [1] PNP exiting (runtime 0.01096s) ...

Code: Select all

[root@lpnagv03 xidpe]# tail -50 /usr/local/nagios/var/npcd.log
[04-30-2014 15:18:25] NPCD: ThreadCounter 0/5 File is .
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 0/5 File is ..
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 0/5 File is 1398885482.perfdata.host
[04-30-2014 15:18:25] NPCD: Regular File: 1398885482.perfdata.host
[04-30-2014 15:18:25] NPCD: A thread was started on thread_counter = 0
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 1/5 File is 1398885487.perfdata.host
[04-30-2014 15:18:25] NPCD: Regular File: 1398885487.perfdata.host
[04-30-2014 15:18:25] NPCD: Processing file 1398885482.perfdata.host with ID 140645669369600 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1398885482.perfdata.host
[04-30-2014 15:18:25] NPCD: Processing file '1398885482.perfdata.host'
[04-30-2014 15:18:25] NPCD: A thread was started on thread_counter = 1
[04-30-2014 15:18:25] NPCD: Processing file 1398885487.perfdata.host with ID 140645648393984 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1398885487.perfdata.host
[04-30-2014 15:18:25] NPCD: Processing file '1398885487.perfdata.host'
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 2/5 File is 1398885492.perfdata.host
[04-30-2014 15:18:25] NPCD: Regular File: 1398885492.perfdata.host
[04-30-2014 15:18:25] NPCD: A thread was started on thread_counter = 2
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 3/5 File is 1398885492.perfdata.service
[04-30-2014 15:18:25] NPCD: Regular File: 1398885492.perfdata.service
[04-30-2014 15:18:25] NPCD: A thread was started on thread_counter = 3
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 4/5 File is 1398885497.perfdata.host
[04-30-2014 15:18:25] NPCD: Regular File: 1398885497.perfdata.host
[04-30-2014 15:18:25] NPCD: A thread was started on thread_counter = 4
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.600000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 5/5 File is 1398885502.perfdata.host
[04-30-2014 15:18:25] NPCD: Regular File: 1398885502.perfdata.host
[04-30-2014 15:18:25] NPCD: WARN: MAX Thread reached: 1398885502.perfdata.host comes later with ThreadCounter: 5
[04-30-2014 15:18:25] NPCD: DEBUG: Will wait for th['4']
[04-30-2014 15:18:25] NPCD: Processing file 1398885497.perfdata.host with ID 140645507258112 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1398885497.perfdata.host
[04-30-2014 15:18:25] NPCD: Processing file '1398885497.perfdata.host'
[04-30-2014 15:18:25] NPCD: Processing file 1398885492.perfdata.host with ID 140645549209344 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1398885492.perfdata.host
[04-30-2014 15:18:25] NPCD: Processing file '1398885492.perfdata.host'
[04-30-2014 15:18:25] NPCD: Processing file 1398885492.perfdata.service with ID 140645528233728 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1398885492.perfdata.service
[04-30-2014 15:18:25] NPCD: Processing file '1398885492.perfdata.service'
[04-30-2014 15:18:25] NPCD: DEBUG: Will wait for th['3']
[04-30-2014 15:18:25] NPCD: DEBUG: Will wait for th['2']
[04-30-2014 15:18:25] NPCD: DEBUG: Will wait for th['1']
[04-30-2014 15:18:25] NPCD: DEBUG: Will wait for th['0']
[04-30-2014 15:18:25] NPCD: DEBUG: load 3.710000/30.000000
[04-30-2014 15:18:25] NPCD: ThreadCounter 0/5 File is 1398885502.perfdata.host
[04-30-2014 15:18:25] NPCD: Regular File: 1398885502.perfdata.host
[04-30-2014 15:18:25] NPCD: A thread was started on thread_counter = 0
[04-30-2014 15:18:25] NPCD: Have to wait: Filecounter = 6 - thread_counter = 1
[04-30-2014 15:18:25] NPCD: Processing file 1398885502.perfdata.host with ID 140645669369600 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1398885502.perfdata.host
[04-30-2014 15:18:25] NPCD: Processing file '1398885502.perfdata.host'
[04-30-2014 15:18:25] NPCD: No more files to process... waiting for 15 seconds
[root@lpnagv03 xidpe]#
Top
abrist
Red Shirt
Posts: 8334
Joined: Thu Nov 15, 2012 1:20 pm

Re: host-perfdata continuously grows

Post by abrist »

pnewlon wrote:Actually all is well now - the "process-host-perfdata-file-bulk" command had "/bin/mv" in it twice.
Great. Shall we lock this thread?
Former Nagios employee
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
Top
pnewlon
Posts: 86
Joined: Mon May 16, 2011 2:19 pm

Re: host-perfdata continuously grows

Post by pnewlon »

Yes please!
Top
Locked

Return to “Nagios XI”