Perfdata folder filling, yet is processing

[BanditBBS]

I've run out of ideas.

LOL, me too, that's why I posted in the first place! Its just very weird since some files are being removed but all perfdata seems to be getting processed.

[Box293]

PM the log files and I'll have a look at them.

[BanditBBS]

PM the log files and I'll have a look at them.

I'll do that in a bit. The permissions on npcd.log weren't right and it wasn't being updated. I am going to remove all these old perf data files so we can say "from this time on"

[lmiltchev]

Did you PM Troy the log files? Are you still having an issue with perfdata folder filling up?

[BanditBBS]

Did you PM Troy the log files? Are you still having an issue with perfdata folder filling up?

I'm about to PM them now. Yes I am still having the issue andseeing this in the log over and over:

Code: Select all

[09-03-2014 10:40:40] NPCD: File '1409694688.perfdata.service-PID-2783' is an already in process PNP file. Leaving it untouched.

Looks like the process is grabbing and processing all the files but for some reason the process is not finishing on most of the files and therefore not deleting them.

[slansing]

Hmm, you may want to bump out your timeout value in:

Code: Select all

/usr/local/nagios/etc/pnp/process_perfdata.cfg

If it is the default, I'd move it up to 30 seconds. In the past, I know a user had removed his RRD files and let them regenerate to fix the issue, I don't think it has any negative impact in most cases, but you may be missing some data points thus making your averages display at a lower resolution via the graphs. Of course, removing the RRD files is a bit rough, especially with a lot of valuable historical perf data.

EDIT: Whoa, didn't know it was at 120 already.

[BanditBBS]

Timeout is already 120

[Box293]

I noticed the following warning in perfdata.log:

Code: Select all

RRDs Perl Modules are not installed. Falling back to rrdtool system call.

Try installing the module to see if it helps:

Code: Select all

yum install rrdtool-perl

Timeout was my other suggestion however it appears that has been covered.

In nagios.cfg, what are the settings for:

Code: Select all

service_perfdata_file_mode
service_perfdata_file_processing_command
host_perfdata_file_mode
host_perfdata_file_processing_command

[BanditBBS]

OMG - This day has just sucked!

Anyway, I can't get yum to function on this server, it gets to "Updating Red Hat repositories" and just freezes there for ever....bah

As for the other question:

Code: Select all

service_perfdata_file_mode=a
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
host_perfdata_file_mode=a
host_perfdata_file_processing_command=process-host-perfdata-file-bulk

Those commands are here:

Code: Select all

define command {
       command_name                             process-host-perfdata-file-bulk
       command_line                             /bin/mv /var/nagiosramdisk/host-perfdata /var/nagiosramdisk/spool/xidpe/$TIMET$.perfdata.host
}
define command {
       command_name                             process-service-perfdata-file-bulk
       command_line                             /bin/mv /var/nagiosramdisk/service-perfdata /var/nagiosramdisk/spool/xidpe/$TIMET$.perfdata.service
}

[lmiltchev]

These commands look fine. Can you run the following commands and show us the output?

Code: Select all

ls /var/nagiosramdisk/spool/xidpe | wc -l
ls /var/nagiosramdisk/spool/perfdata/ | wc -l
ls /var/nagiosramdisk/spool/checkresults/ | wc -l
which rrdtool
rrdtool --version