Perfdata folder filling, yet is processing

[BanditBBS]

Code: Select all

[root@iss-chi-nag02 ~]# ls /var/nagiosramdisk/spool/xidpe | wc -l
0
[root@iss-chi-nag02 ~]# ls /var/nagiosramdisk/spool/perfdata/ | wc -l
2037
[root@iss-chi-nag02 ~]# ls /var/nagiosramdisk/spool/checkresults/ | wc -l
48
[root@iss-chi-nag02 ~]# which rrdtool
/usr/bin/rrdtool
[root@iss-chi-nag02 ~]# rrdtool --version
RRDtool 1.4.4  Copyright 1997-2010 by Tobias Oetiker <[email protected]>
               Compiled May  6 2014 04:05:34

Usage: rrdtool [options] command command_options
Valid commands: create, update, updatev, graph, graphv,  dump, restore,
                last, lastupdate, first, info, fetch, tune,
                resize, xport, flushcached

RRDtool is distributed under the Terms of the GNU General
Public License Version 2. (www.gnu.org/copyleft/gpl.html)

For more information read the RRD manpages

[lmiltchev]

Hm-m, usually if there are way too many files in the "perfdata" directory, this means that npcd is not (was not) running and not processing these files...
Do you have a high load issues on this server?
It's possible that npcd was not running, these files were not processed, and got piled up in the the "perfdata" directory. I am not sure if nagios can "catch up" from now on with so many files. If you don't care about historical data (whatever is in the "perfdata" directory at the moment), you can try:

Code: Select all

cd /var/nagiosramdisk/spool/
rm -rf perfdata
mkdir perfdata
chown nagios.nagios perfdata
chmod 755 perfdata

then watch "/var/nagiosramdisk/spool/perfdata/" directory for a while to see if the files will get processed.

BTW, is npcd running at the moment?

Code: Select all

service npcd status

[BanditBBS]

Ludmil,

Short reply - no, not doing that

Longer reply - As I have already mentioned I have removed the files a couple times just to free up space. The more important item to make note of, every service is running that should be running. These files are being picked up and processed. They are having a pid# amended to the file name and future scans by npcd ignore the files because they are already in process of being processed. They just never finish completely and therefore don't get deleted, but the information in them is being processed and all graphs are working fine. Some files are completing fully and being deleted, its completely random. To answer your other question, load is fine.

[Box293]

Anyway, I can't get yum to function on this server, it gets to "Updating Red Hat repositories" and just freezes there for ever....bah

Did you have any luck getting rrdtool-perl installed?

[BanditBBS]

Anyway, I can't get yum to function on this server, it gets to "Updating Red Hat repositories" and just freezes there for ever....bah

Did you have any luck getting rrdtool-perl installed?

I just did, will let you know if that made any change

[BanditBBS]

Ok, this seems to be the issue....every time this error pops up, it is these files that end up not getting deleted:

Code: Select all

[09-04-2014 15:01:10] NPCD: Processing file '1409860859.perfdata.service'
[09-04-2014 15:01:14] NPCD: ERROR: Executed command exits with return code '13'
[09-04-2014 15:01:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1409860859.perfdata.service'

[tmcdonald]

Send over that 1409860859.perfdata.service file either in a post or PM - I have a feeling something in there is borking the perfdata processor.

[BanditBBS]

Here is the file:(I included a few that got that error)

perf.zip

I also found this that is a perfect match, but I can't read it http://www.monitoring-portal.org/wbb/in ... adID=15815

[Box293]

Here's the translation of that URL you provided

https://translate.google.com/translate? ... edit-text=

[BanditBBS]

Here's the translation of that URL you provided

https://translate.google.com/translate? ... edit-text=

"Look at me, I can use google!" LOL! I was just too lazy and busy....

But anyway, that link mentions file system perms:

Code: Select all

drwxrwxr-x 221 nagios nagios 12288 Sep  3 02:28 perfdata

and all the folders under that:

Code: Select all

drwxrwxr-x 2 nagios nagios  4096

So those permissions look fine to me and as I said, all rrds seem to be getting updated fine, the perfdata process is just ending on some of these files and just not deleting them. Hopefully they can see something in the perfdata files I attached earlier that could be causing this issue.

Thanks for looking at this Box and please continue to throw out ideas if you think of anything else.