Page 3 of 4
Re: Perfdata folder filling, yet is processing
Posted: Thu Sep 04, 2014 11:17 am
by BanditBBS
Code: Select all
[root@iss-chi-nag02 ~]# ls /var/nagiosramdisk/spool/xidpe | wc -l
0
[root@iss-chi-nag02 ~]# ls /var/nagiosramdisk/spool/perfdata/ | wc -l
2037
[root@iss-chi-nag02 ~]# ls /var/nagiosramdisk/spool/checkresults/ | wc -l
48
[root@iss-chi-nag02 ~]# which rrdtool
/usr/bin/rrdtool
[root@iss-chi-nag02 ~]# rrdtool --version
RRDtool 1.4.4 Copyright 1997-2010 by Tobias Oetiker <[email protected]>
Compiled May 6 2014 04:05:34
Usage: rrdtool [options] command command_options
Valid commands: create, update, updatev, graph, graphv, dump, restore,
last, lastupdate, first, info, fetch, tune,
resize, xport, flushcached
RRDtool is distributed under the Terms of the GNU General
Public License Version 2. (www.gnu.org/copyleft/gpl.html)
For more information read the RRD manpages
Re: Perfdata folder filling, yet is processing
Posted: Thu Sep 04, 2014 12:12 pm
by lmiltchev
Hm-m, usually if there are way too many files in the "perfdata" directory, this means that npcd is not (was not) running and not processing these files...
Do you have a high load issues on this server?
It's possible that npcd was not running, these files were not processed, and got piled up in the the "perfdata" directory. I am not sure if nagios can "catch up" from now on with so many files. If you don't care about historical data (whatever is in the "perfdata" directory at the moment), you can try:
Code: Select all
cd /var/nagiosramdisk/spool/
rm -rf perfdata
mkdir perfdata
chown nagios.nagios perfdata
chmod 755 perfdata
then watch "/var/nagiosramdisk/spool/perfdata/" directory for a while to see if the files will get processed.
BTW, is npcd running at the moment?
Re: Perfdata folder filling, yet is processing
Posted: Thu Sep 04, 2014 12:22 pm
by BanditBBS
Ludmil,
Short reply - no, not doing that
Longer reply - As I have already mentioned I have removed the files a couple times just to free up space. The more important item to make note of, every service is running that should be running. These files are being picked up and processed. They are having a pid# amended to the file name and future scans by npcd ignore the files because they are already in process of being processed. They just never finish completely and therefore don't get deleted, but the information in them is being processed and all graphs are working fine. Some files are completing fully and being deleted, its completely random. To answer your other question, load is fine.
Re: Perfdata folder filling, yet is processing
Posted: Thu Sep 04, 2014 2:01 pm
by Box293
BanditBBS wrote:Anyway, I can't get yum to function on this server, it gets to "Updating Red Hat repositories" and just freezes there for ever....bah
Did you have any luck getting rrdtool-perl installed?
Re: Perfdata folder filling, yet is processing
Posted: Thu Sep 04, 2014 2:53 pm
by BanditBBS
Box293 wrote:BanditBBS wrote:Anyway, I can't get yum to function on this server, it gets to "Updating Red Hat repositories" and just freezes there for ever....bah
Did you have any luck getting rrdtool-perl installed?
I just did, will let you know if that made any change
Re: Perfdata folder filling, yet is processing
Posted: Thu Sep 04, 2014 3:03 pm
by BanditBBS
Ok, this seems to be the issue....every time this error pops up, it is these files that end up not getting deleted:
Code: Select all
[09-04-2014 15:01:10] NPCD: Processing file '1409860859.perfdata.service'
[09-04-2014 15:01:14] NPCD: ERROR: Executed command exits with return code '13'
[09-04-2014 15:01:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1409860859.perfdata.service'
Re: Perfdata folder filling, yet is processing
Posted: Thu Sep 04, 2014 3:07 pm
by tmcdonald
Send over that 1409860859.perfdata.service file either in a post or PM - I have a feeling something in there is borking the perfdata processor.
Re: Perfdata folder filling, yet is processing
Posted: Thu Sep 04, 2014 3:22 pm
by BanditBBS
Here is the file:(I included a few that got that error)
perf.zip
I also found this that is a perfect match, but I can't read it
http://www.monitoring-portal.org/wbb/in ... adID=15815
Re: Perfdata folder filling, yet is processing
Posted: Thu Sep 04, 2014 3:55 pm
by Box293
Re: Perfdata folder filling, yet is processing
Posted: Fri Sep 05, 2014 8:30 am
by BanditBBS
"Look at me, I can use google!" LOL! I was just too lazy and busy....
But anyway, that link mentions file system perms:
Code: Select all
drwxrwxr-x 221 nagios nagios 12288 Sep 3 02:28 perfdata
and all the folders under that:
So those permissions look fine to me and as I said, all rrds seem to be getting updated fine, the perfdata process is just ending on some of these files and just not deleting them. Hopefully they can see something in the perfdata files I attached earlier that could be causing this issue.
Thanks for looking at this Box and please continue to throw out ideas if you think of anything else.