Page 1 of 4
Perfdata folder filling, yet is processing
Posted: Sun Aug 31, 2014 10:20 am
by BanditBBS
This is for my 2012 2.9 server:
/var/nagiosramdisk/spool/perfdata had 2+ weeks worth of files in it. I wiped them out as the drive was near full. The odd thing is, graphs are all showing data, it is as though the perfdata is being processed, just not removed from that folder. I know the normal stuff to check when perfdata isn't working at all and checked that stuff, everything looks fine. Any hints at what to look at here?
Re: Perfdata folder filling, yet is processing
Posted: Tue Sep 02, 2014 1:50 pm
by abrist
What permissions do the files have? They could have their permissions overwritten by a umask (should be 0022), so lets check that as well.
Code: Select all
umask
ls -la <path to perfdata file>
The expected permissions are:
Code: Select all
-rw-r--r-- 1 nagios nagios ........
Is the nagios xi ui updating the last check time, state, etc?
Re: Perfdata folder filling, yet is processing
Posted: Tue Sep 02, 2014 1:55 pm
by BanditBBS
As the nagios user:
Code: Select all
[nagios@iss-chi-nag02 perfdata]$ umask
0002
Small example of the ls:
Code: Select all
-rw-rw-r-- 1 nagios nagios 752157 Sep 2 13:38 1409683086.perfdata.service-PID-29700
-rw-rw-r-- 1 nagios nagios 107438 Sep 2 13:39 1409683146.perfdata.service-PID-31787
-rw-rw-r-- 1 nagios nagios 2038258 Sep 2 13:41 1409683266.perfdata.service-PID-7592
-rw-rw-r-- 1 nagios nagios 7737 Sep 2 13:44 1409683446.perfdata.host-PID-18235
-rw-rw-r-- 1 nagios nagios 135385 Sep 2 13:44 1409683446.perfdata.service-PID-18236
-rw-rw-r-- 1 nagios nagios 2069892 Sep 2 13:46 1409683566.perfdata.service-PID-24938
-rw-rw-r-- 1 nagios nagios 44748 Sep 2 13:48 1409683686.perfdata.host-PID-32438
-rw-rw-r-- 1 nagios nagios 741479 Sep 2 13:48 1409683686.perfdata.service-PID-32437
-rw-rw-r-- 1 nagios nagios 128494 Sep 2 13:49 1409683746.perfdata.service-PID-3170
-rw-rw-r-- 1 nagios nagios 2094572 Sep 2 13:51 1409683866.perfdata.service-PID-11290
The UI all looks good, as I said it seems like everything is being processed, just the files are not being removed.
Re: Perfdata folder filling, yet is processing
Posted: Tue Sep 02, 2014 2:50 pm
by Box293
What is the permissions of the directory itself?
For example:
Code: Select all
ls /usr/local/nagios/var/spool/ -al
total 20
drwxr-xr-x 5 nagios nagios 4096 Aug 14 02:33 .
drwxrwxr-x 6 nagios nagios 4096 Sep 2 15:45 ..
drwxrwsr-x 2 nagios nagios 4096 Sep 2 14:19 checkresults
drwxr-xr-x 2 nagios nagios 4096 Sep 2 15:46 perfdata
drwxr-xr-x 2 nagios nagios 4096 Sep 2 15:46 xidpe
Re: Perfdata folder filling, yet is processing
Posted: Tue Sep 02, 2014 2:53 pm
by BanditBBS
Code: Select all
total 0
drwxrwxr-x 5 nagios nagios 100 Aug 1 08:20 .
drwxrwxrwt 4 nagios nagios 160 Sep 2 14:52 ..
drwxrwxr-x 2 nagios nagios 40 Sep 2 14:52 checkresults
drwxrwxr-x 2 nagios nagios 50560 Sep 2 14:52 perfdata
drwxrwxr-x 2 nagios nagios 40 Sep 2 14:52 xidpe
Re: Perfdata folder filling, yet is processing
Posted: Tue Sep 02, 2014 3:35 pm
by Box293
That all looks good.
Lets turn on debug logging and see what this tells us:
Edit
/usr/local/nagios/etc/pnp/process_perfdata.cfg
And set
LOG_LEVEL = 2
Also
Edit
/usr/local/nagios/etc/pnp/npcd.cfg
And set
log_level = -1
Now this will create two log files:
/usr/local/nagios/var/perfdata.log
/usr/local/nagios/var/npcd.log
See if this indicates what is going on.
Re: Perfdata folder filling, yet is processing
Posted: Tue Sep 02, 2014 4:01 pm
by BanditBBS
Nothing staring me in the face pointing to an issue. It appears it is finding perf data and deleting those files properly. The files that are being left behind are all filled with hosts or services that don't have perf data with the results(i.e. switch port status saying it is UP).
Re: Perfdata folder filling, yet is processing
Posted: Tue Sep 02, 2014 4:04 pm
by Box293
Perhaps an alternative solution here is to turn off processing of performance data (process_perf_data) for those services.
Re: Perfdata folder filling, yet is processing
Posted: Tue Sep 02, 2014 4:14 pm
by BanditBBS
Box293 wrote:Perhaps an alternative solution here is to turn off processing of performance data (process_perf_data) for those services.
Forget what I said. There is legit perf data in the files that are not being deleted. From what I can see it is processing the files and the data is being graphed. It is randomly deleting some of the files in the folder though.
Re: Perfdata folder filling, yet is processing
Posted: Tue Sep 02, 2014 4:22 pm
by Box293
I've run out of ideas.