Perfdata folder filling, yet is processing

[BanditBBS]

This is for my 2012 2.9 server:

/var/nagiosramdisk/spool/perfdata had 2+ weeks worth of files in it. I wiped them out as the drive was near full. The odd thing is, graphs are all showing data, it is as though the perfdata is being processed, just not removed from that folder. I know the normal stuff to check when perfdata isn't working at all and checked that stuff, everything looks fine. Any hints at what to look at here?

[abrist]

What permissions do the files have? They could have their permissions overwritten by a umask (should be 0022), so lets check that as well.

Code: Select all

umask
ls -la <path to perfdata file>

The expected permissions are:

Code: Select all

-rw-r--r-- 1 nagios nagios  ........

Is the nagios xi ui updating the last check time, state, etc?

[BanditBBS]

As the nagios user:

Code: Select all

[nagios@iss-chi-nag02 perfdata]$ umask
0002

Small example of the ls:

Code: Select all

-rw-rw-r-- 1 nagios nagios  752157 Sep  2 13:38 1409683086.perfdata.service-PID-29700
-rw-rw-r-- 1 nagios nagios  107438 Sep  2 13:39 1409683146.perfdata.service-PID-31787
-rw-rw-r-- 1 nagios nagios 2038258 Sep  2 13:41 1409683266.perfdata.service-PID-7592
-rw-rw-r-- 1 nagios nagios    7737 Sep  2 13:44 1409683446.perfdata.host-PID-18235
-rw-rw-r-- 1 nagios nagios  135385 Sep  2 13:44 1409683446.perfdata.service-PID-18236
-rw-rw-r-- 1 nagios nagios 2069892 Sep  2 13:46 1409683566.perfdata.service-PID-24938
-rw-rw-r-- 1 nagios nagios   44748 Sep  2 13:48 1409683686.perfdata.host-PID-32438
-rw-rw-r-- 1 nagios nagios  741479 Sep  2 13:48 1409683686.perfdata.service-PID-32437
-rw-rw-r-- 1 nagios nagios  128494 Sep  2 13:49 1409683746.perfdata.service-PID-3170
-rw-rw-r-- 1 nagios nagios 2094572 Sep  2 13:51 1409683866.perfdata.service-PID-11290

The UI all looks good, as I said it seems like everything is being processed, just the files are not being removed.

[Box293]

What is the permissions of the directory itself?

For example:

Code: Select all

ls /usr/local/nagios/var/spool/ -al
total 20
drwxr-xr-x 5 nagios nagios 4096 Aug 14 02:33 .
drwxrwxr-x 6 nagios nagios 4096 Sep  2 15:45 ..
drwxrwsr-x 2 nagios nagios 4096 Sep  2 14:19 checkresults
drwxr-xr-x 2 nagios nagios 4096 Sep  2 15:46 perfdata
drwxr-xr-x 2 nagios nagios 4096 Sep  2 15:46 xidpe

[BanditBBS]

Code: Select all

total 0
drwxrwxr-x 5 nagios nagios   100 Aug  1 08:20 .
drwxrwxrwt 4 nagios nagios   160 Sep  2 14:52 ..
drwxrwxr-x 2 nagios nagios    40 Sep  2 14:52 checkresults
drwxrwxr-x 2 nagios nagios 50560 Sep  2 14:52 perfdata
drwxrwxr-x 2 nagios nagios    40 Sep  2 14:52 xidpe

[Box293]

That all looks good.

Lets turn on debug logging and see what this tells us:
Edit
/usr/local/nagios/etc/pnp/process_perfdata.cfg
And set
LOG_LEVEL = 2

Also
Edit
/usr/local/nagios/etc/pnp/npcd.cfg
And set
log_level = -1

Now this will create two log files:
/usr/local/nagios/var/perfdata.log
/usr/local/nagios/var/npcd.log

See if this indicates what is going on.

[BanditBBS]

Nothing staring me in the face pointing to an issue. It appears it is finding perf data and deleting those files properly. The files that are being left behind are all filled with hosts or services that don't have perf data with the results(i.e. switch port status saying it is UP).

[Box293]

Perhaps an alternative solution here is to turn off processing of performance data (process_perf_data) for those services.

[BanditBBS]

Perhaps an alternative solution here is to turn off processing of performance data (process_perf_data) for those services.

Forget what I said. There is legit perf data in the files that are not being deleted. From what I can see it is processing the files and the data is being graphed. It is randomly deleting some of the files in the folder though.

[Box293]

I've run out of ideas.