URGENT - Perfdata processing is off, but perfdata exists???

[cscholz]

I just ran out of inodes because /usr/local/nagios/var/spool/xidpe was full of approximately 40,000 files, all named something similar to 1340467343.perfdata.service and 1340467380.perfdata.host. Additionally, seemingly out of nowhere, /usr/local/nagios/var/spool/perfdata started logging perfdata files after two new hosts were added (With perfdata = skip in the host declaration page in CCM)

I have perfdata processing turned off in the nagios core config because this is a VM with limited disk space and io capabilities. We are keeping performance data off until we can migrate this VM to a physical machine.

Please, someone explain to me how to ENSURE 100% that performance data is NOT being written. We lost total use of Nagios XI for 10 hours overnight because of this inode problem, and we can't let it happen again.

For now I'm setting up a monitor on <...>/var/spool/xidpe inode use, but this shouldn't even be happening in the first place..

Help! Thank you!!!!

[mguthrie]

What version of Nagios XI are you running?

Are you utilizing a RAM disk at all?

Can you access the Admin- page and verify that all subsystem processes are running? There is a subsystem process that runs on a 5mn cron that should be keeping that directory cleaned.

[cscholz]

What version of Nagios XI are you running?

Are you utilizing a RAM disk at all?

Can you access the Admin- page and verify that all subsystem processes are running? There is a subsystem process that runs on a 5mn cron that should be keeping that directory cleaned.

2011R2.1

No RAM disk

All processes are running according to Admin page. Also:

[root@nagsys perfdata]# service crond status
crond (pid 2188) is running...

[root@nagsys perfdata]# ~/statusall.sh
httpd (pid 11793) is running...
mysqld (pid 11953) is running...
postmaster (pid 12019) is running...
nagios (pid 25841) is running...
ndo2db (pid 12052) is running...
NPCD running (pid 12064).
nrpe (pid 21594) is running...

Are these the proper cron jobs?
######################################
# Nagios XI Cron Jobs - Run as nagios user
######################################
#* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
#* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
#* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
#* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
#* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
#* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1
#* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1
#*/5 * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
#* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1

[cscholz]

I got that file by doing "crontab -u nagios -e" is that the correct file? Why are all the lines commented out. I'm 100% positive that this was not done by me, so it could only have been done by one of the scripts I've run downloaded from nagios.

Are you aware of any scripts that comment out crontab lines?

[mguthrie]

We did move the Nagios XI cron jobs to /etc/cron.d/nagiosxi, I don't remember which version that took place. More recent versions clean up that cron tab more nicely. If the cron jobs are completing successfully the Admin->System Component Status dashlet should be all green.

[cscholz]

We did move the Nagios XI cron jobs to /etc/cron.d/nagiosxi, I don't remember which version that took place. More recent versions clean up that cron tab more nicely. If the cron jobs are completing successfully the Admin->System Component Status dashlet should be all green.

The dashlet is all green.

Any other ideas on why perfdata is being written? It's still going:

[root@nagsys perfdata]# ll
total 73608
-rw-r--r-- 1 nagios users 37686013 Jul 13 10:19 host-perfdata.1338828170-PID-1713
-rw-r--r-- 1 nagios users 37686013 Jul 13 10:10 host-perfdata.1338828170-PID-26688

[mguthrie]

Can you check the Nagios XI Home->Process Info page and take a look at the monitoring engine process dashlet and verify that the runtime variable for processing performance data is turned off?

[cscholz]

Can you check the Nagios XI Home->Process Info page and take a look at the monitoring engine process dashlet and verify that the runtime variable for processing performance data is turned off?

It's red. I haven't seen any new files written since the last two I posted.

[mguthrie]

Ok. I'm a little stumped as to how those files got there, that part happens as part of the Core daemon as checks are being processed. When results are received it checks if perfdata processing is enabled globally, then locally for the host/service, and if it is then it drops it into that spool directory.

Keep that check running against that directory, let us know if it shows up again and we'll keep digging on it.

[cscholz]

Nothing since... I guess this is a closed issue...

On another note, I see that Nagios CCM audit logs are only available through the UI for the current day. Where are the rest? If I want to see who made a change yesterday, where would I find the logs on the filesystem?