Page 1 of 1
200,000+ files created in xidpe directory
Posted: Fri Mar 09, 2012 12:28 am
by BWNetworks
Hi
I am currently running a Nagios XI test box while we wait for our license to be purchased. The server locked up as the file system run out of inodes.
Upon investigation, I found that /usr/local/nagios/var/spool/xidpe/ had over 200,000 files.
I cleared out the directory to restore service, however, it looks like it is filling up again. Example of these files are below:
Code: Select all
-rw-rw-r-- 1 nagios nagios 245 Mar 9 13:25 1331270724.perfdata.host
-rw-rw-r-- 1 nagios nagios 0 Mar 9 13:25 1331270724.perfdata.service
-rw-rw-r-- 1 nagios nagios 0 Mar 9 13:25 1331270739.perfdata.host
-rw-rw-r-- 1 nagios nagios 0 Mar 9 13:25 1331270739.perfdata.service
-rw-rw-r-- 1 nagios nagios 241 Mar 9 13:25 1331270754.perfdata.host
-rw-rw-r-- 1 nagios nagios 0 Mar 9 13:25 1331270754.perfdata.service
After some digging around, I found the code that puts the files there.
Code: Select all
[root@nagiostest xidpe]# grep -rn xidpe /usr/local/nagios/
/usr/local/nagios/var/objects.cache:703: command_line /bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.host
/usr/local/nagios/var/objects.cache:718: command_line /bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.service
Does anyone know why these files are not being cleared?
Many Thanks
Kyle
Re: 200,000+ files created in xidpe directory
Posted: Fri Mar 09, 2012 2:26 am
by pouda
Hi
Nothing for me on this folder ...i think that you install something like a plugin which is writing on this folder continously
Re: 200,000+ files created in xidpe directory
Posted: Fri Mar 09, 2012 10:40 am
by mguthrie
I believe these temp files are created for the outbound transfer feature in XI, however, they're supposed to be cleared immediately after they're forwarded to another server. There could be a permissions issue preventing them from being deleted properly.
Are you using the Outbound Transfer feature in XI?
Can you clear the contents in that directory, and then once a few new files are in it, run:
Code: Select all
ll /usr/local/nagios/var/spool/xidpe
That directory should also have the following permissions:
Code: Select all
drwxr-xr-x 2 nagios nagios 503808 Mar 9 09:37 xidpe
Re: 200,000+ files created in xidpe directory
Posted: Mon Mar 12, 2012 1:07 am
by BWNetworks
I have checked the Global Options, NRDP, and NSCA tabs under "Outbound Check Transfer Settings" and they are all disabled.
Code: Select all
[root@nagiostest xidpe]# pwd
/usr/local/nagios/var/spool/xidpe
[root@nagiostest xidpe]# rm -f *
[root@nagiostest xidpe]# ll /usr/local/nagios/var/spool/xidpe
total 4
-rw-rw-r-- 1 nagios nagios 0 Mar 12 14:02 1331532174.perfdata.host
-rw-rw-r-- 1 nagios nagios 0 Mar 12 14:02 1331532174.perfdata.service
-rw-rw-r-- 1 nagios nagios 245 Mar 12 14:03 1331532189.perfdata.host
-rw-rw-r-- 1 nagios nagios 0 Mar 12 14:02 1331532189.perfdata.service
-rw-rw-r-- 1 nagios nagios 0 Mar 12 14:03 1331532204.perfdata.host
-rw-rw-r-- 1 nagios nagios 0 Mar 12 14:03 1331532204.perfdata.service
[root@nagiostest xidpe]# ll /usr/local/nagios/var/spool/
total 11480
drwxrwxr-x 2 nagios nagios 4096 Mar 12 14:03 checkresults
drwxr-xr-x 2 nagios nagios 4096 Feb 16 08:00 perfdata
drwxr-xr-x 2 nagios nagios 11743232 Mar 12 14:03 xidpe
Nagios XI is also telling me that I have exceeded my license limits, could that be the cause of something not running properly?
Code: Select all
License Type: Free - Purchase a license and receive support and upgrade benefits!
Number of Licensed Hosts: 7
Current Hosts: 9
License Usage: 129%
It's odd because I only have 6 hosts in Core Config Manager!
Thank you
Kind Regards
Kyle
Re: 200,000+ files created in xidpe directory
Posted: Mon Mar 12, 2012 10:54 am
by mguthrie
Can you check the Admin page in XI and look at the XI System Component Status and make sure that everything appears to be running correctly?
Re: 200,000+ files created in xidpe directory
Posted: Mon Mar 12, 2012 9:35 pm
by BWNetworks
Hmmm that may be it!
Database Maintenance [Last Run 26d 2h 36m 28s Ago]
Command Subsystem [Last Run 26d 2h 31m 28s Ago]
Event Manager [Last Run 26d 2h 31m 28s Ago]
Feed Processor [Last Run 26d 2h 31m 27s Ago]
Report Engine [Last Run 26d 2h 31m 28s Ago]
Cleaner [Last Run 26d 2h 32m 28s Ago]
Nonstop Operations Manager [Last Run 26d 2h 32m 28s Ago]
System Statistics [Last Updated 26d 2h 31m 42s Ago]
My guess is that this is due to the trial license running out.
Is there any way to fix Nagios XI to properly detect that I have only 6 devices configured, not the 9 as it suggests (I used to have 9).
Thanks
Kyle
Re: 200,000+ files created in xidpe directory
Posted: Tue Mar 13, 2012 11:24 am
by lmiltchev
Kyle,
I don't believe that your expired license is the issue here, but if you want to monitor 6 hosts, you can do the following:
Go to: Admin->System Config->License Information. Make sure your "License Type" is set as "Free". Click on the "Update Key".
Delete 3 out of 9 hosts, that you don't need (delete the services first!). Apply Configuration.
Now, you should be able to monitor 6 hosts.
Hope this helps.
Re: 200,000+ files created in xidpe directory
Posted: Wed Jun 20, 2012 1:45 pm
by kinnema
Hello,
I'm having the same issue - 500,000+ files using up all the inodes in /usr/local/nagios/var/spool/xidpe. Although, this is happening on three DNX nodes and not the master NagiosXI server. Permissions look fine for the directory. What is supposed to actually remove these files?
I only see the following references in commands.cfg to move perfdata into the directory, but don't see anything about removing the files:
define command {
command_name process-host-perfdata-file-bulk
command_line /bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.host
}
define command {
command_name process-service-perfdata-file-bulk
command_line /bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.service
}
I'm getting about 8 of these flies moved there per second, although it seems like it can't be keeping that up indefinitely since we only noticed the problem today and at that rate we'd hit well over 600,000 over a day.
Any ideas?
Re: 200,000+ files created in xidpe directory
Posted: Wed Jun 20, 2012 1:49 pm
by slansing
Hi kinnema, could you please re post this in a stand alone thread as to not clutter this one with two support issues? Thank you!