200,000+ files created in xidpe directory

[BWNetworks]

Hi

I am currently running a Nagios XI test box while we wait for our license to be purchased. The server locked up as the file system run out of inodes.
Upon investigation, I found that /usr/local/nagios/var/spool/xidpe/ had over 200,000 files.

I cleared out the directory to restore service, however, it looks like it is filling up again. Example of these files are below:

Code: Select all

-rw-rw-r-- 1 nagios nagios      245 Mar  9 13:25 1331270724.perfdata.host
-rw-rw-r-- 1 nagios nagios        0 Mar  9 13:25 1331270724.perfdata.service
-rw-rw-r-- 1 nagios nagios        0 Mar  9 13:25 1331270739.perfdata.host
-rw-rw-r-- 1 nagios nagios        0 Mar  9 13:25 1331270739.perfdata.service
-rw-rw-r-- 1 nagios nagios      241 Mar  9 13:25 1331270754.perfdata.host
-rw-rw-r-- 1 nagios nagios        0 Mar  9 13:25 1331270754.perfdata.service

After some digging around, I found the code that puts the files there.

Code: Select all

[root@nagiostest xidpe]# grep -rn xidpe /usr/local/nagios/
/usr/local/nagios/var/objects.cache:703:        command_line    /bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.host
/usr/local/nagios/var/objects.cache:718:        command_line    /bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.service

Does anyone know why these files are not being cleared?

Many Thanks
Kyle

[pouda]

Hi

Nothing for me on this folder ...i think that you install something like a plugin which is writing on this folder continously

[mguthrie]

I believe these temp files are created for the outbound transfer feature in XI, however, they're supposed to be cleared immediately after they're forwarded to another server. There could be a permissions issue preventing them from being deleted properly.

Are you using the Outbound Transfer feature in XI?

Can you clear the contents in that directory, and then once a few new files are in it, run:

Code: Select all

ll /usr/local/nagios/var/spool/xidpe

That directory should also have the following permissions:

Code: Select all

drwxr-xr-x 2 nagios nagios 503808 Mar  9 09:37 xidpe

[BWNetworks]

I have checked the Global Options, NRDP, and NSCA tabs under "Outbound Check Transfer Settings" and they are all disabled.

Code: Select all

[root@nagiostest xidpe]# pwd
/usr/local/nagios/var/spool/xidpe

[root@nagiostest xidpe]# rm -f *

[root@nagiostest xidpe]# ll /usr/local/nagios/var/spool/xidpe
total 4
-rw-rw-r-- 1 nagios nagios   0 Mar 12 14:02 1331532174.perfdata.host
-rw-rw-r-- 1 nagios nagios   0 Mar 12 14:02 1331532174.perfdata.service
-rw-rw-r-- 1 nagios nagios 245 Mar 12 14:03 1331532189.perfdata.host
-rw-rw-r-- 1 nagios nagios   0 Mar 12 14:02 1331532189.perfdata.service
-rw-rw-r-- 1 nagios nagios   0 Mar 12 14:03 1331532204.perfdata.host
-rw-rw-r-- 1 nagios nagios   0 Mar 12 14:03 1331532204.perfdata.service

[root@nagiostest xidpe]# ll /usr/local/nagios/var/spool/
total 11480
drwxrwxr-x 2 nagios nagios     4096 Mar 12 14:03 checkresults
drwxr-xr-x 2 nagios nagios     4096 Feb 16 08:00 perfdata
drwxr-xr-x 2 nagios nagios 11743232 Mar 12 14:03 xidpe

Nagios XI is also telling me that I have exceeded my license limits, could that be the cause of something not running properly?

Code: Select all

License Type:	Free - Purchase a license and receive support and upgrade benefits!
Number of Licensed Hosts:	7
Current Hosts:	9
License Usage:	129%

It's odd because I only have 6 hosts in Core Config Manager!

Thank you
Kind Regards
Kyle

[mguthrie]

Can you check the Admin page in XI and look at the XI System Component Status and make sure that everything appears to be running correctly?

[BWNetworks]

Hmmm that may be it!

Database Maintenance [Last Run 26d 2h 36m 28s Ago]
Command Subsystem [Last Run 26d 2h 31m 28s Ago]
Event Manager [Last Run 26d 2h 31m 28s Ago]
Feed Processor [Last Run 26d 2h 31m 27s Ago]
Report Engine [Last Run 26d 2h 31m 28s Ago]
Cleaner [Last Run 26d 2h 32m 28s Ago]
Nonstop Operations Manager [Last Run 26d 2h 32m 28s Ago]
System Statistics [Last Updated 26d 2h 31m 42s Ago]

My guess is that this is due to the trial license running out.
Is there any way to fix Nagios XI to properly detect that I have only 6 devices configured, not the 9 as it suggests (I used to have 9).

Thanks
Kyle

[lmiltchev]

Kyle,

I don't believe that your expired license is the issue here, but if you want to monitor 6 hosts, you can do the following:

Go to: Admin->System Config->License Information. Make sure your "License Type" is set as "Free". Click on the "Update Key".
Delete 3 out of 9 hosts, that you don't need (delete the services first!). Apply Configuration.

Now, you should be able to monitor 6 hosts.

Hope this helps.

[kinnema]

Hello,

I'm having the same issue - 500,000+ files using up all the inodes in /usr/local/nagios/var/spool/xidpe. Although, this is happening on three DNX nodes and not the master NagiosXI server. Permissions look fine for the directory. What is supposed to actually remove these files?

I only see the following references in commands.cfg to move perfdata into the directory, but don't see anything about removing the files:

define command {
command_name process-host-perfdata-file-bulk
command_line /bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.host
}

define command {
command_name process-service-perfdata-file-bulk
command_line /bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.service
}

I'm getting about 8 of these flies moved there per second, although it seems like it can't be keeping that up indefinitely since we only noticed the problem today and at that rate we'd hit well over 600,000 over a day.

Any ideas?

[slansing]

Hi kinnema, could you please re post this in a stand alone thread as to not clutter this one with two support issues? Thank you!