/var/spool/snmptt - many files

[gravitas]

We are running Nagiox 2014R2.6. The lvm_root filed up. it ran out of space, but also inodes. df -i showed 0 inodes available even after I deleted some files.

Eventually, I moved some files and extended lvm_root.


The problem seems to stem from the 1.6 million files in /var/spool/snmptt. It has a size of 7GB, but why so many files. There has to be something wrong. It just keeps spitting out files,,,

-rw-r--r-- 1 root root 359 Apr 1 15:48 #snmptt-trap-1427903338760530
-rw-r--r-- 1 root root 357 Apr 1 15:48 #snmptt-trap-1427903339067666
-rw-r--r-- 1 root root 362 Apr 1 15:48 #snmptt-trap-1427903339216766
-rw-r--r-- 1 root root 362 Apr 1 15:48 #snmptt-trap-1427903339302945
-rw-r--r-- 1 root root 357 Apr 1 15:48 #snmptt-trap-1427903339403718
-rw-r--r-- 1 root root 361 Apr 1 15:48 #snmptt-trap-1427903339565997
-rw-r--r-- 1 root root 362 Apr 1 15:49 #snmptt-trap-1427903346306110
-rw-r--r-- 1 root root 362 Apr 1 15:49 #snmptt-trap-1427903346393499
-rw-r--r-- 1 root root 359 Apr 1 15:49 #snmptt-trap-1427903346775887
-rw-r--r-- 1 root root 240 Apr 1 15:49 #snmptt-trap-1427903347585478
-rw-r--r-- 1 root root 240 Apr 1 15:49 #snmptt-trap-1427903347681689
-rw-r--r-- 1 root root 240 Apr 1 15:49 #snmptt-trap-1427903347784686
-rw-r--r-- 1 root root 240 Apr 1 15:49 #snmptt-trap-1427903347887742
-rw-r--r-- 1 root root 240 Apr 1 15:49 #snmptt-trap-1427903347981256
-rw-r--r-- 1 root root 285 Apr 1 15:49 #snmptt-trap-1427903348088280
-rw-r--r-- 1 root root 285 Apr 1 15:49 #snmptt-trap-1427903348202722
-rw-r--r-- 1 root root 285 Apr 1 15:49 #snmptt-trap-1427903348295305
-rw-r--r-- 1 root root 285 Apr 1 15:49 #snmptt-trap-1427903348392888
-rw-r--r-- 1 root root 285 Apr 1 15:49 #snmptt-trap-1427903348474349
-rw-r--r-- 1 root root 360 Apr 1 15:49 #snmptt-trap-1427903348852597
-rw-r--r-- 1 root root 359 Apr 1 15:49 #snmptt-trap-1427903349356114
-rw-r--r-- 1 root root 359 Apr 1 15:49 #snmptt-trap-1427903359250641
-rw-r--r-- 1 root root 362 Apr 1 15:49 #snmptt-trap-1427903366170041
-rw-r--r-- 1 root root 359 Apr 1 15:49 #snmptt-trap-1427903369116686
-rw-r--r-- 1 root root 359 Apr 1 15:49 #snmptt-trap-1427903369439603
[root@nagios snmptt]#


Does anybody have a suggestion?

[tmcdonald]

The first thing is those trap files will need to be removed. With that many in the directory most processes can't stat() them so they do not get processed. That also means that a regular rm will not work, you need to use this:

Code: Select all

find /var/spool/snmptt -type f -delete

This command can take a while to complete. Then make sure that snmptt is running:

Code: Select all

service snmptt status
service snmptt restart

Please show the output of those commands.

[jolson]

It looks like you are receiving traps that are not being processed.

The daemon snmptrapd writes the files to the spool directory. (/var/spool/snmptt)

If you have no need for these files, you could stop the snmptrapd service and delete them.

Code: Select all

service snmptrapd stop

Code: Select all

find /var/spool/snmptt/. -name snmptt-trap-* -delete

Keep in mind that this will delete all of the trap data in that directory - the command may take a while to run.

There may be a problem with the snmptt daemon, since your traps are not being processed fast enough.
The snmptt daemon picks up the files from the spool directory and processes them.

Is The snmptt Service Running?

Code: Select all

service snmptt status

You could also check on the following log files:

Code: Select all

tail /var/log/snmptt/snmpttsystem.log
tail /var/log/snmptt/snmptt.log
tail /var/log/snmptt/snmpttunknown.log

Thanks - let me know if the above helps!

[gravitas]

Yes, it was hard to delete then all. I just removed the directory and recreated it with proper permission.

Then, I started the snmptt service again. After watching the directory "watch -d -n 1 ls -l". It is processing them now. thx