MRTG limitations

[vAJ]

So, most of you guys @ HQ know I've been pushing through this SNMPv3 load of all of my network gear in this setup with the new company.

Everything's been going OK, but I think I've hit the wall.

Noticed today that after adding the latest device, with lots of interfaces that MRTG seems to have fallen over. Running MRTG command from CLI results in:

Code: Select all

ERROR: unable to open include file: ...

But that config file had been in place for days and worked fine. Interface charts all looked good. So why only after adding this latest device?

OK. Dropped all cfg files for the latest device and re-ran MRTG from CLI. No errors.

Then I thought, what limitation am I hitting here? System load seems OK (not great, this VM is running warm), memory is fine... Then wondered how many cfg files I was including, 964, before the new device was added. The new device would've brought me over 1K. Could that be the problem? Is MRTG choking on too many included config files?

I can't find anything in MRTG docs or forum posts that suggest this. All told, I have about 170 network devices to load into NagiosXI. Figure close to a 2K interfaces...

Thoughts?

[tmcdonald]

Might be hitting a timeout trying to process them all. I know back in the day when they were all one big config, we would split it into two and put them on separate cron timings:

https://support.nagios.com/forum/viewto ... 654#p75223

Though I tend to agree with your findings, I am not familiar with any such limit.

Update: Actually, I am familiar with a limit:

http://www.cyberciti.biz/faq/linux-incr ... pen-files/

You likely are hitting a kernel file limit. Try doubling and see what it leaves you with.

Thanks @bheden for pointing that out.

[vAJ]

Code: Select all

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 47342
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 47342
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Yep... 1024 open file limit. This also explains why I'm getting occasional UNKNOWN status on bandwidth checks.

/var/lib/mrtg/10.10.0.101_12.rrd does not exist.

I've updated to 4k and we'll see how it handles it.

[ssax]

vAJ, if you have problems trying to adjust the limits and getting them to stick, it may be related to this (if you have it):

Code: Select all

/etc/security/limits.d/90-nproc.conf

Took me a while to find that...

[vAJ]

Nope, we're all good. Sticking. Added to /etc/secutity/limits.conf :

Code: Select all

* soft nofile 4096      #open files
* hard nofile 4096

Added a bunch more devices and interfaces. I think we can lock it up.