MRTG problems

[gfesj69]

Hi, I am running Nagios XI 2014R1.4, CentOS 6 64bits - VMWARE.

We are having problems to get bandwidth of some devices.

Some times MRTG can get the real traffic, but some times it return 0 bytes, but we have traffic at this links all the time.

How can we fix this?

[slansing]

Looks like it was running for a short period of time, what is the output of:

Code: Select all

LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg –check

How many services (ports/interfaces) are you monitoring with MRTG? What are the system's hardware resources?

[gfesj69]

Hi slansing, the command do not return anything, it appears has no errors.

Code: Select all

[root@SPAM-VM-NGS01 ~]# LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --check
[root@SPAM-VM-NGS01 ~]#

I do not know if is right, but we have 6235 Interfaces at cfgs files.

Code: Select all

 LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --debug=cfg --check > grep Interface | wc -l
6235

This server has 8 CPUS and 8G of RAM.

Code: Select all

top - 13:56:36 up 3 days, 14:42,  1 user,  load average: 2.25, 1.85, 1.72
Tasks: 290 total,   3 running, 287 sleeping,   0 stopped,   0 zombie
Cpu0  : 32.2%us,  5.4%sy,  0.0%ni, 60.1%id,  0.0%wa,  0.3%hi,  2.0%si,  0.0%st
Cpu1  : 30.8%us,  5.7%sy,  0.0%ni, 63.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  : 11.4%us,  3.7%sy,  0.0%ni, 84.2%id,  0.0%wa,  0.0%hi,  0.7%si,  0.0%st
Cpu3  : 33.7%us,  7.7%sy,  0.0%ni, 58.0%id,  0.7%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  8.4%us,  2.7%sy,  0.0%ni, 88.6%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu5  : 24.8%us,  5.0%sy,  0.0%ni, 69.9%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu6  : 16.7%us,  3.7%sy,  0.0%ni, 79.0%id,  0.0%wa,  0.0%hi,  0.7%si,  0.0%st
Cpu7  : 24.9%us,  5.3%sy,  0.0%ni, 69.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8053804k total,  6265400k used,  1788404k free,   162988k buffers
Swap:  2064376k total,    11828k used,  2052548k free,  5202868k cached

  PID  PPID    TIME+  %CPU %MEM  PR  NI S  VIRT SWAP  RES  UID COMMAND
65185  1707   0:00.01  0.0  0.0  20   0 S 79144    0 3380   89 pickup
64073  1621   0:10.95  0.0  0.1  20   0 S  211m  708 7876   26 postmaster
63353  9564   0:00.02  0.7  0.1  20   0 R  125m    0 4264  500 check_ifopersta
63352  9557   0:00.03  1.0  0.1  20   0 R  126m    0 5220  500 check_ifopersta
63343  9562   0:00.11  3.6  0.1  20   0 S  137m    0 9.9m  500 check_dell_open
63308  9552   0:00.00  0.0  0.0  20   0 S  106m    0  776  500 check_icmp
63282  9557   0:00.09  3.0  0.1  20   0 S  137m    0  10m  500 check_snmp_stor
63256  1767   0:00.00  0.0  0.0  20   0 S  134m  216 3024  500 mod_gearman_wor
63209  9564   0:00.00  0.0  0.0  20   0 S  106m    0  776  500 check_icmp
63128  1767   0:00.00  0.0  0.0  20   0 S  134m  216 3024  500 mod_gearman_wor

[sreinhardt]

Can you post your current /etc/mrtg/mrtg.cfg please? If you have that many interfaces, there are some optimizations that should be applied, that likely have not been yet.

[gfesj69]

Sure, here is the mrtg.cfg file.

Code: Select all

######################################################################
# Multi Router Traffic Grapher -- Example Configuration File
######################################################################
# This file is for use with mrtg-2.0
#
# Note:
#
# * Keywords must start at the begin of a line.
#
# * Lines which follow a keyword line which do start
#   with a blank are appended to the keyword line
#
# * Empty Lines are ignored
#
# * Lines starting with a # sign are comments.

# Where should the logfiles, and webpages be created?

# Minimal mrtg.cfg
#--------------------

HtmlDir: /var/www/mrtg
ImageDir: /var/www/mrtg
LogFormat: rrdtool
LogDir: /var/lib/mrtg
ThreshDir: /var/lib/mrtg
WorkDir: /var/lib/mrtg

Include: conf.d/*.cfg

Here is the ls from conf.d directory

Code: Select all

[root@SPAM-VM-NGS01 conf.d]# pwd
/etc/mrtg/conf.d
[root@SPAM-VM-NGS01 conf.d]# ls
10.202.41.186.cfg  10.244.208.91.cfg   10.248.0.83.cfg      192.168.210.5.cfg
10.202.41.187.cfg  10.244.209.145.cfg  10.248.0.84.cfg      192.168.211.124.cfg
10.241.0.175.cfg   10.244.209.149.cfg  10.248.0.85.cfg      192.168.211.126.cfg
10.241.0.22.cfg    10.244.210.3.cfg    10.248.0.86.cfg      192.168.211.12.cfg
10.241.0.24.cfg    10.246.0.2.cfg      10.248.0.87.cfg      192.168.211.132.cfg
10.241.0.251.cfg   10.246.0.3.cfg      10.248.0.88.cfg      192.168.211.133.cfg
10.241.0.2.cfg     10.246.128.2.cfg    10.248.0.89.cfg      192.168.211.134.cfg
10.241.0.35.cfg    10.246.128.3.cfg    10.248.0.90.cfg      192.168.211.149.cfg
10.241.0.3.cfg     10.246.160.3.cfg    10.248.0.91.cfg      192.168.211.227.cfg
10.241.0.70.cfg    10.246.209.145.cfg  10.248.0.92.cfg      192.168.211.228.cfg
10.241.0.71.cfg    10.246.209.149.cfg  10.248.160.3.cfg     192.168.211.86.cfg
10.241.0.72.cfg    10.246.209.57.cfg   10.248.160.4.cfg     192.168.212.148.cfg
10.241.0.73.cfg    10.246.209.58.cfg   10.248.208.20.cfg    192.168.212.4.cfg
10.241.0.74.cfg    10.248.0.21.cfg     10.248.208.21.cfg    192.168.212.5.cfg
10.241.0.84.cfg    10.248.0.2.cfg      10.248.210.3.cfg     192.168.255.225.cfg
10.241.27.72.cfg   10.248.0.30.cfg     10.248.210.4.cfg     192.168.255.241.cfg
10.241.90.11.cfg   10.248.0.32.cfg     10.96.204.41.cfg     192.168.99.124.cfg
10.242.64.253.cfg  10.248.0.33.cfg     10.96.204.45.cfg     192.168.99.148.cfg
10.243.0.2.cfg     10.248.0.34.cfg     10.96.204.50.cfg     192.168.99.149.cfg
10.243.0.3.cfg     10.248.0.35.cfg     10.96.204.51.cfg     192.168.99.18.cfg
10.244.0.2.cfg     10.248.0.3.cfg      172.28.0.182.cfg     192.168.99.228.cfg
10.244.0.60.cfg    10.248.0.46.cfg     172.28.0.183.cfg     192.168.99.237.cfg
10.244.0.61.cfg    10.248.0.50.cfg     172.28.0.184.cfg     192.168.99.238.cfg
10.244.0.88.cfg    10.248.0.51.cfg     172.28.0.185.cfg     192.168.99.33.cfg
10.244.208.20.cfg  10.248.0.52.cfg     172.29.0.2.cfg       192.168.99.41.cfg
10.244.208.21.cfg  10.248.0.53.cfg     172.29.0.3.cfg       192.168.99.49.cfg
10.244.208.44.cfg  10.248.0.80.cfg     186.225.160.132.cfg  192.168.99.57.cfg
10.244.208.45.cfg  10.248.0.81.cfg     186.225.164.132.cfg  192.168.99.98.cfg
10.244.208.53.cfg  10.248.0.82.cfg     189.125.142.225.cfg  192.168.99.99.cfg
[root@SPAM-VM-NGS01 conf.d]#

[sreinhardt]

That looks great for a stock set of configs, but for that many devices and interfaces, we definitely want to help this along a bit. Let's try the following:

Code: Select all

LANG=C LC_ALL=C /usr/bin/mrtg --logging /tmp/mrtg.log --debug="snpo,time" /etc/mrtg/mrtg.cfg
echo "Errors $(cat /tmp/mrtg.log | grep -i 'error' | wc -l)"
echo "Timeouts $(cat /tmp/mrtg.log | grep -i 'timeout' | wc -l)"

Add the following to the mrtg config just before the include line:

Code: Select all

Forks: 10

Rerun the mrtg command again, and lets see if we get less issues:

Code: Select all

LANG=C LC_ALL=C /usr/bin/mrtg --logging /tmp/mrtg-changed.log --debug="snpo,time" /etc/mrtg/mrtg.cfg
echo "Errors $(cat /tmp/mrtg-changed.log | grep -i 'error' | wc -l)"
echo "Timeouts $(cat /tmp/mrtg-changed.log | grep -i 'timeout' | wc -l)"

[gfesj69]

Thanks sreinhardt, here are the outputs.

Code: Select all

[root@SPAM-VM-NGS01 ~]# LANG=C LC_ALL=C /usr/bin/mrtg --logging /tmp/mrtg.log --debug="snpo,time" /etc/mrtg/mrtg.cfg
--time: prog start Wed Dec 10 07:49:46 2014
[root@SPAM-VM-NGS01 ~]# echo "Errors $(cat /tmp/mrtg.log | grep -i 'error' | wc -l)"
Errors 111
[root@SPAM-VM-NGS01 ~]# echo "Timeouts $(cat /tmp/mrtg.log | grep -i 'timeout' | wc -l)"
Timeouts 1
You have new mail in /var/spool/mail/root
[root@SPAM-VM-NGS01 ~]#

Here is the new mrtg.cfg file

Code: Select all

######################################################################
# Multi Router Traffic Grapher -- Example Configuration File
######################################################################
# This file is for use with mrtg-2.0
#
# Note:
#
# * Keywords must start at the begin of a line.
#
# * Lines which follow a keyword line which do start
#   with a blank are appended to the keyword line
#
# * Empty Lines are ignored
#
# * Lines starting with a # sign are comments.

# Where should the logfiles, and webpages be created?

# Minimal mrtg.cfg
#--------------------

HtmlDir: /var/www/mrtg
ImageDir: /var/www/mrtg
LogFormat: rrdtool
LogDir: /var/lib/mrtg
ThreshDir: /var/lib/mrtg
WorkDir: /var/lib/mrtg

Forks: 10
Include: conf.d/*.cfg

Here is the output after I changed the mrtg.cfg file.

Code: Select all

[root@SPAM-VM-NGS01 ~]# LANG=C LC_ALL=C /usr/bin/mrtg --logging /tmp/mrtg-changed.log --debug="snpo,time" /etc/mrtg/mrtg.cfg
--time: prog start Wed Dec 10 07:54:38 2014
[root@SPAM-VM-NGS01 ~]# echo "Errors $(cat /tmp/mrtg-changed.log | grep -i 'error' | wc -l)"
Errors 111
[root@SPAM-VM-NGS01 ~]# echo "Timeouts $(cat /tmp/mrtg-changed.log | grep -i 'timeout' | wc -l)"
Timeouts 1
You have new mail in /var/spool/mail/root
[root@SPAM-VM-NGS01 ~]#

[gfesj69]

I was analyzing the problem and saw something interesting. During the night MRTG can collect the information in this link .

[sreinhardt]

Could you run the mrtg command with logging and PM the resulting log file in /tmp to me please? It might contain community strings and definitely will have internal IPs so I'd prefer pm. I find it interesting that there were no differences between pre and post forking changes. On the link that received data over night, was anything else altered between then and now?