Page 1 of 1
MRTG problems
Posted: Tue Dec 09, 2014 10:30 am
by gfesj69
Hi, I am running Nagios XI 2014R1.4, CentOS 6 64bits - VMWARE.
We are having problems to get bandwidth of some devices.
Some times MRTG can get the real traffic, but some times it return 0 bytes, but we have traffic at this links all the time.
How can we fix this?
Re: MRTG problems
Posted: Tue Dec 09, 2014 10:34 am
by slansing
Looks like it was running for a short period of time, what is the output of:
Code: Select all
LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg –check
How many services (ports/interfaces) are you monitoring with MRTG? What are the system's hardware resources?
Re: MRTG problems
Posted: Tue Dec 09, 2014 11:10 am
by gfesj69
Hi slansing, the command do not return anything, it appears has no errors.
Code: Select all
[root@SPAM-VM-NGS01 ~]# LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --check
[root@SPAM-VM-NGS01 ~]#
I do not know if is right, but we have 6235 Interfaces at cfgs files.
Code: Select all
LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --debug=cfg --check > grep Interface | wc -l
6235
This server has 8 CPUS and 8G of RAM.
Code: Select all
top - 13:56:36 up 3 days, 14:42, 1 user, load average: 2.25, 1.85, 1.72
Tasks: 290 total, 3 running, 287 sleeping, 0 stopped, 0 zombie
Cpu0 : 32.2%us, 5.4%sy, 0.0%ni, 60.1%id, 0.0%wa, 0.3%hi, 2.0%si, 0.0%st
Cpu1 : 30.8%us, 5.7%sy, 0.0%ni, 63.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 11.4%us, 3.7%sy, 0.0%ni, 84.2%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st
Cpu3 : 33.7%us, 7.7%sy, 0.0%ni, 58.0%id, 0.7%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 8.4%us, 2.7%sy, 0.0%ni, 88.6%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu5 : 24.8%us, 5.0%sy, 0.0%ni, 69.9%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu6 : 16.7%us, 3.7%sy, 0.0%ni, 79.0%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st
Cpu7 : 24.9%us, 5.3%sy, 0.0%ni, 69.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8053804k total, 6265400k used, 1788404k free, 162988k buffers
Swap: 2064376k total, 11828k used, 2052548k free, 5202868k cached
PID PPID TIME+ %CPU %MEM PR NI S VIRT SWAP RES UID COMMAND
65185 1707 0:00.01 0.0 0.0 20 0 S 79144 0 3380 89 pickup
64073 1621 0:10.95 0.0 0.1 20 0 S 211m 708 7876 26 postmaster
63353 9564 0:00.02 0.7 0.1 20 0 R 125m 0 4264 500 check_ifopersta
63352 9557 0:00.03 1.0 0.1 20 0 R 126m 0 5220 500 check_ifopersta
63343 9562 0:00.11 3.6 0.1 20 0 S 137m 0 9.9m 500 check_dell_open
63308 9552 0:00.00 0.0 0.0 20 0 S 106m 0 776 500 check_icmp
63282 9557 0:00.09 3.0 0.1 20 0 S 137m 0 10m 500 check_snmp_stor
63256 1767 0:00.00 0.0 0.0 20 0 S 134m 216 3024 500 mod_gearman_wor
63209 9564 0:00.00 0.0 0.0 20 0 S 106m 0 776 500 check_icmp
63128 1767 0:00.00 0.0 0.0 20 0 S 134m 216 3024 500 mod_gearman_wor
Re: MRTG problems
Posted: Tue Dec 09, 2014 11:51 am
by sreinhardt
Can you post your current /etc/mrtg/mrtg.cfg please? If you have that many interfaces, there are some optimizations that should be applied, that likely have not been yet.
Re: MRTG problems
Posted: Tue Dec 09, 2014 12:39 pm
by gfesj69
Sure, here is the mrtg.cfg file.
Code: Select all
######################################################################
# Multi Router Traffic Grapher -- Example Configuration File
######################################################################
# This file is for use with mrtg-2.0
#
# Note:
#
# * Keywords must start at the begin of a line.
#
# * Lines which follow a keyword line which do start
# with a blank are appended to the keyword line
#
# * Empty Lines are ignored
#
# * Lines starting with a # sign are comments.
# Where should the logfiles, and webpages be created?
# Minimal mrtg.cfg
#--------------------
HtmlDir: /var/www/mrtg
ImageDir: /var/www/mrtg
LogFormat: rrdtool
LogDir: /var/lib/mrtg
ThreshDir: /var/lib/mrtg
WorkDir: /var/lib/mrtg
Include: conf.d/*.cfg
Here is the ls from conf.d directory
Code: Select all
[root@SPAM-VM-NGS01 conf.d]# pwd
/etc/mrtg/conf.d
[root@SPAM-VM-NGS01 conf.d]# ls
10.202.41.186.cfg 10.244.208.91.cfg 10.248.0.83.cfg 192.168.210.5.cfg
10.202.41.187.cfg 10.244.209.145.cfg 10.248.0.84.cfg 192.168.211.124.cfg
10.241.0.175.cfg 10.244.209.149.cfg 10.248.0.85.cfg 192.168.211.126.cfg
10.241.0.22.cfg 10.244.210.3.cfg 10.248.0.86.cfg 192.168.211.12.cfg
10.241.0.24.cfg 10.246.0.2.cfg 10.248.0.87.cfg 192.168.211.132.cfg
10.241.0.251.cfg 10.246.0.3.cfg 10.248.0.88.cfg 192.168.211.133.cfg
10.241.0.2.cfg 10.246.128.2.cfg 10.248.0.89.cfg 192.168.211.134.cfg
10.241.0.35.cfg 10.246.128.3.cfg 10.248.0.90.cfg 192.168.211.149.cfg
10.241.0.3.cfg 10.246.160.3.cfg 10.248.0.91.cfg 192.168.211.227.cfg
10.241.0.70.cfg 10.246.209.145.cfg 10.248.0.92.cfg 192.168.211.228.cfg
10.241.0.71.cfg 10.246.209.149.cfg 10.248.160.3.cfg 192.168.211.86.cfg
10.241.0.72.cfg 10.246.209.57.cfg 10.248.160.4.cfg 192.168.212.148.cfg
10.241.0.73.cfg 10.246.209.58.cfg 10.248.208.20.cfg 192.168.212.4.cfg
10.241.0.74.cfg 10.248.0.21.cfg 10.248.208.21.cfg 192.168.212.5.cfg
10.241.0.84.cfg 10.248.0.2.cfg 10.248.210.3.cfg 192.168.255.225.cfg
10.241.27.72.cfg 10.248.0.30.cfg 10.248.210.4.cfg 192.168.255.241.cfg
10.241.90.11.cfg 10.248.0.32.cfg 10.96.204.41.cfg 192.168.99.124.cfg
10.242.64.253.cfg 10.248.0.33.cfg 10.96.204.45.cfg 192.168.99.148.cfg
10.243.0.2.cfg 10.248.0.34.cfg 10.96.204.50.cfg 192.168.99.149.cfg
10.243.0.3.cfg 10.248.0.35.cfg 10.96.204.51.cfg 192.168.99.18.cfg
10.244.0.2.cfg 10.248.0.3.cfg 172.28.0.182.cfg 192.168.99.228.cfg
10.244.0.60.cfg 10.248.0.46.cfg 172.28.0.183.cfg 192.168.99.237.cfg
10.244.0.61.cfg 10.248.0.50.cfg 172.28.0.184.cfg 192.168.99.238.cfg
10.244.0.88.cfg 10.248.0.51.cfg 172.28.0.185.cfg 192.168.99.33.cfg
10.244.208.20.cfg 10.248.0.52.cfg 172.29.0.2.cfg 192.168.99.41.cfg
10.244.208.21.cfg 10.248.0.53.cfg 172.29.0.3.cfg 192.168.99.49.cfg
10.244.208.44.cfg 10.248.0.80.cfg 186.225.160.132.cfg 192.168.99.57.cfg
10.244.208.45.cfg 10.248.0.81.cfg 186.225.164.132.cfg 192.168.99.98.cfg
10.244.208.53.cfg 10.248.0.82.cfg 189.125.142.225.cfg 192.168.99.99.cfg
[root@SPAM-VM-NGS01 conf.d]#
Re: MRTG problems
Posted: Tue Dec 09, 2014 5:52 pm
by sreinhardt
That looks great for a stock set of configs, but for that many devices and interfaces, we definitely want to help this along a bit. Let's try the following:
Code: Select all
LANG=C LC_ALL=C /usr/bin/mrtg --logging /tmp/mrtg.log --debug="snpo,time" /etc/mrtg/mrtg.cfg
echo "Errors $(cat /tmp/mrtg.log | grep -i 'error' | wc -l)"
echo "Timeouts $(cat /tmp/mrtg.log | grep -i 'timeout' | wc -l)"
Add the following to the mrtg config just before the include line:
Rerun the mrtg command again, and lets see if we get less issues:
Code: Select all
LANG=C LC_ALL=C /usr/bin/mrtg --logging /tmp/mrtg-changed.log --debug="snpo,time" /etc/mrtg/mrtg.cfg
echo "Errors $(cat /tmp/mrtg-changed.log | grep -i 'error' | wc -l)"
echo "Timeouts $(cat /tmp/mrtg-changed.log | grep -i 'timeout' | wc -l)"
Re: MRTG problems
Posted: Wed Dec 10, 2014 5:02 am
by gfesj69
Thanks sreinhardt, here are the outputs.
Code: Select all
[root@SPAM-VM-NGS01 ~]# LANG=C LC_ALL=C /usr/bin/mrtg --logging /tmp/mrtg.log --debug="snpo,time" /etc/mrtg/mrtg.cfg
--time: prog start Wed Dec 10 07:49:46 2014
[root@SPAM-VM-NGS01 ~]# echo "Errors $(cat /tmp/mrtg.log | grep -i 'error' | wc -l)"
Errors 111
[root@SPAM-VM-NGS01 ~]# echo "Timeouts $(cat /tmp/mrtg.log | grep -i 'timeout' | wc -l)"
Timeouts 1
You have new mail in /var/spool/mail/root
[root@SPAM-VM-NGS01 ~]#
Here is the new mrtg.cfg file
Code: Select all
######################################################################
# Multi Router Traffic Grapher -- Example Configuration File
######################################################################
# This file is for use with mrtg-2.0
#
# Note:
#
# * Keywords must start at the begin of a line.
#
# * Lines which follow a keyword line which do start
# with a blank are appended to the keyword line
#
# * Empty Lines are ignored
#
# * Lines starting with a # sign are comments.
# Where should the logfiles, and webpages be created?
# Minimal mrtg.cfg
#--------------------
HtmlDir: /var/www/mrtg
ImageDir: /var/www/mrtg
LogFormat: rrdtool
LogDir: /var/lib/mrtg
ThreshDir: /var/lib/mrtg
WorkDir: /var/lib/mrtg
Forks: 10
Include: conf.d/*.cfg
Here is the output after I changed the mrtg.cfg file.
Code: Select all
[root@SPAM-VM-NGS01 ~]# LANG=C LC_ALL=C /usr/bin/mrtg --logging /tmp/mrtg-changed.log --debug="snpo,time" /etc/mrtg/mrtg.cfg
--time: prog start Wed Dec 10 07:54:38 2014
[root@SPAM-VM-NGS01 ~]# echo "Errors $(cat /tmp/mrtg-changed.log | grep -i 'error' | wc -l)"
Errors 111
[root@SPAM-VM-NGS01 ~]# echo "Timeouts $(cat /tmp/mrtg-changed.log | grep -i 'timeout' | wc -l)"
Timeouts 1
You have new mail in /var/spool/mail/root
[root@SPAM-VM-NGS01 ~]#
Re: MRTG problems
Posted: Wed Dec 10, 2014 11:35 am
by gfesj69
I was analyzing the problem and saw something interesting. During the night MRTG can collect the information in this link .
Re: MRTG problems
Posted: Wed Dec 10, 2014 2:09 pm
by sreinhardt
Could you run the mrtg command with logging and PM the resulting log file in /tmp to me please? It might contain community strings and definitely will have internal IPs so I'd prefer pm. I find it interesting that there were no differences between pre and post forking changes. On the link that received data over night, was anything else altered between then and now?