Bandwidth graphs not updating

[akrradford]

During November maintenance, I patched the Nagios app and OS. We are on CentOS 6 and running XI version 5.5.8.

I have tried a few different things but nothing seems to solve the issue. I did notice when I run top, MRTG runs 100%, I also see a lot of messages that the process was killed due to lack of memory:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
82308 apache 20 0 2891m 2.2g 312 R 100.0 28.3 21:15.42 mrtg
82309 apache 20 0 2942m 2.2g 312 R 98.5 28.3 21:35.26 mrtg
82310 apache 20 0 2846m 2.2g 312 R 98.5 28.2 21:05.12 mrtg

Jan 31 11:18:55 ops-nagiosxi kernel: [63950] 500 63950 28265 195 12 0 0 mrtg
Jan 31 11:18:55 ops-nagiosxi kernel: [64128] 500 64128 1159218 919880 5 0 0 mrtg
Jan 31 11:18:55 ops-nagiosxi kernel: [64130] 500 64130 1126569 905766 0 0 0 mrtg
Jan 31 11:18:55 ops-nagiosxi kernel: Out of memory: Kill process 64128 (mrtg) score 443 or sacrifice child
Jan 31 11:18:55 ops-nagiosxi kernel: Killed process 64128, UID 500, (mrtg) total-vm:4636872kB, anon-rss:3679356kB, file-rss:164kB
Jan 31 11:26:50 ops-nagiosxi kernel: mrtg invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jan 31 11:26:50 ops-nagiosxi kernel: mrtg cpuset=/ mems_allowed=0
Jan 31 11:26:50 ops-nagiosxi kernel: Pid: 3302, comm: mrtg Not tainted 2.6.32-754.10.1.el6.x86_64 #1
Jan 31 11:26:50 ops-nagiosxi kernel: [63950] 500 63950 28265 195 12 0 0 mrtg
Jan 31 11:26:50 ops-nagiosxi kernel: [64130] 500 64130 1394971 1019095 2 0 0 mrtg
Jan 31 11:26:50 ops-nagiosxi kernel: [ 3134] 500 3134 28265 5430 0 0 0 mrtg
Jan 31 11:26:50 ops-nagiosxi kernel: [ 3301] 500 3301 235844 210455 11 0 0 mrtg
Jan 31 11:26:50 ops-nagiosxi kernel: [ 3302] 500 3302 251404 226015 4 0 0 mrtg
Jan 31 11:26:50 ops-nagiosxi kernel: [ 3303] 500 3303 229144 203755 5 0 0 mrtg
Jan 31 11:26:50 ops-nagiosxi kernel: [ 3304] 500 3304 256070 231717 13 0 0 mrtg
Jan 31 11:26:50 ops-nagiosxi kernel: Out of memory: Kill process 64130 (mrtg) score 536 or sacrifice child
Jan 31 11:26:50 ops-nagiosxi kernel: Killed process 64130, UID 500, (mrtg) total-vm:5579884kB, anon-rss:4076124kB, file-rss:236kB

Any ideas?

[scottwilkerson]

Can I have you run the following commands

Code: Select all

chown apache:nagios /etc/mrtg -R
chmod 775 /etc/mrtg -R
chown apache:nagios /var/lib/mrtg -R
chmod 775 /var/lib/mrtg -R
killall -9 mrtg
rm -f /var/lib/mrtg/mrtg.lock

then, I would also go to Admin -> Manage Config Wizards
Click check for updates
and update the Switch/Router wizard if an update is available

[akrradford]

Thanks for the tips Scott. I had run the chown/chmod commands based on another post, but I didn't kill processes or remove the lock file. We will see what the next 15 mins brings.

I also updated the wizard, thanks for the tip.

Ryan

[akrradford]

I see now that the files in /var/lib/mrtg are updating (their previous date was Nov 20), but the graphs are lagging behind.

I have entered the commands you recommended, its been about 20 minutes and still showing Current BW 0.

Any other ideas?

Thanks

Ryan

[scottwilkerson]

There can be about a 20 minute delay to get first graphs, are they still blank?

[akrradford]

Yes, a day later we still have no bandwidth updating.

[scottwilkerson]

Can I have you run the following from the command line

Code: Select all

LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --confcache-file /var/lib/mrtg/mrtg.ok --user=nagios --group=nagios > /tmp/mrtglog 2>&1

and then send the result in /tmp/mrtglog

[akrradford]

SNMPopen failed: No response from remote host '192.168.100.201' during discovery
at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 1580
Net_SNMP_util::snmpopen('[email protected]:::::3:v4only', 0, 'ARRAY(0x1d20c58)') called at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 565
Net_SNMP_util::snmpget('[email protected]:::::3:v4only', 'HASH(0x1f47fc8)', 'ifHCInOctets.2', 'ifHCOutOctets.2') called at /usr/bin/mrtg line 2330
main::getsnmparg('HASH(0x13c0300)', 'HASH(0x1f4cbf0)', 'HASH(0x12b7188)', 'HASH(0x1750be8)') called at /usr/bin/mrtg line 2510
main::readtargets('HASH(0x13c0300)', 'ARRAY(0x15f6ae0)', 'HASH(0x12b7188)') called at /usr/bin/mrtg line 403
main::main called at /usr/bin/mrtg line 143
SNMPGET Problem for [email protected]:::::3:v4only at /usr/bin/mrtg line 2330
2019-02-04 09:53:41: WARNING: skipping because at least the query for ifHCInOctets.2 on 192.168.100.201 did not succeed
2019-02-04 09:53:41: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 192.168.100.201 in this round.
SNMPopen failed: No response from remote host '192.168.100.213' during discovery
at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 1580
Net_SNMP_util::snmpopen('[email protected]:::::3:v4only', 0, 'ARRAY(0x1d20c58)') called at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 565
Net_SNMP_util::snmpget('[email protected]:::::3:v4only', 'HASH(0x1ece240)', 'ifHCInOctets.2', 'ifHCOutOctets.2') called at /usr/bin/mrtg line 2330
main::getsnmparg('HASH(0x13c0300)', 'HASH(0x1ed2e78)', 'HASH(0x12b7188)', 'HASH(0x1750be8)') called at /usr/bin/mrtg line 2510
main::readtargets('HASH(0x13c0300)', 'ARRAY(0x15f6ae0)', 'HASH(0x12b7188)') called at /usr/bin/mrtg line 403
main::main called at /usr/bin/mrtg line 143
SNMPGET Problem for [email protected]:::::3:v4only at /usr/bin/mrtg line 2330
2019-02-04 09:53:41: WARNING: skipping because at least the query for ifHCInOctets.2 on 192.168.100.213 did not succeed
2019-02-04 09:53:41: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 192.168.100.213 in this round.
SNMPopen failed: No response from remote host '192.168.20.7' during discovery
at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 1580
Net_SNMP_util::snmpopen('[email protected]:::::3:v4only', 0, 'ARRAY(0x1d20c58)') called at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 565
Net_SNMP_util::snmpget('[email protected]:::::3:v4only', 'HASH(0x17f6228)', 'ifHCInOctets.11', 'ifHCOutOctets.11') called at /usr/bin/mrtg line 2330
main::getsnmparg('HASH(0x13c0300)', 'HASH(0x17e7178)', 'HASH(0x12b7188)', 'HASH(0x1750be8)') called at /usr/bin/mrtg line 2510
main::readtargets('HASH(0x13c0300)', 'ARRAY(0x15f6ae0)', 'HASH(0x12b7188)') called at /usr/bin/mrtg line 403
main::main called at /usr/bin/mrtg line 143
SNMPGET Problem for [email protected]:::::3:v4only at /usr/bin/mrtg line 2330
2019-02-04 09:53:41: WARNING: skipping because at least the query for ifHCInOctets.11 on 192.168.20.7 did not succeed
2019-02-04 09:53:41: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 192.168.20.7 in this round.
SNMPopen failed: No response from remote host '192.168.100.223' during discovery
at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 1580
Net_SNMP_util::snmpopen('[email protected]:::::3:v4only', 0, 'ARRAY(0x1d20c58)') called at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 565
Net_SNMP_util::snmpget('[email protected]:::::3:v4only', 'HASH(0x1e441d8)', 'ifInOctets.50', 'ifOutOctets.50') called at /usr/bin/mrtg line 2330
main::getsnmparg('HASH(0x13c0300)', 'HASH(0x1e48070)', 'HASH(0x12b7188)', 'HASH(0x1750be8)') called at /usr/bin/mrtg line 2510
main::readtargets('HASH(0x13c0300)', 'ARRAY(0x15f6ae0)', 'HASH(0x12b7188)') called at /usr/bin/mrtg line 403
main::main called at /usr/bin/mrtg line 143
SNMPGET Problem for [email protected]:::::3:v4only at /usr/bin/mrtg line 2330
2019-02-04 09:53:41: WARNING: skipping because at least the query for ifInOctets.50 on 192.168.100.223 did not succeed
2019-02-04 09:53:41: WARNING: no data for ifInOctets&ifOutOctets:[email protected]. Skipping further queries for Host 192.168.100.223 in this round.
SNMPopen failed: No response from remote host '172.16.11.203' during discovery
at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 1580
Net_SNMP_util::snmpopen('[email protected]:::::3:v4only', 0, 'ARRAY(0x1d20c58)') called at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 565
Net_SNMP_util::snmpget('[email protected]:::::3:v4only', 'HASH(0x201c5d8)', 'ifHCInOctets.14', 'ifHCOutOctets.14') called at /usr/bin/mrtg line 2330
main::getsnmparg('HASH(0x13c0300)', 'HASH(0x2021430)', 'HASH(0x12b7188)', 'HASH(0x1750be8)') called at /usr/bin/mrtg line 2510
main::readtargets('HASH(0x13c0300)', 'ARRAY(0x15f6ae0)', 'HASH(0x12b7188)') called at /usr/bin/mrtg line 403
main::main called at /usr/bin/mrtg line 143
SNMPopen failed: No response from remote host '192.168.100.217' during discovery
at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 1580
Net_SNMP_util::snmpopen('[email protected]:::::3:v4only', 0, 'ARRAY(0x1d20c58)') called at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 565
Net_SNMP_util::snmpget('[email protected]:::::3:v4only', 'HASH(0x1e82218)', 'ifHCInOctets.1', 'ifHCOutOctets.1') called at /usr/bin/mrtg line 2330
main::getsnmparg('HASH(0x13c0300)', 'HASH(0x1e85ea0)', 'HASH(0x12b7188)', 'HASH(0x1750be8)') called at /usr/bin/mrtg line 2510
main::readtargets('HASH(0x13c0300)', 'ARRAY(0x15f6ae0)', 'HASH(0x12b7188)') called at /usr/bin/mrtg line 403
main::main called at /usr/bin/mrtg line 143

[akrradford]

Truncated because the forum thought it had "too many URLs"

[scottwilkerson]

you have a quite a few line like the following

Code: Select all

SNMPopen failed: No response from remote host '172.16.11.203' during discovery

If any these hosts are no longer active I would recommend removing them from being polled via MRTG by removing their .cfg file from /etc/mrtg/conf.d

for example,

Code: Select all

rm -f /etc/mrtg/conf.d/172.16.11.203.cfg