Page 1 of 6
missing bandwidth perf data for network devices
Posted: Thu Feb 11, 2016 8:40 am
by bosecorp
hen looking at bandwidth graph for our net devices we are missing data for about 5 hours
All the server check, like CPU, Memory, disk do not have the gap.
I reviewed this thread and follow some of the instructions but without luck.
https://support.nagios.com/wiki/index.p ... leshooting
https://support.nagios.com/forum/viewto ... 57#p150057
this start happening about a week ago, but it was happening before that but before it was not as frequent as it's today
You have new mail in /var/spool/mail/root
root@nagmonus1:(02-11 08:36): /usr/local/nagios/var
# tail -999 /usr/local/nagios/var/perfdata.log | grep TIMEOUT
root@nagmonus1:(02-11 08:36): /usr/local/nagios/var
# tail -999 /usr/local/nagios/var/npcd.log | grep "MAX load reached"
Re: missing bandwidth perf data for network devices
Posted: Thu Feb 11, 2016 11:45 am
by rkennedy
This could be a few different things, can you answer a few questions?
- To clarify, is it one individual service, or all network devices?
- During the 5 hours, were they still reporting an OK state?
- Are these active or passive checks?
Re: missing bandwidth perf data for network devices
Posted: Thu Feb 11, 2016 11:53 am
by bosecorp
- To clarify, is it one individual service, or all network devices? all network devices
- During the 5 hours, were they still reporting an OK state? Yes
- Are these active or passive checks? active
Re: missing bandwidth perf data for network devices
Posted: Thu Feb 11, 2016 4:32 pm
by tgriep
The Bandwidth graphs for Network devices are controlled by the cron daemon. Run the following to restart it to see if this fixes the issue.
Try that and let us know if this fixes it for you.
Re: missing bandwidth perf data for network devices
Posted: Thu Feb 11, 2016 4:33 pm
by bosecorp
it didn't, I tried that yesterday and it didn't work
Re: missing bandwidth perf data for network devices
Posted: Thu Feb 11, 2016 4:57 pm
by tgriep
Can you go Service Details from the Home screen, select one of the services that isn't updating, click on the Advanced tab, screen capture that and post that?
Another thing, can you login as root, run the following commands and post the output?
Code: Select all
LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok
ls -l /var/lib/mrtg/
ps -ef
Thanks
Re: missing bandwidth perf data for network devices
Posted: Sun Feb 21, 2016 5:58 pm
by bosecorp
the first command displays few SNMP errors
the second command is a big output, I see all the rdd files
the problem is that I don;t see graphs at all, sometimes I do, sometimes I don;t. I am attaching a screenshoot
Re: missing bandwidth perf data for network devices
Posted: Sun Feb 21, 2016 6:01 pm
by Box293
bosecorp wrote:the first command displays few SNMP errors
Can you post those errors please.
Re: missing bandwidth perf data for network devices
Posted: Sun Feb 21, 2016 6:16 pm
by bosecorp
# LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok
Code: Select all
SNMP Error:
no response received
SNMPv2c_Session (remote host: "192.253.174.10" [192.253.174.10].161)
community: "nc_BOSE"
request ID: 1046312840
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on [email protected]:::::2:v4only
at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifHCInOctets.1 on 192.253.174.10 did not succeed
2016-02-21 17:53:24: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 192.253.174.10 in this round.
SNMP Error:
no response received
SNMPv2c_Session (remote host: "usfm-rd-ti-a.bose.com" [192.103.0.16].161)
community: "mysnmpstring"
request ID: 1027561565
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on [email protected]:::::2:v4only
at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifHCInOctets.1 on usfm-rd-ti-a.bose.com did not succeed
2016-02-21 17:53:24: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host usfm-rd-ti-a.bose.com in this round.
SNMP Error:
no response received
SNMPv2c_Session (remote host: "192.192.71.164" [192.192.71.164].161)
community: "mysnmpstring"
request ID: 61609210
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifInOctets.2 ifOutOctets.2 on [email protected]:::::2:v4only
at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifInOctets.2 on 192.192.71.164 did not succeed
2016-02-21 17:53:24: WARNING: no data for ifInOctets&ifOutOctets:[email protected]. Skipping further queries for Host 192.192.71.164 in this round.
SNMP Error:
no response received
SNMPv2c_Session (remote host: "192.253.173.194" [192.253.173.194].161)
community: "mysnmpstring"
request ID: 1677143977
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on [email protected]:161::::2:v4only
at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifHCInOctets.1 on 192.253.173.194 did not succeed
2016-02-21 17:53:24: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 192.253.173.194 in this round.
SNMP Error:
no response received
SNMPv2c_Session (remote host: "192.1.68.33" [192.1.68.33].161)
community: "mysnmpstring"
request ID: 2027967774
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifInOctets.2 ifOutOctets.2 on [email protected]:::::2:v4only
at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifInOctets.2 on 192.1.68.33 did not succeed
2016-02-21 17:53:24: WARNING: no data for ifInOctets&ifOutOctets:[email protected]. Skipping further queries for Host 192.1.68.33 in this round.
SNMP Error:
no response received
SNMPv2c_Session (remote host: "192.253.138.202" [192.253.138.202].161)
community: "nc_BOSE"
request ID: 663507993
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on [email protected]:::::2:v4only
at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifHCInOctets.1 on 192.253.138.202 did not succeed
2016-02-21 17:53:24: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 192.253.138.202 in this round.
SNMP Error:
no response received
SNMPv2c_Session (remote host: "192.253.95.2" [192.253.95.2].161)
community: "mysnmpstring"
request ID: 364165223
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on [email protected]:::::2:v4only
at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifHCInOctets.1 on 192.253.95.2 did not succeed
2016-02-21 17:53:24: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 192.253.95.2 in this round.
SNMP Error:
n
Re: missing bandwidth perf data for network devices
Posted: Sun Feb 21, 2016 6:30 pm
by Box293
Let's delete these old config files as they appear to be for devices no longer monitored. They will be adding extra time to the checks while they timeout, which can lead to gaps in the graphs:
Code: Select all
rm -f /etc/mrtg/conf.d/192.253.174.10.cfg
rm -f /etc/mrtg/conf.d/usfm-rd-ti-a.bose.com.cfg
rm -f /etc/mrtg/conf.d/192.192.71.164.cfg
rm -f /etc/mrtg/conf.d/192.253.173.194.cfg
rm -f /etc/mrtg/conf.d/192.1.68.33.cfg
rm -f /etc/mrtg/conf.d/192.253.138.202.cfg
rm -f /etc/mrtg/conf.d/192.253.95.2.cfg
Of course if some of these are actually down and will be up again don't delete them.
After you've done that, run the command again to make sure there are no more errors.
Now we just need to wait to see if the graphs no longer have gaps.
Also, what is the output of:
Code: Select all
time LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok