missing bandwidth perf data for network devices

[bosecorp]

hen looking at bandwidth graph for our net devices we are missing data for about 5 hours

All the server check, like CPU, Memory, disk do not have the gap.

I reviewed this thread and follow some of the instructions but without luck.

https://support.nagios.com/wiki/index.p ... leshooting

https://support.nagios.com/forum/viewto ... 57#p150057

this start happening about a week ago, but it was happening before that but before it was not as frequent as it's today


You have new mail in /var/spool/mail/root
root@nagmonus1:(02-11 08:36): /usr/local/nagios/var
# tail -999 /usr/local/nagios/var/perfdata.log | grep TIMEOUT
root@nagmonus1:(02-11 08:36): /usr/local/nagios/var
# tail -999 /usr/local/nagios/var/npcd.log | grep "MAX load reached"

[rkennedy]

This could be a few different things, can you answer a few questions?
- To clarify, is it one individual service, or all network devices?
- During the 5 hours, were they still reporting an OK state?
- Are these active or passive checks?

[bosecorp]

- To clarify, is it one individual service, or all network devices? all network devices
- During the 5 hours, were they still reporting an OK state? Yes
- Are these active or passive checks? active

[tgriep]

The Bandwidth graphs for Network devices are controlled by the cron daemon. Run the following to restart it to see if this fixes the issue.

Code: Select all

service crond restart

Try that and let us know if this fixes it for you.

[bosecorp]

it didn't, I tried that yesterday and it didn't work

[tgriep]

Can you go Service Details from the Home screen, select one of the services that isn't updating, click on the Advanced tab, screen capture that and post that?
Another thing, can you login as root, run the following commands and post the output?

Code: Select all

LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok
ls -l  /var/lib/mrtg/
ps -ef

Thanks

[bosecorp]

the first command displays few SNMP errors

the second command is a big output, I see all the rdd files

the problem is that I don;t see graphs at all, sometimes I do, sometimes I don;t. I am attaching a screenshoot

[Box293]

the first command displays few SNMP errors

Can you post those errors please.

[bosecorp]

# LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok

Code: Select all

SNMP Error:
no response received
SNMPv2c_Session (remote host: "192.253.174.10" [192.253.174.10].161)
                   community: "nc_BOSE"
                  request ID: 1046312840
                 PDU bufsize: 8000 bytes
                     timeout: 2s
                     retries: 5
                     backoff: 1)
 at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on [email protected]:::::2:v4only
 at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifHCInOctets.1 on  192.253.174.10 did not succeed
2016-02-21 17:53:24: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 192.253.174.10 in this round.
SNMP Error:
no response received
SNMPv2c_Session (remote host: "usfm-rd-ti-a.bose.com" [192.103.0.16].161)
                   community: "mysnmpstring"
                  request ID: 1027561565
                 PDU bufsize: 8000 bytes
                     timeout: 2s
                     retries: 5
                     backoff: 1)
 at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on [email protected]:::::2:v4only
 at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifHCInOctets.1 on  usfm-rd-ti-a.bose.com did not succeed
2016-02-21 17:53:24: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host usfm-rd-ti-a.bose.com in this round.
SNMP Error:
no response received
SNMPv2c_Session (remote host: "192.192.71.164" [192.192.71.164].161)
                   community: "mysnmpstring"
                  request ID: 61609210
                 PDU bufsize: 8000 bytes
                     timeout: 2s
                     retries: 5
                     backoff: 1)
 at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifInOctets.2 ifOutOctets.2 on [email protected]:::::2:v4only
 at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifInOctets.2 on  192.192.71.164 did not succeed
2016-02-21 17:53:24: WARNING: no data for ifInOctets&ifOutOctets:[email protected]. Skipping further queries for Host 192.192.71.164 in this round.
SNMP Error:
no response received
SNMPv2c_Session (remote host: "192.253.173.194" [192.253.173.194].161)
                   community: "mysnmpstring"
                  request ID: 1677143977
                 PDU bufsize: 8000 bytes
                     timeout: 2s
                     retries: 5
                     backoff: 1)
 at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on [email protected]:161::::2:v4only
 at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifHCInOctets.1 on  192.253.173.194 did not succeed
2016-02-21 17:53:24: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 192.253.173.194 in this round.
SNMP Error:
no response received
SNMPv2c_Session (remote host: "192.1.68.33" [192.1.68.33].161)
                   community: "mysnmpstring"
                  request ID: 2027967774
                 PDU bufsize: 8000 bytes
                     timeout: 2s
                     retries: 5
                     backoff: 1)
 at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifInOctets.2 ifOutOctets.2 on [email protected]:::::2:v4only
 at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifInOctets.2 on  192.1.68.33 did not succeed
2016-02-21 17:53:24: WARNING: no data for ifInOctets&ifOutOctets:[email protected]. Skipping further queries for Host 192.1.68.33 in this round.
SNMP Error:
no response received
SNMPv2c_Session (remote host: "192.253.138.202" [192.253.138.202].161)
                   community: "nc_BOSE"
                  request ID: 663507993
                 PDU bufsize: 8000 bytes
                     timeout: 2s
                     retries: 5
                     backoff: 1)
 at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on [email protected]:::::2:v4only
 at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifHCInOctets.1 on  192.253.138.202 did not succeed
2016-02-21 17:53:24: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 192.253.138.202 in this round.
SNMP Error:
no response received
SNMPv2c_Session (remote host: "192.253.95.2" [192.253.95.2].161)
                   community: "mysnmpstring"
                  request ID: 364165223
                 PDU bufsize: 8000 bytes
                     timeout: 2s
                     retries: 5
                     backoff: 1)
 at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on [email protected]:::::2:v4only
 at /usr/bin/mrtg line 2330
2016-02-21 17:53:24: WARNING: skipping because at least the query for ifHCInOctets.1 on  192.253.95.2 did not succeed
2016-02-21 17:53:24: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 192.253.95.2 in this round.
SNMP Error:
n

[Box293]

Let's delete these old config files as they appear to be for devices no longer monitored. They will be adding extra time to the checks while they timeout, which can lead to gaps in the graphs:

Code: Select all

rm -f /etc/mrtg/conf.d/192.253.174.10.cfg
rm -f /etc/mrtg/conf.d/usfm-rd-ti-a.bose.com.cfg
rm -f /etc/mrtg/conf.d/192.192.71.164.cfg
rm -f /etc/mrtg/conf.d/192.253.173.194.cfg
rm -f /etc/mrtg/conf.d/192.1.68.33.cfg
rm -f /etc/mrtg/conf.d/192.253.138.202.cfg
rm -f /etc/mrtg/conf.d/192.253.95.2.cfg

Of course if some of these are actually down and will be up again don't delete them.

After you've done that, run the command again to make sure there are no more errors.
Now we just need to wait to see if the graphs no longer have gaps.

Also, what is the output of:

Code: Select all

time LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok