No response from remote host on interface bandwidth monitor

[kcprospect]

So we just tried to change the default value:

'timeout' => 120,

However, we started getting this error:

Bandwidth;CRITICAL;HARD;3;CRITICAL SNMP connection failed on 172.22.255.33 with error: Invalid timeout value [120], range 1.0 - 60.0 seconds

So we've now changed it to 60 seconds.

But still receiving the timeouts.

I've attached a sample of our code.

We've also tried to restart nrpe and adjust the nrpe command timeout to 80 sec.

Any additional help would be greatly appreciated.

[tgriep]

Can you run a snmpwalk against the device and get a response?
Here is an example of the snmpwalk, you may have to adjust the community string and the version for your device.

Code: Select all

snmpwalk 172.22.255.33 -v 2v -c public

Is it failing all of the time now or is it intermittent?
Can you login to your server as the nagios user and run the check from a command line and see if that works?

[kcprospect]

When we receive a Bandwidth alert on a specific interface:

I am able to walk that specific alerting interface from our nagios server. I am walking for the specific OIDs that are used to calculate the bandwidth.

I've also ran a tcpdump on a specific interface and it there are no dropped packets ... even when we receive the "No response from remote host ..." alert.


snmpbulkwalk -v2c -c Blur-RO 172.22.255.33 .1.3.6.1.2.1.2.2.1.2 | grep 7568
IF-MIB::ifDescr.440557568 = STRING: Ethernet9/39

snmpbulkwalk -v2c -c Blur-RO 172.22.255.33 .1.3.6.1.2.1.31.1.1.1.15 | grep 7568
IF-MIB::ifHighSpeed.440557568 = Gauge32: 10000

snmpbulkwalk -v2c -c Blur-RO 172.22.255.33 .1.3.6.1.2.1.31.1.1.1.10 | grep 7568
IF-MIB::ifHCOutOctets.440557568 = Counter64: 17291247279510

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:20:58.233429 IP 172.22.255.33.snmp > nagios003.50257: C=Blur-RO GetResponse(239) interfaces.ifTable.ifEntry.ifDescr.440557568="Ethernet9/39" interfaces.ifTable.ifEntry.ifSpeed.440557568=4294967295 31.1.1.1.15.440557568=10000 interfaces.ifTable.ifEntry.ifOperStatus.440557568=1 interfaces.ifTable.ifEntry.ifAdminStatus.440557568=1 interfaces.ifTable.ifEntry.ifInOctets.440557568=278357697 interfaces.ifTable.ifEntry.ifOutOctets.440557568=2933774613 31.1.1.1.6.440557568=7997507462849.000000 31.1.1.1.10.440557568=17277292239125.000000
17:26:48.224539 IP 172.22.255.33.snmp > nagios003.44670: C=Blur-RO GetResponse(239) interfaces.ifTable.ifEntry.ifDescr.440557568="Ethernet9/39" interfaces.ifTable.ifEntry.ifSpeed.440557568=4294967295 31.1.1.1.15.440557568=10000 interfaces.ifTable.ifEntry.ifOperStatus.440557568=1 interfaces.ifTable.ifEntry.ifAdminStatus.440557568=1 interfaces.ifTable.ifEntry.ifInOctets.440557568=869888659 interfaces.ifTable.ifEntry.ifOutOctets.440557568=3711076504 31.1.1.1.6.440557568=7998098993811.000000 31.1.1.1.10.440557568=17278069541016.000000
17:33:48.303582 IP 172.22.255.33.snmp > nagios003.33101: C=Blur-RO GetResponse(238) interfaces.ifTable.ifEntry.ifDescr.440557568="Ethernet9/39" interfaces.ifTable.ifEntry.ifSpeed.440557568=4294967295 31.1.1.1.15.440557568=10000 interfaces.ifTable.ifEntry.ifOperStatus.440557568=1 interfaces.ifTable.ifEntry.ifAdminStatus.440557568=1 interfaces.ifTable.ifEntry.ifInOctets.440557568=1624889746 interfaces.ifTable.ifEntry.ifOutOctets.440557568=316387805 31.1.1.1.6.440557568=7998853994898.000000 31.1.1.1.10.440557568=17278969819613.000000
17:38:58.351645 IP 172.22.255.33.snmp > nagios003.56780: C=Blur-RO GetResponse(238) interfaces.ifTable.ifEntry.ifDescr.440557568="Ethernet9/39" interfaces.ifTable.ifEntry.ifSpeed.440557568=4294967295 31.1.1.1.15.440557568=10000 interfaces.ifTable.ifEntry.ifOperStatus.440557568=1 interfaces.ifTable.ifEntry.ifAdminStatus.440557568=1 interfaces.ifTable.ifEntry.ifInOctets.440557568=2131379515 interfaces.ifTable.ifEntry.ifOutOctets.440557568=1020610781 31.1.1.1.6.440557568=7999360484667.000000 31.1.1.1.10.440557568=17279674042589.000000
17:46:02.290719 IP 172.22.255.33.snmp > nagios003.36810: C=Blur-RO GetResponse(239) interfaces.ifTable.ifEntry.ifDescr.440557568="Ethernet9/39" interfaces.ifTable.ifEntry.ifSpeed.440557568=4294967295 31.1.1.1.15.440557568=10000 interfaces.ifTable.ifEntry.ifOperStatus.440557568=1 interfaces.ifTable.ifEntry.ifAdminStatus.440557568=1 interfaces.ifTable.ifEntry.ifInOctets.440557568=2821068902 interfaces.ifTable.ifEntry.ifOutOctets.440557568=1970399078 31.1.1.1.6.440557568=8000050174054.000000 31.1.1.1.10.440557568=17280623830886.000000
17:46:02.292502 IP 172.22.255.33.snmp > nagios003.36810: C=Blur-RO GetResponse(239) interfaces.ifTable.ifEntry.ifDescr.440557568="Ethernet9/39" interfaces.ifTable.ifEntry.ifSpeed.440557568=4294967295 31.1.1.1.15.440557568=10000 interfaces.ifTable.ifEntry.ifOperStatus.440557568=1 interfaces.ifTable.ifEntry.ifAdminStatus.440557568=1 interfaces.ifTable.ifEntry.ifInOctets.440557568=2821068902 interfaces.ifTable.ifEntry.ifOutOctets.440557568=1970399078 31.1.1.1.6.440557568=8000050174054.000000 31.1.1.1.10.440557568=17280623830886.000000
17:50:02.255460 IP 172.22.255.33.snmp > nagios003.54911: C=Blur-RO GetResponse(240) interfaces.ifTable.ifEntry.ifDescr.440557568="Ethernet9/39" interfaces.ifTable.ifEntry.ifSpeed.440557568=4294967295 31.1.1.1.15.440557568=10000 interfaces.ifTable.ifEntry.ifOperStatus.440557568=1 interfaces.ifTable.ifEntry.ifAdminStatus.440557568=1 interfaces.ifTable.ifEntry.ifInOctets.440557568=3210814624 interfaces.ifTable.ifEntry.ifOutOctets.440557568=2481483163 31.1.1.1.6.440557568=8000439919776.000000 31.1.1.1.10.440557568=17281134914971.000000
17:55:17.163623 IP 172.22.255.33.snmp > nagios003.33746: C=Blur-RO GetResponse(240) interfaces.ifTable.ifEntry.ifDescr.440557568="Ethernet9/39" interfaces.ifTable.ifEntry.ifSpeed.440557568=4294967295 31.1.1.1.15.440557568=10000 interfaces.ifTable.ifEntry.ifOperStatus.440557568=1 interfaces.ifTable.ifEntry.ifAdminStatus.440557568=1 interfaces.ifTable.ifEntry.ifInOctets.440557568=3765162379 interfaces.ifTable.ifEntry.ifOutOctets.440557568=3225353585 31.1.1.1.6.440557568=8000994267531.000000 31.1.1.1.10.440557568=17281878785393.000000
18:00:17.222061 IP 172.22.255.33.snmp > nagios003.51140: C=Blur-RO GetResponse(240) interfaces.ifTable.ifEntry.ifDescr.440557568="Ethernet9/39" interfaces.ifTable.ifEntry.ifSpeed.440557568=4294967295 31.1.1.1.15.440557568=10000 interfaces.ifTable.ifEntry.ifOperStatus.440557568=1 interfaces.ifTable.ifEntry.ifAdminStatus.440557568=1 interfaces.ifTable.ifEntry.ifInOctets.440557568=4269632882 interfaces.ifTable.ifEntry.ifOutOctets.440557568=3873394124 31.1.1.1.6.440557568=8001498738034.000000 31.1.1.1.10.440557568=17282526825932.000000
18:05:17.173057 IP 172.22.255.33.snmp > nagios003.58291: C=Blur-RO GetResponse(238) interfaces.ifTable.ifEntry.ifDescr.440557568="Ethernet9/39" interfaces.ifTable.ifEntry.ifSpeed.440557568=4294967295 31.1.1.1.15.440557568=10000 interfaces.ifTable.ifEntry.ifOperStatus.440557568=1 interfaces.ifTable.ifEntry.ifAdminStatus.440557568=1 interfaces.ifTable.ifEntry.ifInOctets.440557568=538055466 interfaces.ifTable.ifEntry.ifOutOctets.440557568=330021319 31.1.1.1.6.440557568=8002062127914.000000 31.1.1.1.10.440557568=17283278420423.000000
^C5615 packets captured
5615 packets received by filter
0 packets dropped by kernel

[tgriep]

Have you tried to add -t 60 to the check that is failing to see if this helps by increasing the timeout to 60 seconds?
Is it failing all of the time now or is it intermittent?
Can you login to your server as the nagios user and run the check from a command line and see if that works?

[kcprospect]

It seems like we receive two types of alerts:

1. Bandwidth;CRITICAL;SOFT;1;CRITICAL SNMP agent failure: No response from remote host

2. Bandwidth;CRITICAL;SOFT;2;CRITICAL Count not communitate with SNMP agent: No response from remote host

[lgroschen]

I have experienced some interesting output specifically from Cisco Nexus devices. Especially concerning a Nexus OS device with ifHighSpeed interfaces that aren't being treated as highspeed, but this is probably unrelated to your issue. It does give me the idea of maybe using the script as you have it, or the original that it was written from, to test it on another device and see if you get the same error.

Also, what version of Perl are you running on this machine? The Net::SNMP module requires 5.6 or newer.

[kcprospect]

Is there any updated/newer plugin version that can effectively monitor Nexus interface bandwidth (64bit).

I believe the OIDs used for 64bit interface bandwidth calculations are:
'ifRX64' => '.1.3.6.1.2.1.31.1.1.1.6.', # interface bytes in (64-bit)
'ifTX64' => '.1.3.6.1.2.1.31.1.1.1.10.' # interface bytes out (64-bit)
'ifHighSpeed' => '.1.3.6.1.2.1.31.1.1.1.15.', # interface bandwidth limit (64-bit)

Thanks again for your time!

[kcprospect]

In addition, we have nearly 75 interfaces that we are performing the interface bandwidth check on.

We have another pair of other Cisco Nexus switches that are experiencing the same issue, but just not as frequent ... these switches only have 58 interfaces that have the bandwidth monitor.

Thanks!

[tgriep]

If the errors are intermittent, it sounds like the plugin is timing out.
But, if you want to look for a different plugin, take a look at the Exchange site at this link.
https://exchange.nagios.org/

[kcprospect]

We've built out another test environment that mimics our production environment and noticed that when we reduce the number of interfaces that we monitor on our switch, the alerts seem to stop.

We've also adjusted the normal_check_interval from 5 to 10

Each of our switches has approx. 70 interfaces each that use the bandwidth monitor.

We were wondering if there is a way to tune nagios to handle the additional load for bandwidth monitor checks?

Thanks!