/var/lib/mrtg/6.-.-.-_154.rrd does not exist, but it does

[scottwilkerson]

Hmm,it is normal to see that error on newly added items (for about 15 minutes), but once the file exists this shouldn't be the case.

[kotterbein]

bumping this because after redoing a discovery, we are seeing this problem on multiple interfaces. I followed this document:

http://assets.nagios.com/downloads/nagi ... Router.pdf

to see if this would remedy the issue, and it seems to have created other problems. I now see on a different node after applying the re-discovery the following errors on status:

Code: Select all

WARNING: SNMP error: Received noSuchName(2) error-status at error-index 3

Is there any further way to troubleshoot this?

[scottwilkerson]

can you run the following and post any errors

Code: Select all

env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg

[kotterbein]

quite a few errors now:

Code: Select all

at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 490
SNMPGET Problem for ifInOctets.97 ifOutOctets.97 on [email protected]::::::v4only
 at /usr/bin/mrtg line 2035
SNMP Error:
Received SNMP response with error code
  error status: noSuchName
  index 1 (OID: 1.3.6.1.2.1.2.2.1.10.99)
SNMPv1_Session (remote host: "6.0.x.xxx" [6.0.x.xxx].161)
                  community: ""
                 request ID: 1500442607
                PDU bufsize: 8000 bytes
                    timeout: 2s
                    retries: 5
                    backoff: 1)
 at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 490
SNMPGET Problem for ifInOctets.99 ifOutOctets.99 on [email protected]::::::v4only
 at /usr/bin/mrtg line 2035
SNMP Error:
Received SNMP response with error code
  error status: noSuchName

This shows up repeatedly:

Code: Select all

Wednesday, 27 February 2013 at 14:03: ERROR: Target[6.x.xxx.xx_5184][_OUT_] ' $target->[3818]{$mode} ' did not eval into defined data

this is the last error:

Code: Select all

Wednesday, 27 February 2013 at 14:03: ERROR: Cannot tune logfile: mmaping file '/var/lib/mrtg/6.0.x.xxx_154.rrd': Invalid argument
Wednesday, 27 February 2013 at 14:03: ERROR: Cannot update /var/lib/mrtg/6.0.x.xxx_154.rrd with '1361991809:526726574:600555016' mmaping file '/var/lib/mrtg/6.0.x.xxx_154.rrd': Invalid argument
Use of uninitialized value in modulus (%) at /usr/bin/mrtg line 802.
Use of uninitialized value in modulus (%) at /usr/bin/mrtg line 802.

Its kind of all over the map here...

[scottwilkerson]

This appears your are trying to monitor items that no longer exist on the device or are now administratively down.

[kotterbein]

so we have stripped down everything, and tried a full re-discovery of the node that is having problems, and yet they persist. I have gone as far as to completedly remove all services and the host completely, restart nagios and do a full re-discovery, yet the same 4 interfaces give us the same error:

Code: Select all

2013-03-12T15:26:09.116261-04:00 id-xxx01 nagios: SERVICE ALERT: xx-xx02.inf.xxx.com;US-AKU01 7041417 Bandwidth;UNKNOWN;HARD;1;/var/lib/mrtg/6.x.x.xxx_193.rrd does not exist.
2013-03-12T15:26:27.207510-04:00 id-xxx01 nagios: SERVICE ALERT: xx-xx02.inf.xxx.com;US-CWN02 7804049 Pico Bandwidth;UNKNOWN;HARD;1;/var/lib/mrtg/6.x.x.xxx_108.rrd does not exist.
2013-03-12T15:27:03.172382-04:00 id-xxx01 nagios: SERVICE ALERT: xx-xx02.inf.xxx.com;US-MPS 7733832 Bandwidth;UNKNOWN;HARD;1;/var/lib/mrtg/6.x.x.xxx_204.rrd does not exist.
2013-03-12T15:27:12.211708-04:00 id-xxx01 nagios: SERVICE ALERT: xx-xx02.inf.xxx.com;US-STK01 +6335418 Bandwidth;UNKNOWN;HARD;1;/var/lib/mrtg/6.x.x.xxx_11.rrd does not exist.

this is throwing into question our ability to really effectively manage newly joined interfaces if even on a full re-discovery we are seeing the same errors. is there anything further we can try? can we use the auto-discovery effectively?

[kotterbein]

some more info- from the mrtg.cfg file, one of the interfactes in question:

Code: Select all

### Interface 193 >> Descr: 'GigabitEthernet8/1' | Name: 'Gi8/1' | Ip: 'xx.x.x.141' | Eth: '' ###

Target[6.x.x.xxx_193]: 193:[email protected]:::::2
SetEnv[6.x.x.xxx_193]: MRTG_INT_IP="10.xx.xx.xxx" MRTG_INT_DESCR="GigabitEthernet8/1"
MaxBytes[6.x.x.xxx_193]: 125000000
Title[6.x.x.xxx_193]: Traffic Analysis for 193 -- xx-xx02.xxx.com
PageTop[6.x.x.xxx_193]: <h1>Traffic Analysis for 193 -- xx-xx02.xxx.com</h1>
                <div id="sysdetails">
                        <table>
                                <tr>
                                        <td>System:</td>
                                        <td>xx-xx02.xxx.com in </td>
                                </tr>
                                <tr>
                                        <td>Maintainer:</td>
                                        <td></td>
                                </tr>
                                <tr>
                                        <td>Description:</td>
                                        <td>GigabitEthernet8/1 US-AKU01 7041417 </td>
                                </tr>
                                <tr>
                                        <td>ifType:</td>
                                        <td>ethernetCsmacd (6)</td>
                                </tr>
                                <tr>
                                        <td>ifName:</td>
                                        <td>Gi8/1</td>
                                </tr>
                                <tr>
                                        <td>Max Speed:</td>
                                        <td>125.0 MBytes/s</td>
                                </tr>
                                <tr>
                                        <td>Ip:</td>
                                        <td>xx.x.x.141 ()</td>
                                </tr>
                        </table>
                </div>

[slansing]

To be completely sure, when you said you "stripped down everything" what steps did you take? Did you run through the documentation on adding new devices to the existing mrtg.cfg file? Specifically these parts?:

Removing the temp files:

Code: Select all

rm /usr/local/nagiosxi/tmp/mrtgscan* -f

MRTG will attempt to add a service to its configuration using cfgmaker command, however, if it runs into any previous configuration for
the same host, it will not add the new configuration for that host and will simply not add anything new for that host. What we must do is
go through our
/etc/mrtg/mrtg.cfg
and remove any lines referring to the previous configuration for the host. Here is a sample
snippet from an example configuration. In the following example we see some configuration for 192.168.5.41, a switch that we wish add
more services to.

### Interface 1 >> Descr: 'Port:-1-Gigabit---Level' | Name: '' | Ip: '' | Eth: '' ###
Target[192.168.5.41_1]: 1:[email protected]:::::1
SetEnv[192.168.5.41_1]: MRTG_INT_IP="" MRTG_INT_DESCR="Port:-1-Gigabit---Level"
MaxBytes[192.168.5.41_1]: 125000000
Title[192.168.5.41_1]: Traffic Analysis for 1 -- PROCURVE J9450A
PageTop[192.168.5.41_1]: <h1>Traffic Analysis for 1 -- PROCURVE J9450A</h1>
<div id="sysdetails">
<table>
<tr>
<td>System:</td>

[kotterbein]

Yes, I ran through those steps. I do realize now though that the mrtg.ok file had no updated since feb26th, and the newly added / reconfigured devices rrd files have not updated since either. I just ran


LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok


And received several errors that I believe I need to cleanup, such as

Received SNMP response with error code
error status: noSuchName

Looks like my mrtg.cfg file is corrupt.

[slansing]

Do you have rrdcached set up on this Nagios XI system? Can you run the following?:

Code: Select all

ls -l /tmp/rrd.journal*
cat /etc/sysconfig/rrdcached