RRD Files Do Not Exist...but they do.

[mikew]

Yes the version is 1.3.8

[lmiltchev]

Mike, I haven't been able to recreate the issue. I spun a Nagios XI R2.5 VM and ran the switch/router wizard against one of our test switches in house. It seems to be working just fine - it took several minutes before rrds got created, but not too long. Do you want to try this one more time?

Code: Select all

rm /usr/local/nagiosxi/tmp/mrtgscan* -f
rm -rf /var/lib/mrtg/*.rrd
cp /etc/mrtg/mrtg.cfg /etc/mrtg/mrtg.cfg.bkp

Open the "/etc/mrtg/mrtg.cfg" file in a text editor, and delete everything below this section:

Code: Select all

######################################################################
# Multi Router Traffic Grapher -- Example Configuration File
######################################################################
# This file is for use with mrtg-2.0
#
# Note:
#
# * Keywords must start at the begin of a line.
#
# * Lines which follow a keyword line which do start
#   with a blank are appended to the keyword line
#
# * Empty Lines are ignored
#
# * Lines starting with a # sign are comments.

# Where should the logfiles, and webpages be created?

# Minimal mrtg.cfg
#--------------------

HtmlDir: /var/www/mrtg
ImageDir: /var/www/mrtg
LogFormat: rrdtool
LogDir: /var/lib/mrtg
ThreshDir: /var/lib/mrtg

#Target[r1]: 2:[email protected]
#MaxBytes[r1]: 1250000
#Title[r1]: Traffic Analysis
#PageTop[r1]: <H1>Stats for our Ethernet</H1>

WorkDir: /var/lib/mrtg

Verify the mrtg syntax:

Code: Select all

LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg -check

Re-run the wizard against one switch, and wait for 10-15 min to see if you are going to experience the same issue.

[mikew]

Yep...exactly the same

[lmiltchev]

Mike, is there a chance we can set up a remote session? What's a good time for you?

[mikew]

I cannot do a remote session, this is a large customer install.

[sreinhardt]

Not sure if this is the same system as the other post you have, but I thought I would confirm from the other post, have you upgraded to 2012r2.8c? I believe this should finally fix it.

[mikew]

No this system is a 2012R2.5. After the problems with the fresh install on the 2012R2.8b I did not want to move this to the new version until I could figure out the issues with MRTG.

[scottwilkerson]

Out of curiosity, can the nagios user see the rrd from the command line? Can you run the following replacing 192.168.5.41_10 with an actual ip_port

Code: Select all

su nagios -c 'ls -l /var/lib/mrtg/192.168.5.41_10.rrd'

[mikew]

Solved: This was an ugly unusual situation. I am documenting what I did for a solution in case it happens to some other unlucky person.

This is a large scale install with around 4000 port checks, 32 vCPU, 16 GB of RAM on a 2013R2.5 system. Upgrading to 2012R2.8c did not help.

Problem:
There were multiple problems that contributed to the situation. I found out that after I performed the install, individuals from the company changed the IP Address on about 10 routers and switches. What they did was only change the IP at the host and did not think about what that meant to ports being monitored. However, Nagios did change most (not all) of the port IP to the correct IP Address. So the mrtg.cfg file was correct, the rrd files did exist and they had the right permissions. Multiple removals and re-installs failed to solve the problem. As this was something I had to solve I stuck with it constantly trying to figure out each of the elements that contributed to the problem.

The output on random ports of these machines was:
".rrd file do not exist"

Of course they did exist.

Solution:
I worked my way through all of the MRTG settings and could find nothing wrong. Then I started backtracking the check_rrdtraf plugin. I found that if I ran that from the command line it worked:
/usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/192.158.1.1_34 500,500 800,800 M

So the problem was not the MRTG is was Nagios recognizing the rrd files from the command interface in the GUI. To solve it I created a script for each host that executed one port bandwidth check correctly and then the rest of the ports on that device would work. So once Nagios saw one port was OK ....magically the rest started working.

[scottwilkerson]

Wow, I have never seen this before... thanks for outlining everything in details Mike..