Page 2 of 2
Re: RRD Files Do Not Exist...but they do.
Posted: Tue Jan 21, 2014 6:29 am
by mikew
Yes the version is 1.3.8
Re: RRD Files Do Not Exist...but they do.
Posted: Tue Jan 21, 2014 12:02 pm
by lmiltchev
Mike, I haven't been able to recreate the issue. I spun a Nagios XI R2.5 VM and ran the switch/router wizard against one of our test switches in house. It seems to be working just fine - it took several minutes before rrds got created, but not too long. Do you want to try this one more time?
Code: Select all
rm /usr/local/nagiosxi/tmp/mrtgscan* -f
rm -rf /var/lib/mrtg/*.rrd
cp /etc/mrtg/mrtg.cfg /etc/mrtg/mrtg.cfg.bkp
Open the "/etc/mrtg/mrtg.cfg" file in a text editor, and delete everything below this section:
Code: Select all
######################################################################
# Multi Router Traffic Grapher -- Example Configuration File
######################################################################
# This file is for use with mrtg-2.0
#
# Note:
#
# * Keywords must start at the begin of a line.
#
# * Lines which follow a keyword line which do start
# with a blank are appended to the keyword line
#
# * Empty Lines are ignored
#
# * Lines starting with a # sign are comments.
# Where should the logfiles, and webpages be created?
# Minimal mrtg.cfg
#--------------------
HtmlDir: /var/www/mrtg
ImageDir: /var/www/mrtg
LogFormat: rrdtool
LogDir: /var/lib/mrtg
ThreshDir: /var/lib/mrtg
#Target[r1]: 2:[email protected]
#MaxBytes[r1]: 1250000
#Title[r1]: Traffic Analysis
#PageTop[r1]: <H1>Stats for our Ethernet</H1>
WorkDir: /var/lib/mrtg
Verify the mrtg syntax:
Code: Select all
LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg -check
Re-run the wizard against one switch, and wait for 10-15 min to see if you are going to experience the same issue.
Re: RRD Files Do Not Exist...but they do.
Posted: Tue Jan 21, 2014 7:22 pm
by mikew
Yep...exactly the same
Re: RRD Files Do Not Exist...but they do.
Posted: Wed Jan 22, 2014 11:52 am
by lmiltchev
Mike, is there a chance we can set up a remote session? What's a good time for you?
Re: RRD Files Do Not Exist...but they do.
Posted: Fri Jan 24, 2014 5:46 am
by mikew
I cannot do a remote session, this is a large customer install.
Re: RRD Files Do Not Exist...but they do.
Posted: Fri Jan 24, 2014 3:22 pm
by sreinhardt
Not sure if this is the same system as the other post you have, but I thought I would confirm from the other post, have you upgraded to 2012r2.8c? I believe this should finally fix it.
Re: RRD Files Do Not Exist...but they do.
Posted: Sun Jan 26, 2014 5:31 am
by mikew
No this system is a 2012R2.5. After the problems with the fresh install on the 2012R2.8b I did not want to move this to the new version until I could figure out the issues with MRTG.
Re: RRD Files Do Not Exist...but they do.
Posted: Mon Jan 27, 2014 11:29 am
by scottwilkerson
Out of curiosity, can the nagios user see the rrd from the command line? Can you run the following replacing 192.168.5.41_10 with an actual ip_port
Code: Select all
su nagios -c 'ls -l /var/lib/mrtg/192.168.5.41_10.rrd'
Re: RRD Files Do Not Exist...but they do.
Posted: Mon Jan 27, 2014 8:44 pm
by mikew
Solved: This was an ugly unusual situation. I am documenting what I did for a solution in case it happens to some other unlucky person.
This is a large scale install with around 4000 port checks, 32 vCPU, 16 GB of RAM on a 2013R2.5 system. Upgrading to 2012R2.8c did not help.
Problem:
There were multiple problems that contributed to the situation. I found out that after I performed the install, individuals from the company changed the IP Address on about 10 routers and switches. What they did was only change the IP at the host and did not think about what that meant to ports being monitored. However, Nagios did change most (not all) of the port IP to the correct IP Address. So the mrtg.cfg file was correct, the rrd files did exist and they had the right permissions. Multiple removals and re-installs failed to solve the problem. As this was something I had to solve I stuck with it constantly trying to figure out each of the elements that contributed to the problem.
The output on random ports of these machines was:
".rrd file do not exist"
Of course they did exist.
Solution:
I worked my way through all of the MRTG settings and could find nothing wrong. Then I started backtracking the check_rrdtraf plugin. I found that if I ran that from the command line it worked:
/usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/192.158.1.1_34 500,500 800,800 M
So the problem was not the MRTG is was Nagios recognizing the rrd files from the command interface in the GUI. To solve it I created a script for each host that executed one port bandwidth check correctly and then the rest of the ports on that device would work. So once Nagios saw one port was OK ....magically the rest started working.
Re: RRD Files Do Not Exist...but they do.
Posted: Mon Jan 27, 2014 9:44 pm
by scottwilkerson
Wow, I have never seen this before... thanks for outlining everything in details Mike..