MRTG problems with RRD returning "got: ???/???"

[msbensonstk]

Ok, took a while but I have all the errors fixed and restared crond.

That didn't fix the issue, however after removing the /var/lib/mrtg/*.rrd files now I'm starting to get real values again.

Now the big question: Why did this suddenly stop working? I understand the fix now, but why would MRTG suddenly start reporting ???/??? for interface bandwidth?

[tgriep]

Could you run the following and post back the results?

Code: Select all

LANG=C LC_ALL=C /usr/bin/mrtg
rrdtool --version

[msbensonstk]

Code: Select all

[msb@nagiosxi0 ~]$ LANG=C LC_ALL=C /usr/bin/mrtg
Usage: mrtg <config-file>

mrtg-2.17.4 - Multi Router Traffic Grapher

Copyright 1995-2006 by Tobias Oetiker
Licensed under the Gnu GPL.

If you want to know more about this tool, you might want
to read the docs. You can find everything on the
mrtg website:

http://oss.oetiker.ch/mrtg/

[msb@nagiosxi0 ~]$ rrdtool --version
RRDtool 1.4.7  Copyright 1997-2012 by Tobias Oetiker <[email protected]>
               Compiled Apr  5 2012 23:38:48

Usage: rrdtool [options] command command_options
Valid commands: create, update, updatev, graph, graphv,  dump, restore,
		last, lastupdate, first, info, fetch, tune,
		resize, xport, flushcached

RRDtool is distributed under the Terms of the GNU General
Public License Version 2. (www.gnu.org/copyleft/gpl.html)

For more information read the RRD manpages

[msb@nagiosxi0 ~]$ 

[tgriep]

I don't have enough data to figure out why they were corrupted.
Can I check one more thing, can you run this and post back?

Code: Select all

cat /etc/mrtg/mrtg.cfg

[cfgriffith]

I am having some issues getting graphs to work for some of my Juniper switches with the check_xi_service_mrtgtraf check.
The checks all seem to be working fine, I also see the bandwidth usage going up and down.

A bunch of my switch checks that I had working yesterday do not seem to be graphing anything now. They still have the graph visible but it is empty. Some of the newer checks I made today do not have a graph at all but again, the actual check is showing the bandwidth usage still going up and down.

When I run:

Code: Select all

sudo env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --debug="snpo,log" --logging $HOME/mrtg_troubleshooting.txt

and less -i the .txt file for 'error' I don't see any errors.

I also see that the appropriately named .rrd files were generated for the switches in question. So I am at a loss.

Now I did recently delete and import some switch configuration files. Essentially I copied the existing files out of usr/local/nagios/etc/services and edited them slightly, deleted the services that I edited in XI and then re-imported the .cfg files via Nagios XI.

Whether or not this caused the problem I am not sure. I have also since upgraded the switch creation wizard but I don't think that would have any effect.

Here is the current check for one that was working but is now not:

Code: Select all

check_xi_service_mrtgtraf!10.32.21.70_505.rrd!5000,5000!8000,8000!M!!!!

Here is the check for one that doesn't seem to have a graph generated at all:

Code: Select all

check_xi_service_mrtgtraf!10.32.20.1_675.rrd!500,500!800,800!M

Also note while I do have both Gig and TenGig interfaces, I measure bandwidth in Mb. This seemed to be working fine with the graphing before..

[tgriep]

In the Nagios XI WIKI, there is a section on trouble shooting Performance Graphing.
Could you go through it to see if this fixes your problem?
http://support.nagios.com/wiki/index.ph ... h_Problems

[cfgriffith]

Will do and get back to you.

[jolson]

Sounds good - let us know. Thanks!

[cfgriffith]

Here are some things I have determined after following the Wiki page:

Some objects that are having graphing issues don't appear to have been generated in /usr/local/nagios/share/perfdata while some of them have been.

Nothing is in /usr/local/nagios/var/spool/xidpe

/usr/local/nagios/share/perfdata is read and execute for everyone, does it need to be rwx? It is rwx for nagios user and group.

When I do ./check_rrdtraf -f '/var/lib/mrtg/192.168.6.1_1.rrd' on the checks they all return expected values.

After turning logging on as requested in the wiki, all of the tail commands do not return anything which I am assuming means there has been no errors.

All bandwidth graphs seem to be broken. Ones that were working before show an empty graph and ones that were created after the breakage do not have a graph at all. However they are all still polling correctly and showing their current bandwidth usage, just no graphs.

Example: Performance Data: in=121.869643Mb/s;500;800 out=130.687588Mb/s;500;800

The /var/lock/mrtg directory exists but has no files in it. Pretty sure this is intended.

[tgriep]

OK, check this folder and see if they being updated. The mrtg cron updates them

Code: Select all

/var/lib/mrtg

If they are up to date, that means they could be corrupt and they need to be deleted.
Make a backup of them in case you need to go back.
Those files will be recreated and hopefully after 20 minutes, the graphs will populate.