Missing RRD data

steliopappas · Post by **steliopappas** » Thu Nov 14, 2019 10:01 pm

Hi guys

I made a post back in March about some missing data in my graphs. Unfortunately, other work got in the way which prevented me from investigating further. That was at a time when the missing data was minimal and there was lower impact. I've since returned to the original post, but it appears to be locked. I guess because I left it too long

-Sorry.

The original post was:
"Missing data in graph, but not in RRD file"
https://support.nagios.com/forum/viewto ... 16&t=52944

What I've learnt so far, is that it seems the original RRD files contain data in the following location:
/var/lib/mrtg/my_hostname.rrd -Performing an rrdtool dump reveals all the data appears to be present.

It seems we also create data in the following location:
/usr/local/nagios/share/perfdata/my_hostname/GigabitEthernet0_0_Bandwidth.rrd -Performing an rrdtool dump reveals large chunks of zero'd data. Here is how it looks:

 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>

Omitted data from 13:22:00 to 15:03:00 - It was all zero'd

 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>3.9357020000e+00</v><v>1.1723600000e+00</v></row>
 <row><v>3.9357020000e+00</v><v>1.1723600000e+00</v></row>
 <row><v>3.9357020000e+00</v><v>1.1723600000e+00</v></row>
 <row><v>3.9357020000e+00</v><v>1.1723600000e+00</v></row>
 <row><v>3.8701069667e+00</v><v>1.1528206667e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>
 <row><v>0.0000000000e+00</v><v>0.0000000000e+00</v></row>

From time to time, I can see some data. -See 15:11:00 to 15:15:00 above.

I'm hoping to find some sort of log that might reveal what is causing the zero'd data in the rrd file. I'm guessing there will be a config file somewhere that might need tweaking.

Can someone point me in the right direction?

Thanks in advance
Stel

Post by **tgriep** » Fri Nov 15, 2019 2:01 pm

Can you run the following commands as root and post the the /tmp/mrtg.txt file here?

Code: Select all

LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg -debug=cfg,base,log &> /tmp/mrtg.txt
LANG=C LC_ALL=C /usr/bin/mrtg &>> /tmp/mrtg.txt
LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lib/mrtg/mrtg.lock --confcache-file /var/lib/mrtg/mrtg.ok --user=nagios --group=nagios &>> /tmp/mrtg.txt
{ time LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg 2>1 ; } 2>> /tmp/mrtg.txt

We will also need to know the IP address of the device and the port name to troubleshoot this.
I suspect the MRTG application that gathers the Bandwidth information is where the issue is at.

steliopappas · Post by **steliopappas** » Mon Nov 18, 2019 2:04 am

Hi tgriep

Thanks for the quick reply. I ran the commands you sent through with the only modification being that I separated the output. -I was curious to see the output of each command myself.

It seems like I have permission issues which I'm guessing shouldn't be hard to resolve with a chown here and there. I haven't started down that path yet, as I thought I should let you take a look for yourself first.

I've attached the first three files in this post. -I'll add the fourth in the next post.

Thanks in advance
Stel

steliopappas · Post by **steliopappas** » Mon Nov 18, 2019 2:05 am

Here is the fourth file as promised.

Stel

Post by **tgriep** » Mon Nov 18, 2019 11:21 am

Yes, the permissions of the files and folders are one issue.

First, upgrade the Network Switch / Router wizard in XI by logging in to the XI GUI and going to the Admin > Manage Config Wiwards menu.
Click the Check for Updates button and update the Network Switch / Router wizard.
Then run the following as root to set the permissions of the files the wizard / plugin use to get the bandwidth information.

Code: Select all

chown apache:nagios /etc/mrtg -R
chmod 775 /etc/mrtg -R
chown apache:nagios /var/lib/mrtg -R
chmod 775 /var/lib/mrtg -R

Another issue is that it is taking a long time to run the MRTG application and that has to be fixed.

Edit the /etc/mrtg/mrtg.cfg file and change this line from

Code: Select all

Forks: 4

to

Code: Select all

Forks: 16

Save the file and restart crond by running

Code: Select all

service crond restart

Last thing to do is to go through the mrtg3.txt file and fix the DNS issues for some of the devices and to clean up the MRTG config files for the devices that do not exist.

Look for these messages

does not resolve to an IPv4 address
No response from remote host
did not eval into defined data

If the device does not exist anymore, remove the configuration file for it from the /etc/mrtg/conf.d folder and that will also speed up the process.

steliopappas · Post by **steliopappas** » Tue Nov 19, 2019 4:55 pm

Thanks tgriep! -That did the trick.

Graphs are looking great.

Stel

Post by **tgriep** » Tue Nov 19, 2019 5:23 pm

That is good news and you are very welcome. I'll close and lock the post as solved but feel free to open a new one in the future for and other questions or issues.

Nagios Support Forum

Missing RRD data

Missing RRD data

Re: Missing RRD data

Re: Missing RRD data

Re: Missing RRD data

Re: Missing RRD data

Re: Missing RRD data

Re: Missing RRD data