Performance Graphs are High but Showing in-out as 0-0

[sarfarosh]

Hi Team,
We configured some routers and firewalls in nagios xi latest version. Performance graphs are showing incorrect for some of the interfaces. Please find below the attachment of performance graph and performance data.
I have already deleted related following files and re-created the device but still same issue,
rm -rf /var/lib/mrtg/ip_*.rrd
rm -rf /etc/mrtg/conf.d/ip.cfg
rm -rf /usr/local/nagios/share/perfdata/Hostname/*

Also find the httpd error log:
bash-4.1# tail -30 error_log
No log handling enabled - turning on stderr logging
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/HOST-RESOURCES-MIB.txt)
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/IP-FORWARD-MIB.txt)
No log handling enabled - turning on stderr logging
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/HOST-RESOURCES-MIB.txt)
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/IP-FORWARD-MIB.txt)
No log handling enabled - turning on stderr logging
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/HOST-RESOURCES-MIB.txt)
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/IP-FORWARD-MIB.txt)
No log handling enabled - turning on stderr logging
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/HOST-RESOURCES-MIB.txt)
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/IP-FORWARD-MIB.txt)
zend_mm_heap corrupted
No log handling enabled - turning on stderr logging
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/HOST-RESOURCES-MIB.txt)
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/IP-FORWARD-MIB.txt)
[Thu Jun 29 14:16:54 2017] [notice] caught SIGTERM, shutting down
No log handling enabled - turning on stderr logging
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/HOST-RESOURCES-MIB.txt)
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/IP-FORWARD-MIB.txt)
ERROR: end time: did you really mean month 2017?
No log handling enabled - turning on stderr logging
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/HOST-RESOURCES-MIB.txt)
Did not find 'InterfaceIndexOrZero' in module IF-MIB (/usr/share/snmp/mibs/IP-FORWARD-MIB.txt)
ERROR: end time: did you really mean month 2017?
ERROR: end time: did you really mean month 2017?
ERROR: end time: did you really mean month 2017?
ERROR: end time: did you really mean month 2017?
ERROR: end time: did you really mean month 2017?
ERROR: end time: did you really mean month 2017?
ERROR: end time: did you really mean month 2017?
ERROR: end time: did you really mean month 2017?
ERROR: end time: did you really mean month 2017?

[tgriep]

There are a couple of KB articles that may fix the issue you are having. Try running through them and see if one of them files the issue you are having
https://support.nagios.com/kb/article/n ... *-cfg.html
https://support.nagios.com/kb/article/n ... x-cfg.html

If neither of them work, can you run the following command as root and email back the /tmp/mrtg.txt file?

Code: Select all

LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg -debug=cfg,base &> /tmp/mrtg.txt
ls -l /usr/share/snmp/mibs/ >> /tmp/mrtg.txt

[sarfarosh]

Hi tgriep,
Provided links did not help. As i am receiving correct value for some interfaces of device and zero value (in/out) for some interfaces of same device. This is the problem with many such devices. Please find the attachment below.

Also i have found the below error in '/var/spool/mail/root'
-bash-4.1# tailf /var/spool/mail/root
2017-06-30 16:59:49: ERROR: Target[115.110.157.35_8][_OUT_] ' $target->[14219]{$mode} ' did not eval into defined data
2017-06-30 16:59:49: ERROR: Target[115.110.157.34_9][_IN_] ' $target->[14232]{$mode} ' did not eval into defined data
2017-06-30 16:59:49: ERROR: Target[115.110.157.34_9][_OUT_] ' $target->[14232]{$mode} ' did not eval into defined data
2017-06-30 16:59:49: ERROR: Target[103.9.225.5_8][_IN_] ' $target->[14244]{$mode} ' did not eval into defined data
2017-06-30 16:59:49: ERROR: Target[103.9.225.5_8][_OUT_] ' $target->[14244]{$mode} ' did not eval into defined data
2017-06-30 16:59:49: ERROR: Target[103.246.49.3_17][_IN_] ' $target->[14268]{$mode} ' did not eval into defined data
2017-06-30 16:59:49: ERROR: Target[103.246.49.3_17][_OUT_] ' $target->[14268]{$mode} ' did not eval into defined data
2017-06-30 16:59:49: ERROR: Target[103.246.49.3_18][_IN_] ' $target->[14269]{$mode} ' did not eval into defined data
2017-06-30 16:59:49: ERROR: Target[103.246.49.3_18][_OUT_] ' $target->[14269]{$mode} ' did not eval into defined data

[tgriep]

From what I found in the past, the "did not eval into defined data" message usually means that the interface the MRTG application is polling is down and it cannot get valid data.
If you verify that the interface is down, you would have to edit the configuration file for that device in the following folder and remove the interface that is disabled.

Code: Select all

/etc/mrtg/conf.d

I understand now, the bandwidth is always zero but the graph shows 0 to 0.00 on the Y axis.
If the system's language it not set to english, that could cause the rounding error and the invalid graphs.
Try this and see if it fixes the issue.
https://support.nagios.com/kb/article/n ... stems.html

If not, can you run the following in a shell on the server so I can see which version of the plugin and rrdtools that are on the server?

Code: Select all

/usr/local/nagios/libexec/check_mrtgtraf -V
rrdtool -V
yum list installed |grep rrdtool
locale

Can you post the .rrd file associated to that interface from the /var/lib/mrtg folder so we can view it?

Also, can you post the .xml and .rrd files for that interface? They can be found in this folder.

Code: Select all

/usr/local/nagios/share/perfdata/RTRPUN05DMP02

[sarfarosh]

Hi triep,
bash-4.1# /usr/local/nagios/libexec/check_mrtgtraf -V
check_mrtgtraf v2.0.3 (nagios-plugins 2.0.3)
bash-4.1# rrdtool -V
RRDtool 1.3.8 Copyright 1997-2009 by Tobias Oetiker <[email protected]>
Compiled Oct 10 2016 14:58:55

Usage: rrdtool [options] command command_options

Valid commands: create, update, updatev, graph, graphv, dump, restore,
last, lastupdate, first, info, fetch, tune,
resize, xport

RRDtool is distributed under the Terms of the GNU General
Public License Version 2. (http://www.gnu.org/copyleft/gpl.html)

For more information read the RRD manpages

bash-4.1# yum list installed |grep rrdtool
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
rrdtool.x86_64 1.3.8-10.el6 @rhel-6-server-rpms
rrdtool-devel.x86_64 1.3.8-10.el6 @rhel-6-server-optional-rpms
rrdtool-perl.x86_64 1.3.8-10.el6 @rhel-6-server-optional-rpms
rrdtool-python.x86_64 1.3.8-10.el6 @rhel-6-server-optional-rpms
bash-4.1# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

NOTE: I have a doubt on the value In/Out is Scaled here. For example 0.000123Mb/s is showing as zero.

[tgriep]

Yes, those interfaces have very little amount of traffic that the graph will display the lines but the labels are truncated.
One thing you can do is to edit the service an instead of displaying the data in MegaBits, you can change it to KiloBits and that should display the data in the labels.
To do this, go to the Core Config Manager > Services menu and edit the service. Change $ARG4$ from M to K and save the changes.
This should move the decimal place for the check and you should start to see the values in the graph.
One thing, you will have to delete the .xml and .rrd files for those services so they will be recreated.

[sarfarosh]

Hi tgriep,
your solution worked for me and i am getting graphs in KBs. But client wants to see the graph in MBs only, so is it possible to not truncate the decimal value and show it as it is. for e.g 0.000000128 mbs as 0.000000128 instead of 0.0 mbs.
One more thing that i observed today is i am not able to see graphs for one interface of device. Device is added more than 30 minutes back. Please find the screenshot below.
Note: I have deleted the device from UI and also from CLI Delted /var/lib/mrtg/ip_*.rrd /etc/mrtg/conf.d/ip.cfg /usr/local/nagios/share/perfdata/hostname/*
I suspect its because of the special character in interface description.

[tgriep]

To get more decimal places to display in the graph, you would have to use the Legacy graph as the Highchart graphs cannot be customized.
To switch to the Legacy version, go to the Admin > System Settings menu and click on the Theme & Display tab.
Deselect Use Highcharts for Performance Graphs page and host/service detail pages option and save the settings.

If not enough decimal places are displayed using the legacy graphs, they can be customized by going to the Admin > Manage Graph Templates menu.
Edit the check_xi_service_mrtgtraf.php template and increase the decimal places and save the settings.

The interface that is not displaying, did the rrd file get recreated in the /var/lib/mrtg folder for that interface?
If not, can you run the following command as root,

Code: Select all

LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg -debug=cfg,base &> /tmp/mrtg.txt

Then check the /tmp/mrtg.txt file for any errors on that interface and post them here?

[sarfarosh]

Hi tgriep,
Yes i can see RRD file for that interface in /var/lib/mrtg. Also /usr/local/nagios/share/perfdata/HOSTNAME/interface.xml and interface.rrd are available.
I suspect it is not showing graphs because the interface description contains special chanracter @.

[tgriep]

Delete the existing service with the "@" in it and all of the related files, then recreate the service without the "@" and see if the graph starts to populate for you.