Check_rrdtraf returning 0 following upgrade

[jsharris]

We recently (11/15/2016) upgraded Nagios XI to 5.3.2 and noticed yesterday that all of our services using check_rrdtraf are returning 0MB in and out. I am not 100% sure on the previous version; 5.2.1 or 5.2.2?

Code: Select all

OK - Current BW in: 0Mbps Out: 0Mbps

Was there a change in how the new version handles this check?

[bwallace]

This is the first I've heard of this behaviour regarding 5.3.2 and check_rrdtraf. Can you run the test check command from the XI UI successfully? How about manually running check_rrdtraf via CLI?

If those tests work fine, then I'd suspect a problem with a file(s) in /var/lib/mrtg. These are updated by the mrtg cron tab file and updates to these files would cease if there is problem with permissions, etc
In that case, please run the following commands from the XI server and post the output:

Code: Select all

ls -l /var/lib/mrtg
LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg

[jsharris]

Results from check test in XI UI:

Code: Select all

OK - Current BW in: 0Mbps Out: 0Mbps|in=0Mb/s;50;80 out=0Mb/s;50;80

From CLI:

Code: Select all

[root@spnagios ~]# /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.115.0.5_10101.rrd -w 50,50 -c 80,80 -l M
OK - Current BW in: 0Mbps Out: 0Mbps|in=0Mb/s;50;80 out=0Mb/s;50;80
[root@spnagios ~]#

Results of LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg:

Code: Select all

ERROR: Line 777 (Target[10.115.0.5_10101]: 10101:[email protected]:::::1) in CFG file (/etc/mrtg/conf.d/10.115.0.5.cfg)
contains a duplicate definition for target[10.115.0.5_10101].
First definition is on line 4840

The error is a bit confusing as it references lines 777 and 4840 but the /etc/mrtg/mrtg.cfg file is only 33 lines and /usr/bin/mrtg only 2748. Both mrtg files and the rrd files show a last changed date of 11/15/2016.

Attached is a screenshot of one the bandwidth graphs showing the drop to 0 on the same date.

[bwallace]

Thanks for those details, there was a recent problem with mrtg on Cent/RHEL 7 with particular XI versions.

Can you post the output of:
cat /etc/cron.d/mrtg

Hopefully it won't look like this:

Code: Select all

*/5 * * * * root LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok

If it does then graphs can show 0Mb due to /var/lock being tmpfs if OS in Cent/RHEL 7.
Thing is, that should be fixed in 5.3.2!! --- "- Fixed mrtg lock directory used in cron job to take volatile tmpfs directories into consideration -BH, JO
"
https://assets.nagios.com/downloads/nag ... NGES-5.TXT

[jsharris]

We get this:

Code: Select all

/*/5 * * * * root LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lib/mrtg/mrtg.lock --confcache-file /var/lib/mrtg/mrtg.ok

Also, I found this morning that one of my coworkers has applied the 5.3.3 upgrade. Behavior remains the same.

[bwallace]

Ok, between your recent post and the earlier one where we saw "...contains a duplicate definition," we'll have to examine all of the /etc/mrtg/conf.d/ directory. Can you zip that up and post it? Note that it may contain data your organization considers sensitive. In that case you can PM it to me.

Note our upcoming holiday office hours https://support.nagios.com/forum/viewto ... =3&t=41271

[dwhitfield]

Could you PM me or @bwallace your profile? Right now I'm not sure there is anything in it that will be useful, but it should speed things up come Monday (if Monday seems like a long way off, note the link in the previous post).

If you do PM either of us the profile, please update this thread. Updating this thread will bring it back on the dashboard so it doesn't get lost in the post-Thanksgiving shuffle. Thanks!

[jsharris]

Good morning and I hope everyone had a great holiday weekend. I will be PMing bwallace the conf.d folder in a few moments. I tried to download my system profile but received the following:

Code: Select all

PROFILE BUILD FAILED
Array
(
)

CODE:1

I have copied the profile info to a text file and will PM that as well.

NOTE: Files received. Have posted to shared folder internally.

[dwhitfield]

The profile build issue is likely occurring because the required sudoers entries are missing.

The following steps shows you how to get the correct sudoers entries from the XI installation file (some of these you may already have done, but I am leaving for completeness). It is not possible to just list the entries as they may change in newer versions of XI (at least not if we are thinking about others using this forum post in the future ). The following steps ensure you get the correct entries for your version of Nagios XI.

First you must download the XI installation file to your Nagios XI server. You'll do this with the WGET command. Please refer to the following link to get the correct download link for your version of Nagios XI:

Download Page - Nagios XI Versions
Open an ssh session to your Nagios XI host.

Execute the following commands (use your XI version download link in the wget command):

Code: Select all

cd /tmp
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.3.3.tar.gz

Once downloaded you'll need to use the name of the downloaded file in the next command, in this example xi-5.3.3.tar.gz is the name being used:

Code: Select all

tar xzf xi-5.3.3.tar.gz nagiosxi/nagiosxi/nagiosxi.sudoers --strip-components 2

This will have extracted a file called nagiosxi.sudoers and this file contains all the correct entries.

Run all these commands to fix your /etc/sudoers file to make sure it has all the correct entries:

Code: Select all

grep -v NAGIOSXI /etc/sudoers > /etc/sudoers.new
mv -f /etc/sudoers.new /etc/sudoers
rm -rf /etc/sudoers.d/nagiosxi
sed -i 's/^Defaults    requiretty/#Defaults    requiretty/g' /etc/sudoers
cat /tmp/nagiosxi.sudoers >> /etc/sudoers
chmod 440 /etc/sudoers

After making these changes try and download the system profile, your problem should be resolved.

Of course, that doesn't solve your original issue, but it should help you get back on track.

[jsharris]

Tried the above but the profile download produces the same error. Do I need to restart any services or the server for the changes to be reflected?