rrd files not creating since migration.

[danniiffxi]

Hi Guys

OK so since migrating one of our Nagios servers from CentsOS 6 to CentOS 7 whenever we add a new bandwidth check, the system is failing to create the rrd files that hold the information. So we get a lot of the following alerts:


All existing graphs created prior to the migration are populating just fine.

I have run the following

Code: Select all

ls -l /var/lib/mrtg
LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg

and get these results

Code: Select all

-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 192.168.254.99_10625.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 192.168.254.99_10626.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 192.168.254.99_10627.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 192.168.254.99_10628.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 192.168.254.99_10.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 192.168.254.99_40.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 192.168.254.99_5001.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 192.168.254.99_5179.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 192.168.254.99_5182.rrd
-rwxrwxr-x 1 apache nagios      0 Nov 20 16:11 mrtg.ok
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 wxswocncr16_10101.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 wxswocncr16_10102.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 wxswocncr16_10103.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 wxswocncr16_10104.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 wxswocncr16_10105.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 wxswocncr16_10148.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 wxswocncr16_10149.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 wxswocncr16_12002.rrd
-rw-rw-r-- 1 apache nagios 105320 Nov 20 16:11 wxswocncr16_250.rrd
You have new mail in /var/spool/mail/root
[root@wxlnxnagxip02 mrtg]# LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg
SNMPGET Problem for ifHCInOctets.250 ifHCOutOctets.250 on svc-snmpexec@wxswocncr16:161::::2:v4only: No response from remote host "wxswocncr16" at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 594.
        Net_SNMP_util::snmpget('svc-snmpexec@wxswocncr16:161::::2:v4only', 'HASH(0x2bcd7e0)', 'ifHCInOctets.250', 'ifHCOutOctets.250') called at /usr/bin/mrtg line 2330
        main::getsnmparg('HASH(0x27e3ca0)', 'HASH(0x35499c8)', 'HASH(0x26db538)', 'HASH(0x2b818a8)') called at /usr/bin/mrtg line 2510
        main::readtargets('HASH(0x27e3ca0)', 'ARRAY(0x286ddb0)', 'HASH(0x26db538)') called at /usr/bin/mrtg line 403
        main::main called at /usr/bin/mrtg line 143
2020-11-20 16:13:22: WARNING: skipping because at least the query for ifHCInOctets.250 on  wxswocncr16 did not succeed
2020-11-20 16:13:22: WARNING: no data for ifHCInOctets&ifHCOutOctets:svc-snmpexec@wxswocncr16. Skipping further queries for Host wxswocncr16 in this round.
SNMPGET Problem for ifHCInOctets.10151 ifHCOutOctets.10151 on svc-snmpexec@192.168.251.11:161::::2:v4only: No response from remote host "192.168.251.11" at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 594.
        Net_SNMP_util::snmpget('svc-snmpexec@192.168.251.11:161::::2:v4only', 'HASH(0x5e49160)', 'ifHCInOctets.10151', 'ifHCOutOctets.10151') called at /usr/bin/mrtg line 2330
        main::getsnmparg('HASH(0x27e3ca0)', 'HASH(0x5e4bef0)', 'HASH(0x26db538)', 'HASH(0x2b818a8)') called at /usr/bin/mrtg line 2510
        main::readtargets('HASH(0x27e3ca0)', 'ARRAY(0x286ddb0)', 'HASH(0x26db538)') called at /usr/bin/mrtg line 403
        main::main called at /usr/bin/mrtg line 143
2020-11-20 16:13:22: WARNING: skipping because at least the query for ifHCInOctets.10151 on  192.168.251.11 did not succeed
2020-11-20 16:13:22: WARNING: no data for ifHCInOctets&ifHCOutOctets:svc-snmpexec@192.168.251.11. Skipping further queries for Host 192.168.251.11 in this round.
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on svc-snmpexec@192.168.250.121:161::::2:v4only: No response from remote host "192.168.250.121" at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 594.
        Net_SNMP_util::snmpget('svc-snmpexec@192.168.250.121:161::::2:v4only', 'HASH(0x706ec50)', 'ifHCInOctets.1', 'ifHCOutOctets.1') called at /usr/bin/mrtg line 2330
        main::getsnmparg('HASH(0x27e3ca0)', 'HASH(0x70719d0)', 'HASH(0x26db538)', 'HASH(0x2b818a8)') called at /usr/bin/mrtg line 2510
        main::readtargets('HASH(0x27e3ca0)', 'ARRAY(0x286ddb0)', 'HASH(0x26db538)') called at /usr/bin/mrtg line 403
        main::main called at /usr/bin/mrtg line 143
2020-11-20 16:13:22: WARNING: skipping because at least the query for ifHCInOctets.1 on  192.168.250.121 did not succeed
2020-11-20 16:13:22: WARNING: no data for ifHCInOctets&ifHCOutOctets:svc-snmpexec@192.168.250.121. Skipping further queries for Host 192.168.250.121 in this round.
SNMPGET Problem for ifHCInOctets.251 ifHCOutOctets.251 on svc-snmpexec@192.168.251.31:161::::2:v4only: No response from remote host "192.168.251.31" at /usr/bin/../lib/mrtg2/Net_SNMP_util.pm line 594.
        Net_SNMP_util::snmpget('svc-snmpexec@192.168.251.31:161::::2:v4only', 'HASH(0x54bab10)', 'ifHCInOctets.251', 'ifHCOutOctets.251') called at /usr/bin/mrtg line 2330
        main::getsnmparg('HASH(0x27e3ca0)', 'HASH(0x54bd860)', 'HASH(0x26db538)', 'HASH(0x2b818a8)') called at /usr/bin/mrtg line 2510
        main::readtargets('HASH(0x27e3ca0)', 'ARRAY(0x286ddb0)', 'HASH(0x26db538)') called at /usr/bin/mrtg line 403
        main::main called at /usr/bin/mrtg line 143

Any ideas?

[vtrac]

Hi danniiffxi,
Could you please run the "rpm" command below and see if you have those rrd packages installed.

rpm -qa | grep rrd
rrdtool-perl-1.4.9-1.el7.x86_64
rrdtool-1.4.9-1.el7.x86_64
rrdtool-python-1.4.9-1.el7.x86_64

If you do, hope that you don't mind rerun the wizard again for that new "192.168.251.249" server you would like to add.

After completed, you should see a "192.168.251.249.cfg" file under "/etc/mrtg/conf.d/" folder.

Also, to further investigate the issue, could you please send me profile.zip and the exact name of the host and services sending notications.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.

Best Regards,

Vinh

[danniiffxi]

Hi Vinh

Thanks for getting back to me.

This looks good

Code: Select all

[root@nagxip02 ~]# rpm -qa | grep rrd
rrdtool-python-1.4.9-1.el7.x86_64
rrdtool-perl-1.4.9-1.el7.x86_64
rrdtool-1.4.9-1.el7.x86_64
You have new mail in /var/spool/mail/root
[root@nagxip02 ~]#

This file is in place, and has been for some time, the checks being added are for switch ports that we are now starting to use on this swtch

Code: Select all

[root@nagxip02 ~]# cd /etc/mrtg/conf.d/
You have new mail in /var/spool/mail/root
[root@nagxip02 conf.d]# ls -l | grep 192.168.251.249
-rw-r--r-- 1 root root 201422 Nov 12 15:13 192.168.251.249.cfg
[root@nagxip02 conf.d]#

Now on a side note, If I manually create a new blank rrd to match the name one one that does not exist, file the check comes back as OK and after about 25 mins it starts to populate. It's as if the system does not have permissions to create the new file.

Before I forward you my system profile, is there a forum mod who can vouch for you? The system profile contain hashed passwords and all sorts of sensitive info about the company I work for, and your account looks like that of a standard user. Security would have me fired if I sent that info to anyone other than an employee of Nagios.

[benjaminsmith]

Hi, @danniiffxi,

Before I forward you my system profile, is there a forum mod who can vouch for you?

Vihn is a member of the support team here at Nagios. You can PM to the profile to myself or Vihn.

Thanks,
Benjamin

[vtrac]

Hi danniiffxi,
Yes, sounds like you might have permission issue.
The "/var/lib/mrtg", "/etc/mrtg" and its sub-folders should belong to "apache:nagios" so that user "apache" (Nagios XI) can write|update|create those files.

Below is an example of what I have in my system:

Code: Select all

[root@centos01 conf.d]# ls -la /etc/mrtg
total 16
drwxrwxr-x.  4 apache nagios   66 Nov  5 14:54 .
drwxr-xr-x. 96 root   root   8192 Nov 18 16:23 ..
drwxrwxr-x.  2 apache nagios    6 Nov  2 11:15 archive
drwxrwxr-x.  2 apache nagios   30 Nov  5 14:54 conf.d
-rwxrwxr-x.  1 apache nagios  788 Nov  2 11:15 mrtg.cfg
-rw-r--r--.  1 apache apache    0 Nov  5 14:54 mrtg.ok

[root@centos01 conf.d]# ls -la /etc/mrtg/conf.d/
total 60
drwxrwxr-x. 2 apache nagios    30 Nov  5 14:54 .
drwxrwxr-x. 4 apache nagios    66 Nov  5 14:54 ..
-rw-r--r--. 1 apache apache 61260 Nov  5 14:54 192.168.5.90.cfg

Please run the below commands to set permission on those two folders.

Code: Select all

As "root":
chown -R apache:nagios /etc/mrtg
chown -R apache:nagios /var/lib/mrtg

As to my Nagios user ID, my boss has just updated my login info.

Best Regards,
Vinh

[danniiffxi]

Hi Vinh

Both folders are already set to 'apache nagios' as well as all the files within those folders.

Code: Select all

[root@nagxip02 ~]# ls -l /etc | grep mrtg
drwxrwxr-x   4 apache nagios     87 Nov 23 09:48 mrtg

Code: Select all

[root@nagxip02 ~]# ls -l /var/lib/ | grep mrtg
drwxrwxr-x  2 apache nagios      487424 Nov 24 08:16 mrtg

I'll send over my profile now many thanks


I was also just doing a side by side of both servers, every switch port on every switch is the same.

So the server on the left is our current production server NXI 5.7.1 CentOS 6, the server on the right is the replacement I am building and the server which this thread is about. running NXI 5.7.1 CentOS 7...

Original working server rrd installation and service check test

Code: Select all

[root@nagip02 ~]# rpm -qa | grep rrd
rrdtool-python-1.3.8-10.el6.x86_64
rrdtool-1.3.8-10.el6.x86_64

Manually running the check from the CLI

Code: Select all

[root@nagip02 ~]# /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/192.168.252.243_10110.rrd -w 500.0,500.0 -c 800.0,800.0 -l M
OK - Current BW in: .01Mbps Out: 0Mbps|in=.011425Mb/s;500.0;800.0 out=.003632Mb/s;500.0;800.0

New broken server rrd installation and service check test
New broken server rrd installation

Code: Select all

[root@nagxip02 mrtg]# rpm -qa | grep rrd
rrdtool-python-1.4.9-1.el7.x86_64
rrdtool-perl-1.4.9-1.el7.x86_64
rrdtool-1.4.9-1.el7.x86_64

Manually running the check from the CLI

Code: Select all

[root@nagxip02 libexec]$ /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/192.168.252.243_10110.rrd -w 500.0,500.0 -c 800.0,800.0 -l M
OK - Current BW in: 0Mbps Out: 0Mbps|in=0Mb/s;500.0;800.0 out=0Mb/s;500.0;800.0

The production server on the left was backed up, the data was then transferred to the new server where I followed the documentation and performed a restore and install of the XI Tarball to fix any issues as per this document.
https://assets.nagios.com/downloads/nag ... ios-XI.pdf



So it looks like rrd is completely FUBAR, as it's not just the new files being created..

[vtrac]

Hi danniiffxi,
The first thing I noticed from your reply on 11/22/2020 was the "/etc/mrtg/conf.d/192.168.251.249.cfg" file was owned by "root". That's why I had you run the "chown -R" command to make sure all files under "mrtg" are owned by "apache" .... same for "/var/lib/mrtg" folder.

Your settings was owned by "root" on 11/22/2020:

Code: Select all

[root@nagxip02 ~]# cd /etc/mrtg/conf.d/
You have new mail in /var/spool/mail/root
[root@nagxip02 conf.d]# ls -l | grep 192.168.251.249
-rw-r--r-- 1 root root 201422 Nov 12 15:13 192.168.251.249.cfg
[root@nagxip02 conf.d]#

Did you run the below commands after doing the restore?

Code: Select all

Changed Operating System Version / Architecture / Family
If you are restoring a backup from a different OS version, architecture or family this can be a problem
because the backup is overwriting the compiled binaries. To fix this you will need to execute the following
commands:

cd /tmp/
wget https://assets.nagios.com/downloads/nagiosxi/scripts/restore_repair.sh
chmod +x restore_repair.sh
./restore_repair.sh

Also, I assumed that your old CentOS 6 was a (x86), you must manually export those rrd files to XML, then import them onto Nagios since the architecture are different between CentOS 6 (x86) and CentOS 7 (x86_64).

Please follow the steps outlined in the KB article below:
https://support.nagios.com/kb/article.php?id=166

Hope this helps.

Regards,
Vinh

[danniiffxi]

Hi Vinh

Hi danniiffxi,
The first thing I noticed from your reply on 11/22/2020 was the "/etc/mrtg/conf.d/192.168.251.249.cfg" file was owned by "root". That's why I had you run the "chown -R" command to make sure all files under "mrtg" are owned by "apache" .... same for "/var/lib/mrtg" folder.

Your settings was owned by "root" on 11/22/2020:

All 16k files in mrtg/conf.d are now apache nagios, after this change is rebooted the server. However graphing is still not working and doing the same as it was before.

Code: Select all

[root@nagxip02 conf.d]# ls -l | grep 192.168.251.249
-rw-r--r-- 1 apache nagios 201422 Nov 12 15:13 192.168.251.249.cfg

Yea I ran the Tarball to replace the binaries. probably worth noting that this is the second instance I have migrated from CentOS 6 to 7.. I never had this graphing issue on the first server.

Did you run the below commands after doing the restore?
CODE: SELECT ALL
Changed Operating System Version / Architecture / Family
If you are restoring a backup from a different OS version, architecture or family this can be a problem
because the backup is overwriting the compiled binaries. To fix this you will need to execute the following
commands:

cd /tmp/
wget https://assets.nagios.com/downloads/nag ... _repair.sh
chmod +x restore_repair.sh
./restore_repair.sh

Our CentOS 6 infrastructure is x64 architecture. we don't run any 32bit Linux software

Code: Select all

[root@nagip02 ~]# rpm -qf /etc/redhat-release
centos-release-6-10.el6.centos.12.3.x86_64

[danniiffxi]

IT'S ALIVE!!!

I'm not sure what fixed it, but it seem to be working now. After I changed the permission this morning nothing seemed to happen. Then around 12:00 UK time I thought back to an issue I had a couple of years ago that was similar. I cleared a rrdcache, made sure /usr/local/nagiosxi/rrdcache had the correct permission, run a DB repair, and 30 mins later the graphs started to populate.

Code: Select all

systemctl stop rrdcached
rm -f /usr/local/nagiosxi/rrdcache/*
chown nagios:nagios /usr/local/nagiosxi/rrdcache
/usr/local/nagiosxi/scripts/reconfigure_nagios.sh
/usr/local/nagiosxi/scripts/repair_databases.sh nagios
/usr/local/nagiosxi/scripts/repair_databases.sh nagiosql
systemctl start rrdcached
systemctl restart nagios.service

Thanks for your help on this Vinh

[vtrac]

Wonderful!! ... It was a pleasure working with you ...