missing perfdata -- again

[clauskc]

Hi,

We have several hosts where some counters, all related to disk, on performance graphs and capacity planning it stopped showing data after 20 June.
Have had same issue before: https://support.nagios.com/forum/viewto ... 16&t=43171

Now its a bigger problem since its on many servers the problem is.
Checks are running as they should and data returned, an rrd files are updated.

I checked the perfdata.log and see it has not had a entry in the log since 17-05-2017??

Looked at npcd.log are getting entries but no problems
Looked at nagios.log there are no problems with executing the checks with missing data.

/Claus

[tgriep]

Can you edit the following file

Code: Select all

/usr/local/nagios/etc/pnp/process_perfdata.cfg

Change the following line to the following fro enabling debugging

Code: Select all

LOG_LEVEL = 2

Save the file and restart these processes

Code: Select all

service npcd restart
service nagios restart

Wait for about 30 minutes and email in the log file

Code: Select all

/usr/local/nagios/var/perfdata.log

Thanks

[clauskc]

Have PM the requested log to you.

[tgriep]

For some reason, the log file did not get attached to the PM, can you send it again or post it here?

[clauskc]

Have send it again, did not notice the 1mb limit in the first pm.

[tgriep]

I received the log file but I didn't see and errors or issues for when the performance data is updated in to the RRD files.
It looks like it they are getting updated.
I would have to know the names of the host and service that is exhibiting the issue and also the rrd and xml files for the checks.

It might be an issue with viewing them and not updating them.
Run the following as root on the Nagios server.

Code: Select all

tail -f /var/log/httpd/*_log

Then in the GUI, view one of the graphs that is not updating and post the output of the tail command so we can check it for any errors.

[clauskc]

The files Changed stamp is 20/6, so does not seem to be updated with data.
Are there a max file size or number of records for the RRD files, can see all the files not being updated are over 4mb?


Host: pfanpbisql01 / service: PFA Free Diskspace

The requested files and logs are attached.

[tgriep]

Thanks for the files, I found the reason they are not updating.

2017-07-18 07:59:33 [18446] [2] RRDs::update /usr/local/nagios/share/perfdata/pfanpbisql01/PFA_Free_Diskspace.rrd 1500357545:79:39:11:67:14:99:6:40:19:24
2017-07-18 07:59:33 [18446] [1] RRDs::update ERROR /usr/local/nagios/share/perfdata/pfanpbisql01/PFA_Free_Diskspace.rrd: expected 15 data source readings (got 11) from 1500357545

It looks like the check was changed which caused the number of data sources to change. When this happens, the RRD file would have to be deleted so they will be recreated with the correct number of data sources or you can follow this KB article to update the existing files.
https://support.nagios.com/kb/article/n ... phing.html

If that KB article doesn't work, you could try this one.
https://support.nagios.com/kb/article/n ... -file.html

[clauskc]

Ok, that make sense why all the same disk counters on some host went bad at the same time.

We are running on RHEL 6.9 now, as I read the articles we should be over this problem with RRDs not updating if we go to RHEL 7.+
Is that correct?

[tgriep]

If the expected data sources are incorrect, it doesn't matter if the OS is Centos6 or Centos7.
One thing you need to note, if the Centos6 system is 32 bit and Centos7 system is 64 bit, then all of the RRD files would have to be converted.
See this KB article.
https://support.nagios.com/kb/article/n ... -data.html