Perfdata not graphing

[bomahony]

And it is me again!
So in one of my environments out of the 4 i am building concurrently, it seems some checks have stopped graphing. In a weird way these seems to be the same checks for all machine of that node type - eg Netowrk bandwith for X server node type. Disk IO for a different node type [even though we use a generic nrpe.cfg for both these node types, that is the same on all!]. I noticed this by checking ofr the error on the XML files: found extra data on update argument.

I am going to put it down to "weirdness" during the build phase, as this seems to have happened back 2mo ago. So here are my questions, and hopefully what I have as a solution also:
#1 This looks like a solution: https://support.nagios.com/kb/article.php?id=149
Is it still valid? Or should I just delete the RRD files? TBH they are 2mo old data so are useless now anyway. Just wondering for if this happens again.
#2 Is there any way NagiosXI can tell us when it has stopped graphing stuff? [With 3500 or so nodes across these environs with 20-30K checks in each, it could happen a bit ] Or should I use FIM or file-age solution to ensure RRD files are updating?

[npolovenko]

Hello, @bomahony. Yes, you can still use that KB article when the number of data sources have changed. Although, simply deleting the old rrd file from /usr/local/nagios/share/perfdata/ and letting Nagios recreate it would be much faster.
If you decide to use the script please make a backup of the XI first:
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

I haven't heard of a solution to monitor individual graphs to see when they stop graphing. However, you could probably use the check_log plugin to search the /usr/local/nagios/var/perfdata.log for new errors and alert when the plugin finds some.
https://support.nagios.com/kb/article/l ... s-787.html

We're also planning on adding a more dynamic graphing engine in XI 6. Nothing concrete yet, but it is on our roadmap:
https://www.nagios.com/roadmaps/

[bomahony]

I just deleted the RRDs for now. The old data wasnt required.


Last Quick Question on rrd data. Lets say some idiot went and renamed a check. This creates a new RRD file. Is is possible to just copy the previous data in so it is not lost [provided the check perfdata names dont change].

Actually that was second last. If I change a perfdata name, but not the check name will that matter? [My Net throughput check doesnt have the actual bon names on the 20G NIC bonds, making it hard to troubleshoot]. It doesnt really matter this time, as I have just fixed the rrd anyway, and we are only beginning perf testing. This is more for a just in case later.

[bomahony]

Also I am impressed with the V6 roadmap, but it is still far far away!

[npolovenko]

@bomahony, If you change a service name in XI it will force it to create the new RRD and same applies if you change the RRD name without changing the service name.
If you want to change a service name and keep the graph simply rename the corresponding RRD and XML files.
If someone renamed the service and its been running for a while you can combine the new graph(rrd file) with the old one using the Performance Data Tool:
https://exchange.nagios.org/directory/A ... ol/details
Or using something like this:
https://gist.github.com/arantius/2166343

[bomahony]

Thanks for that. And what happens if the perfdata that is being reported changes, without the check name changing.

[ssax]

perfdata that is being reported changes

Please explain more thoroughly of what you mean by that.

If the number of perfdata metrics changes, you'll experience the issue you did earlier with the data not being inserted into the RRD. Is that what you're asking?

[bomahony]

My issue is the check_eth_all script is not showing the interface name. I am going to edit the perl script to put the $iface var before each perfdata value

I am assuming that this will impact the graphs? I havent had time to test yet due to other commitments

[scottwilkerson]

I'm not familiar with the plugin check_eth_all script, however you can change the label in the performance output per the following guidelines:

https://nagios-plugins.org/doc/guidelines.html

[bomahony]

Yeah I can change the label alright, I was wondering on the impact of the historical data. However after recent screw ups by other teams involving not using the 10G networking correctly, the data is rubbish and I no longer am interested in that data so you can close this