RESOLVED - No Performance Graphs, data is being processed

[gregbeyer]

Hi, I'm running XI 2024ver 1.1.3, and I'm getting no performance charts generated. Working off of this document: https://support.nagios.com/kb/article/n ... ems-9.html

Performance Data is green.

I can see files coming into /usr/local/nagios/var/spool/xidpe and disappearing after a few seconds, so it seems perfdata is being processed.

Looking in /usr/local/nagios/var/spool/perfdata/ | wc -l the file count just grows to 2 - 4 file but then drops to 0

I have logging turned up on /usr/local/nagios/etc/pnp/process_perfdata.cfg

tail -f /usr/local/nagios/var/perfdata.log, I see lines being processed, .rrd files being updated. No errors shown.

in /var/log/nagios/perfdata I've got <hostname> directories containing .rrd and .xml files of today's date, with minutes-old timestamps.

So it seems that data is being processed all the way around, I'm just not getting any host or service graphs generated.

What else to check?

[ajcoil]

Hello @gregbeyer,

Thanks for reaching out!

How long had your hosts/services been running before checking the performance graphs?

[kg2857]

Delete the rrd and xml files.

[gregbeyer]

I waited at least 30 minutes for something to show up. It's now been 36 hours, with still nothing showing in any graphs.

I started out troubleshooting by deleting everything in /var/log/nagios/perfdata. Day before yesterday. New folders for hosts, and rrds and xml were regenerated. Everything is fresh date/time stamps, so no old files hanging around.

Nothing backing up in /usr/local/nagios/var/spool/xidpe

These are what I'm seeing in my perfdata.log. If I then go check host atl1-1-02-017-11-1-mgt I find "No performance graphs were found for this host."

2024-06-20 10:37:19 [559548] [2] RRD Datatype is GAUGE
2024-06-20 10:37:19 [559548] [2] Template is check-host-alive.php
2024-06-20 10:37:19 [559548] [2] data2rrd called
2024-06-20 10:37:19 [559548] [2] RRDs::update /var/log/nagios/perfdata/atl1-1-02-017-11-1-mgt/_HOST_.rrd 1718894230:8.091000:0
2024-06-20 10:37:19 [559548] [2] /var/log/nagios/perfdata/atl1-1-02-017-11-1-mgt/_HOST_.rrd updated
2024-06-20 10:37:19 [559548] [2] Processing Line 807
2024-06-20 10:37:19 [559548] [2] Datatype set to 'HOSTPERFDATA'
2024-06-20 10:37:19 [559548] [1] Found Performance Data for atl1-1-02-018-15-1-ib / _HOST_ (rta=0.094000ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)

[gregbeyer]

Delete the rrd and xml files.

@kg2857 -- I assume you are suggesting to delete the rrd and xml file that are in /var/log/nagios/perfdata/<hostname>, correct?

I have done so for one of my hosts. New service rrd and xml _HOST_.rrd and _HOST_.xml are re-appearing.

So if, whenever I have problems with charts I just wholesale delete rrd and xml files, am I not loosing the accumulated perfdata that create the perfdata graphs? After months or more, that would be painful to loose all of that history.

There must be something less precipitous to try that wiping all historical data. Nagios team?

[gregbeyer]

Since the files in perfdata aren't doing me any good, I went ahead and deleted everything in /var/log/nagios/perfdata/, cycled nagios and npcd.

Host directories and their contents have regenerated afresh. After running overnight, still no performance graphs.

I'm tailing both npcd.log (debug) and perfdata.log. Seeing files processed and zero errors.

What's next to troubleshoot? I've done everything in the troubleshooting document ref'd in my OP.

Thanks.

[jmichaelson]

Since the files have been created after you deleted them, I'm going to assume that they're being changed as the logs are reflecting that they are. can you verify that.

In the meantime, I don't see a mention of what Distro you're using. Could you share? We have a couple of distro-specific mrtg bugs in the pipeline and I'm wondering if this is related to one of them or to something else.

[gregbeyer]

Yes, that's exactly what's happening. The newly created files continue to be updated as new perfdata is processed.

We're using RHEL 9.3.

Have I checked all I can check? What would be the reason for perfdata processing without errors shown in the log, but no graphs?

[swolf]

Hi @gregbeyer,

First, I can confirm that @kg2857 is giving you reasonable advice - currently, the performance data processing backend isn't robust to changes in the order or quantity of performance data points in a check, so if someone updates their check to, say, track an additional hard drive, that can break the graph. Many users find that deleting the existing RRD is easier than migrating the schema.

The second thing I'm noticing is that you mentioned your performance data is in /var/log/nagios/perfdata. I can't guarantee that this is the source of your problems, but it is an abnormal place to store this data. It might be worth double-checking whether moving some of the performance data back to /usr/local/nagios/share/perfdata and updating your config.inc.php "fixes" the issue. That would still be a bug that we'd want to address, but we don't thoroughly test abnormal filesystem setups internally. If you do have working graphs on some hosts or services, this probably isn't worth investigating.

The last thing I'd recommend is that you open a case on Answer Hub. While most of the answers here are coming from Nagios staff, this forum is intended to be a "community support" tier. The ticket support system will allow the Support Team to dig into your environment and help you troubleshoot exactly what's happening.

-Sebastian

[gregbeyer]

Thanks @swolf,

I've changed config.inc.php back to /usr/local/nagios/share/perfdata, cycled all services. Nevertheless, .rrd files are still being updated in /var/log/nagios/perfdata. So it would seem that something more than the config.inc.php and nagios.cfg control where performance files for graphs are generated and used. AFAIK there are only two files in play for performance data and graphs. Are there any others I need to check, ensure consistent with these?

Here's my config.inc.php showing the performance file paths:

"pnp" => array(
"perfdata_dir" => "/usr/local/nagios/share/perfdata",
// "perfdata_dir" => "/var/log/nagios/perfdata",
"share_dir" => "/usr/local/nagios/share/pnp",

Here's my nagios.cfg showing the performance file paths:

# PNP settings - bulk mode with NCPD
process_performance_data=1
# service performance data
service_perfdata_file=/usr/local/nagios/var/service-perfdata
#service_perfdata_file=/var/log/nagios/perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\>
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
# host performance data
host_perfdata_file=/usr/local/nagios/var/host-perfdata
#host_perfdata_file=/var/log/nagios/perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tH>


I have no more ticket allowances with answerhub.