Performance Graphs tab freezing browser

[gwakem]

XI 5.4.8, tested in Firefox 55.0.3 64 bit and Chrome 61.0.3163.100 (Official Build) (64-bit).
When I access the performance Graphs tab of a specific service check, the browser freezes and has to be restarted. This does NOT happen with all service performance graphs, and it does not happen when I access the graph symbol to bring up the pop up window with graphs from the host, showing all services.
I do not want to loose all the data in this specific graph, is there a way to "fix" the rrd or diagnose the issue to determine next steps?

[tgriep]

It could be some Apache and PHP settings causing the issue, try doing the following on the Nagios server.

Edit the /etc/php.ini file and change the following from

Code: Select all

max_execution_time = 30
max_input_time = 60
memory_limit = 128M

to

Code: Select all

max_execution_time = 60
max_input_time = 120
memory_limit = 512M

add this to the bottom of that file

Code: Select all

max_input_vars=5000

Save the file

Then edit this file

Code: Select all

/etc/httpd/conf/httpd.conf

add this to the bottom of that file

Code: Select all

LimitRequestLine 100000

Save the file and restart Apache for the changes to take affect.

Code: Select all

service httpd restart

Then clear your browsers cache and see if that fixes the issue.

[gwakem]

It turns out I already had most of those settings adjusted at or above your recommended settings. I upped the memory_limit from 256M to 512M in the php.ini, and added LimitRequestLine 100000 to the bottom of the httpd.conf, and then cleared cache and restarted.
Unfortunately it still happened.

I used the firefox developer tools to attempt to diagnose where the slowdown was occurring, and it appears it's stalling out on the call to the gauge. Now, this check is dealing with byte size on a file on hpux, so the output is something like 2048141587, and I suspect that the gauge is having a liiiiiiiiittle issue handling that. Here is the call itself that makes the browser start asking if I want to stop the scripts running that are slowing things down:

Code: Select all

https://$SERVER/nagiosxi//includes/dashlets/gauges/getdata.php?host=$HOST&service=NRPE%20-%20DRY%20TRAN.DAT&ds=_triprod_dry_TRAN.DAT

Any idea on how best to handle this?

A interesting side note, I was watching my memory. By the time I closed the page after about 40 seconds or so, the used memory on my system had ballooned from the normal 3.5GB to 10.6GB. Killing the perfdata tab immediately returned my memory usage back to 3.5GB.

[tgriep]

It could be that the rrd file is corrupted on the system and the following process is a way to dump the rrd file and restore it back as an rrd file to see if that fixes it.
Run the following commands and replace the text between the < > with valid names.

Code: Select all

cd /usr/local/nagios/share/perfdata/<hostname>
rrdtool dump <original servicename>.rrd > temp.xml
mv <original service>.rrd  backup.rrd
rrdtool restore temp.xml <original servicename>.rrd

Then see if you can view the Graph.
If not, can you post the .rrd and .xml file here so we can view them?

[gwakem]

Unfortunately, that didn't do the trick. I verified the permissions on the file were good, but its displaying the same behavior. One thing I should note: when using the graph link next to the service itself, (when looking at the host with all services below the host, there is the red and blue lines that you can click to open the graph in a pop up window,) it doesn't freeze and you can adjust to view specific lengths of time with no issues. The RRD displays fine there. It's only when using the performance graphs tab does it freeze when attempting to pull the gauge unit.

[gwakem]

I have attached the RRD and XML.

[dwasswa]

Hi @ gwakem,

Try running this command

Code: Select all

tail -f /var/log/httpd/*_log

while you try viewing the graph and post the output.

[gwakem]

I have attached two files: one that shows the entirety of the output, which is busy as multiple people are hitting it, and a second one (the http_output_filtered.txt) that greps based on my workstation's IP.

[dwasswa]

Hi @gwakem,

Can you please run the following comands and post the output.

Code: Select all

ls /usr/local/nagios/var/spool/perfdata/ | wc -l
ls /usr/local/nagios/var/spool/xidpe/ | wc -l

[gwakem]

Here you go:

Code: Select all

# ls /usr/local/nagios/var/spool/perfdata/ | wc -l
3
# ll /usr/local/nagios/var/spool/perfdata/
total 28K
drwxr-xr-x 2 nagios nagios 131 Oct 16 15:42 .
drwxr-xr-x 5 nagios nagios  52 Feb 18  2016 ..
-rw-r--r-- 1 nagios nagios   0 Feb 22  2016 1456142744.perfdata.host-PID-20394
-rw-r--r-- 1 nagios nagios   0 Feb 22  2016 1456142744.perfdata.service-PID-20395
-rw-r--r-- 1 nagios nagios 25K Mar 23  2016 1458754280.perfdata.host-PID-3348
# ls /usr/local/nagios/var/spool/xidpe/ | wc -l
0