PDF Report Creation extremely slow

[avocette]

I'm running Nagios 5.7.2 on a CentOS 7 VM. Virtualized on Hyper-V (Server 2016) and the VM has 8GB Ram and 4 CPU cores allocated. I found that running capacity reports was stalling for hours until I reset apache. I'm able to create a PDF successfully for a single page or small report but if it gets much beyond that (for example a 9 page PDF report) it just doesn't seem to be able to do it.

Monitoring with TOP -i, the apache process seems to be constantly busy during this time - it uses about 16 to 20% total CPU.

When I initially tried this it was actually bogging the system down to the point where checks and other monitoring data weren't updating in realtime. I followed the Nagios document to create and use a Ramdisk (500MB) and now the system is responsive during these report creation attempts but the report PDFs still don't seem to be created.

I also modified the php.ini a couple of times and the last time I went to:

max_execution_time = 1200
max_input_time = 1200
max_input_vars = 100000
memory_limit = 2048M

these numbers were suggested in another similar thread. This also has not made a difference. When I monitor the Hypervisor, the Nagios VM is not using much in terms of disk IO and there is no issue with the host not being able to provide the disk performance that's needed. Screenshot of top output below - this one is only 20 minutes into generating a report that is 12 pages (each page with 5 items)

[benjaminsmith]

Hi @avocette,

Just clarification here, for the larger report, does it not ever finish, or is the PDF corrupted or empty. If that's the case, can you attach the failed pdf to the thread?

Also, try to run the larger report, and then immediately download a system profile and send it over in a PM so we can check out the logs for errors. Thanks, Benjamin

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.

[avocette]

Just submitted profile.zip via PM.

When I ran a capacity report this morning, I found that I couldn't browse or log in on another tab - I had to restart apache in order to regain access. That wasn't the case last week - it does seem to be really using resources for this.

The reports never finish unless they're really small - they just go on for hours - I tried one for about 7 hours ( a 9 page PDF) and it never finished so I had to eventually restart apache.

Thanks,
scott

[benjaminsmith]

Hi Scott,

Thanks for the profile, it definitely was timing out before.

Counting pages (2/6)
[============================================================] Object 1 of 1
Read error (Connection timed out) in headers.
Retrying.

However, I believe Apache maybe hanging due to memory issues, what do you have the memory limit set at in the php.ini file?

Code: Select all

memory_limit = 1024M

Can you upload the following Apache configuration files for us to check?

Code: Select all

/etc/httpd/conf/httpd.conf
/etc/httpd/conf.d/ssl.conf
/etc/hosts

Can you try to run the Legacy Availablity report and let me know if see the same behavior? Go to Reports > Legacy Reports > Availability

[avocette]

Our memory_limit is at 2048M

I'll PM the requested files. The legacy reports seem to run fine - takes under 10 seconds to do fairly complex reports.

Thanks,
Scott

[benjaminsmith]

Hi Scott,

Thanks for the file and your continued patience. Since the legacy reports are working, it's most likely an issue with the php/apache setup on this server. We may have to move this over to a ticket to fully resolve, before that, I'd like to get a couple more bits of data.

The HTTP service and Apache process are stalling at times. You can see this in the nagios.log.

Aug 10 10:19:56 nagios nagios: SERVICE ALERT: AVOCETTE - RHEL7;Service Status - httpd;UNKNOWN;SOFT;1;● httpd.service - The Apache HTTP Server
Aug 10 10:20:18 nagios nagios: SERVICE ALERT: AVOCETTE - RHEL7;HTTP;CRITICAL;SOFT;1;connect to address 127.0.0.1 and port 80: Connection refused
Aug 10 10:20:35 nagios systemd: httpd.service stop-sigterm timed out. Killing.
Aug 10 10:20:35 nagios systemd: httpd.service: main process exited, code=killed, status=9/KILL

This maybe during report generation, but do you have any kind of proxy server set up in this environment? Please post the output of the following curl tests.

Code: Select all

curl -k -L -v http://127.0.0.1/nagiosxi/
curl -k -L -v https://127.0.0.1/nagiosxi/

At one time it looks like a Memcached hosts was added to the configurations but is now disabled.

Code: Select all

// MEMCACHED SETUP	
$cfg['memcached_enable'] = false; // should we use memcached or not?
$cfg['memcached_hosts'] = array('127.0.0.1', '192.x.x.3'); // one or more 

Is the Memcached extension still enabled in the php.ini file, if so try to comment that out. Can you share the php.ini file with us?
Thanks, Benjamin

[avocette]

I'd like to keep this open - I got pulled away for other work and this isn't "critical" now but I'll come back to it in the next week and see if we can get it resolved.

[benjaminsmith]

Hi,

I'd like to keep this open - I got pulled away for other work and this isn't "critical" now but I'll come back to it in the next week and see if we can get it resolved.

Sounds good. We'll be here. If you can post the php.ini file when you return, that would be very helpful.

Benjamin

[avocette]

To answer, your questions, we don't have any sort of Proxy set up. I also looked through php.ini and don't see any reference to Memcached. I'll PM the php.ini and also a couple of .txt files that contain the output of the curl commands you requested.

Thanks,
Scott

[benjaminsmith]

Hi Scot,

Ok that's good. The curl commands look fine, but go ahead uncomment the following line in the php.ini file:

Code: Select all

max_input_vars=100000

This setting is really critical for running larger reports. After making the changes be sure to restart the Apache webserver.

Code: Select all

systemctl restart httpd

Let me know if it's working better.