Report Generation extremly slow/failing

[domsch1988]

Besides my Topic on a feature i'm trying to get up and running we are having a Problem with the reporting in our current XI. First the important stuff:

What version of Nagios XI are you using?
5.4.8
Linux Distribution and version?
CentOS 7.3.1611 Core
32 or 64bit?
64bit
VMware Image or Manual Install of XI?
manual install
Are there special configurations on your system, ie; is Gnome installed? Are you using a proxy? Are you using SSL?
No

Basically, generating reports has always been kind of slow for us. In recent weeks it's been getting worse. At the moment, if i want to create an Availability Report (stock configuratione, no changes to the report) for one hostgroup (24 Servers, 150 Services) for one month (January) I'm waiting about 10 Minutes for the report to show up (no matter which format).
Additionally, making it so the report should be directly send per mail (PDF), the creating fails entirely. After about 15 Minutes i get an error That Report Creation failed (the error message only says that and the complete html code for the mail). I can attach the error message although there is no information on why it failed.

We do not have any other performance issues really. Checks are running well, Web Interface is responsive, changing hosts and services works flawless. Some Data for reference:
We are running 4 cores and 8GB of Ram. About 650 Hosts, 2500 Services.
Active Service Checks per 5 Minutes: 3650
Avg Service check Latency: 0.01 sec
Avg Service Execution Time: 1.07 sec
Load Averges: 2.2 (over 1, 5 and 15)
Swap useage is around 7% Max

Basically, our Server is working, but certainly not overloaded from my feelings. Also, its running on a Full SSD SAN if this matters.
Any Help on troubleshooting this further would be highly appreciated. My Management would like to schedule the reports to be automatically send every month to nearly 90 Customers. With the current performance (if working at all) this is not possible.

[kyang]

You can try increasing the php values in the /etc/php.inifile.
Try doubling the values that you have for these 3 fields.

Code: Select all

max_execution_time = 60     ; Maximum execution time of each script, in  seconds
max_input_time = 60     ; Maximum amount of time each script may spend parsing request data
memory_limit = 256M      ; Maximum amount of memory a script may consume 

Then restart apache.

Code: Select all

service httpd restart

My Management would like to schedule the reports to be automatically send every month to nearly 90 Customers.

Do you have these scheduled reports sending right now? Is it 90 reports at one time? Or is the time when it's sent separate from each other?

[domsch1988]

You can try increasing the php values in the /etc/php.inifile.
Try doubling the values that you have for these 3 fields.

Code: Select all

max_execution_time = 60     ; Maximum execution time of each script, in  seconds
max_input_time = 60     ; Maximum amount of time each script may spend parsing request data
memory_limit = 256M      ; Maximum amount of memory a script may consume 

Then restart apache.

Code: Select all

service httpd restart

My Management would like to schedule the reports to be automatically send every month to nearly 90 Customers.

Do you have these scheduled reports sending right now? Is it 90 reports at one time? Or is the time when it's sent separate from each other?

Thanks for the input:
- Increasing the values won't speed up report generation, will it? It's only to get the email report through?
- No, they are not send all at once. Currently about 10 are set up and they are running once a month. When they run they are spread by an hour apart each because of the current performance.

I really appreciate the input on how to get the mails out, but i'd too like to figure out why those take so lone

[tgriep]

The Availability Report uses a mixture of PHP and a cgi script to generate the data so increasing the settings in the php.ini file will help out in the performance.

The failure to create the PDF file could be caused by the External URL settings in the Admin > System Settings menu.
If it is set, make sure the hostname in the field is correct and that the name is resolvable in DNS.
Or, you can use the IP address in place of the host name and see if that helps.

If not, if you can post the error, that would help out in troubleshooting the issue.

[domsch1988]

The Availability Report uses a mixture of PHP and a cgi script to generate the data so increasing the settings in the php.ini file will help out in the performance.

The failure to create the PDF file could be caused by the External URL settings in the Admin > System Settings menu.
If it is set, make sure the hostname in the field is correct and that the name is resolvable in DNS.
Or, you can use the IP address in place of the host name and see if that helps.

If not, if you can post the error, that would help out in troubleshooting the issue.

So, i set the values as posted. This helped with actually getting the report out. That's working now. It's not any faster, but it's working at all, so that's great.
My Problem is, that i can't find any clue on why the generation is that slow. I watched the system while generating a report and neither CPU, RAM or Swap are particularly hit. I'm not quite sure how i could measure/watch disk or database performance though.

[kyang]

I found this post in a previous customer support thread.

by tmcdonald
So a few things:

• I do believe your issue has to do with the number of objects being monitored, as 20k checks is usually the most I recommend for a single XI server for just this reason. Especially if your checks are running on average more than once every 5 minutes, you are going to see a lot of historical data that the reports need to dig through.
• You can somewhat mitigate the timeouts by increasing some PHP limits:
  

  Code: Select all

  max_execution_time = 60     ; Maximum execution time of each script, in  seconds
  max_input_time = 60     ; Maximum amount of time each script may spend parsing request data
  memory_limit = 256M      ; Maximum amount of memory a script may consume 

  but even that will only get you so far, and will not help too much with the speed.
  
  As for the original issue itself, we now generate reports using AJAX so the timeouts should not be due to any hanging on the frontend. It still may take a long time, but as long as your PHP settings in /etc/php.ini are high enough it should eventually complete. It's basically now just a "too much data" issue causing the slowness.
  
  We do have a developer here who has a caching component for reports in the works that should help a bit, but it will not be available until 5.3.0. I won't have many more details until then, unfortunately.

I'm not sure how many host/service checks you have. Also I don't know the time interval, but this has a little bit better explanation as to what is going on maybe?

Otherwise we have a maxmizing performance doc that you can read through.
https://assets.nagios.com/downloads/nag ... 1511880526

NOTE: If you make any changes, make sure to always have a snapshot or backup of your XI in a working state. In case you want to revert back.