Reports Extremely Slow

[dbcummings]

When running any reports in Nagios XI, the generation of the reports are very slow. I have tried updating the following in /etc/php.ini file with no impact:

max_execution_time
max_input_time
memory_limit

[benjaminsmith]

Hello @dbcummings,

To help troubleshoot the issue, I'd like to ask you a few questions and also get a fresh copy of you system profile, so we can review the log files.

1. Did this just start happening and have you made any changes to the server?

2. Is this issue impacting all of the reports or only specific reports, if so which ones?

3.Did you follow our guide below for optimizing the php settings?

Nagios XI - Optimizing The PHP Settings File

4. Please PM you system profile. Thanks.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share this in a private message and then reply to this post to bring it up in the queue.

[dbcummings]

1. Did this just start happening and have you made any changes to the server?

Not sure. We just noticed it.

2. Is this issue impacting all of the reports or only specific reports, if so which ones?

All.

3.Did you follow our guide below for optimizing the php settings?

Yes.

Will PM you other info.

[benjaminsmith]

Hello @dbcummings,

Thanks for sending over the system profile. After reviewing the logs, you have over 40k services and nearly 5000 hosts configured on this server and it's experiencing high CPU load due to the volume of checks being executed. If your environment is growing, I would recommend adding another Nagios XI server.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31575 nagios 20 0 10840 1156 836 R 100.0 0.0 0:06.91 nagios
31627 root 0 -20 324108 60172 2056 R 94.7 0.0 976:06.81 perfd
31543 nagios 20 0 10840 1156 840 R 63.2 0.0 0:06.13 nagios
51749 nagios 20 0 205536 12136 4548 S 36.8 0.0 0:00.07 python
51750 nagios 20 0 205536 12144 4552 S 31.6 0.0 0:00.06 python
51753 nagios 20 0 205536 12144 4552 S 31.6 0.0 0:00.06 python
51748 nagios 20 0 205540 12144 4548 S 26.3 0.0 0:00.05 python
51751 nagios 20 0 205540 12144 4548 S 26.3 0.0 0:00.05 python

Most of the load is coming from check_ncpa, and I would reccomend spacing out the check intervals as much as possbile. Check only as often as needed. Please take a look at the guide below for tips on increasing performance.

Maximizing Performance in Nagios XI

Additionally, you may want to consider integrating Mod Gearman with this server to help distribute the load.

Integrating Mod-Gearman With Nagios XI

[dbcummings]

I figured that was the issue. Our checks appear to be working and reporting timely. We keep close tabs on the CPU and load metics. It is only the reporting that seems to impact us.

Also, we have already implemented the performance recommendations and are considering Mod-Gearman. I will PM you our specs.

[benjaminsmith]

Hello @dbcummings,

If you're able to increase the check_interval that will help. Also, run the following query to check the size of the database tables as large table sizes can impact system performance.

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

If you are interested in testing Mod-Gearman, your Nagios XI license is valid for 3 installs - production, test and backup. We also recommend testing any changes before implmenting them on the production server.

Nagios XI - License Entitlements

[dbcummings]

We are planning to being deploying Mod Gearman at the beginning of next year. I have sent you a PM with the table sizes.

[benjaminsmith]

Hello @dbcummings,

Ok. Got the PM and the log entries table is very large, and this will impact performance. I would recommend truncating this table to reduce the size. This will impact the Event Log report length in Nagios XI. The following command will truncate this table and you can adjust it accordingly to the number of days to retain in the database.

Code: Select all

echo 'DELETE FROM nagios_logentries WHERE entry_time < (NOW() - INTERVAL 30 DAY);' |mysql -t -u root -pnagiosxi nagios

If you want to store less data, you can set this in Admin > System Config > Performance Settings > Databases.

boeing-database.png

[dbcummings]

Will this impact the retention of metrics for the performance graphs or just the event logs?

[benjaminsmith]

Hello,

Will this impact the retention of metrics for the performance graphs or just the event logs?

No, performance data is stored in separate RRD files and not the MySQL database.