Page 1 of 4
Reports Extremely Slow
Posted: Tue Nov 12, 2019 7:40 am
by dbcummings
When running any reports in Nagios XI, the generation of the reports are very slow. I have tried updating the following in /etc/php.ini file with no impact:
max_execution_time
max_input_time
memory_limit
Re: Reports Extremely Slow
Posted: Tue Nov 12, 2019 11:50 am
by benjaminsmith
Hello
@dbcummings,
To help troubleshoot the issue, I'd like to ask you a few questions and also get a fresh copy of you system profile, so we can review the log files.
1. Did this just start happening and have you made any changes to the server?
2. Is this issue impacting all of the reports or only specific reports, if so which ones?
3.Did you follow our guide below for optimizing the php settings?
Nagios XI - Optimizing The PHP Settings File
4. Please PM you system profile. Thanks.
To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share this in a private message and then
reply to this post to bring it up in the queue.
Re: Reports Extremely Slow
Posted: Tue Nov 12, 2019 2:50 pm
by dbcummings
1. Did this just start happening and have you made any changes to the server?
Not sure. We just noticed it.
2. Is this issue impacting all of the reports or only specific reports, if so which ones?
All.
3.Did you follow our guide below for optimizing the php settings?
Yes.
Will PM you other info.
Re: Reports Extremely Slow
Posted: Tue Nov 12, 2019 5:06 pm
by benjaminsmith
Hello
@dbcummings,
Thanks for sending over the system profile. After reviewing the logs, you have over 40k services and nearly 5000 hosts configured on this server and it's experiencing high CPU load due to the volume of checks being executed. If your environment is growing, I would recommend adding another Nagios XI server.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31575 nagios 20 0 10840 1156 836 R 100.0 0.0 0:06.91 nagios
31627 root 0 -20 324108 60172 2056 R 94.7 0.0 976:06.81 perfd
31543 nagios 20 0 10840 1156 840 R 63.2 0.0 0:06.13 nagios
51749 nagios 20 0 205536 12136 4548 S 36.8 0.0 0:00.07 python
51750 nagios 20 0 205536 12144 4552 S 31.6 0.0 0:00.06 python
51753 nagios 20 0 205536 12144 4552 S 31.6 0.0 0:00.06 python
51748 nagios 20 0 205540 12144 4548 S 26.3 0.0 0:00.05 python
51751 nagios 20 0 205540 12144 4548 S 26.3 0.0 0:00.05 python
Most of the load is coming from check_ncpa, and I would reccomend spacing out the check intervals as much as possbile. Check only as often as needed. Please take a look at the guide below for tips on increasing performance.
Maximizing Performance in Nagios XI
Additionally, you may want to consider integrating Mod Gearman with this server to help distribute the load.
Integrating Mod-Gearman With Nagios XI
Re: Reports Extremely Slow
Posted: Wed Nov 13, 2019 12:57 pm
by dbcummings
I figured that was the issue. Our checks appear to be working and reporting timely. We keep close tabs on the CPU and load metics. It is only the reporting that seems to impact us.
Also, we have already implemented the performance recommendations and are considering Mod-Gearman. I will PM you our specs.
Re: Reports Extremely Slow
Posted: Wed Nov 13, 2019 2:20 pm
by benjaminsmith
Hello
@dbcummings,
If you're able to increase the check_interval that will help. Also, run the following query to check the size of the database tables as large table sizes can impact system performance.
Code: Select all
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
If you are interested in testing Mod-Gearman, your Nagios XI license is valid for 3 installs - production, test and backup. We also recommend testing any changes before implmenting them on the production server.
Nagios XI - License Entitlements
Re: Reports Extremely Slow
Posted: Thu Nov 14, 2019 6:53 am
by dbcummings
We are planning to being deploying Mod Gearman at the beginning of next year. I have sent you a PM with the table sizes.
Re: Reports Extremely Slow
Posted: Thu Nov 14, 2019 12:51 pm
by benjaminsmith
Hello
@dbcummings,
Ok. Got the PM and the log entries table is very large, and this will impact performance. I would recommend truncating this table to reduce the size. This will impact the Event Log report length in Nagios XI. The following command will truncate this table and you can adjust it accordingly to the number of days to retain in the database.
Code: Select all
echo 'DELETE FROM nagios_logentries WHERE entry_time < (NOW() - INTERVAL 30 DAY);' |mysql -t -u root -pnagiosxi nagios
If you want to store less data, you can set this in
Admin > System Config > Performance Settings > Databases.
boeing-database.png
Re: Reports Extremely Slow
Posted: Thu Nov 14, 2019 12:54 pm
by dbcummings
Will this impact the retention of metrics for the performance graphs or just the event logs?
Re: Reports Extremely Slow
Posted: Thu Nov 14, 2019 1:01 pm
by benjaminsmith
Hello,
Will this impact the retention of metrics for the performance graphs or just the event logs?
No, performance data is stored in separate RRD files and not the MySQL database.