The reports page hangs forever when loaded

[gormank]

Hi,
This also basically hangs the web interface untill avail.cgi is killed.
CPU load also gets high and killing avail.cgi causes the load to go back down.
This has only been happening for a few days.
NXI 5.8.4
RHEL 7.8
Changing the performance setting to no load reports when the page loads and running reports manually still runs avail.cgi.
We had a situation where it said the DB was damaged and to run repair_databases.sh but that's been run many times w/ little help. I also found several posts with the same symptoms, ran
mysqlcheck -r -f -uroot -pnagiosxi --all-databases --use_frm
and cleared archived logs w/o success.
I see the following in the ssl_error_log

Code: Select all

# tail /var/log/httpd/ssl_error_log
[Tue Nov 16 20:55:39.711994 2021] [:error] [pid 26510] [client 172.30.139.80:56575] PHP Notice:  Undefined index: scheme in /usr/local/nagiosxi/html/includes/pageparts.inc.php on line 607, referer: https://qa4am2mlnagx001/nagiosxi/
[Tue Nov 16 21:11:56.466760 2021] [:error] [pid 8331] [client 172.30.139.80:65082] PHP Notice:  Undefined index: scheme in /usr/local/nagiosxi/html/includes/pageparts.inc.php on line 607, referer: https://qa4am2mlnagx001/nagiosxi/includes/page-home-main.php?&=
[Tue Nov 16 21:12:02.862757 2021] [:error] [pid 7444] [client 172.30.139.80:65085] PHP Notice:  Undefined index: scheme in /usr/local/nagiosxi/html/includes/pageparts.inc.php on line 607, referer: https://qa4am2mlnagx001/nagiosxi/admin/?xiwindow=updates.php
[Tue Nov 16 21:12:42.748579 2021] [:error] [pid 21533] [client 172.30.139.80:65303] PHP Notice:  Undefined index: scheme in /usr/local/nagiosxi/html/includes/pageparts.inc.php on line 607, referer: https://qa4am2mlnagx001/nagiosxi/includes/page-home-main.php?&=

Any suggestions?

Thanks!

[pbroste]

Hello @gormank

Thanks for reaching out, you mention 'avail.cgi' is hanging which is related to Nagios Core backend. Want to have you load the report in the web console with Development Tools > Network Tab open, and bing up the report that hangs reload the page and watch the Network Tab populate while it loads, and let us know where it is hanging.

Please send us your system profile so we can see what is going on:
1. Login to the Nagios XI GUI using a web browser.
2. Click the "Admin" > "System Profile" Menu
3. Click the "Download Profile" button
4. Save the profile.zip file and share this in a private message and then reply to this post to bring it up in the queue.

Thanks,
Perry

[gormank]

Sorry, I don't see any thing in XI or core for Development Tools.
I PMed the profile to you a minute ago...

[gormank]

I see the PM w/ the profile is sitting in my outbox so here it is.

Moderator note: removed attached profile and placed it on local shared drive

[pbroste]

Hello @gormank

Thanks for sending over the System Profile, and after reviewing let's start off and update the 'nagios' user expiration.

Code: Select all

chage -I -1 -m 0 -M 99999 -E -1 nagios

Looks like it is hitting the upper limits on the default PHP settings and that's why it's hanging. You'll want to bump up the max_input_vars, memory_limit, max_execution_time, and max_input_time in the php.ini file.

We have an article to guide you through the process on our knowledge-base at:
Nagios XI - Optimizing The PHP Settings File

max_input_vars: 18000
memory_limit: 4096M
max_execution_time: 180
max_input_time: 180

And https://support.nagios.com/kb/article/n ... s-829.html

Restart the services by bouncing:

Code: Select all

systemctl restart httpd nagios

Thanks,
Perry

[gormank]

I updated the php.ini as suggested and restarted services.
This had no impact on the issue. avail.cgi uses 100% CPU and the web UI stalled until avail.cgi is killed.

[gormank]

I restored backups from 11/10 yesterday and the issue seems to be resolved. Both the primary and backup (primary is synced to the backup) are now opening the reports tab ok. I also saved the php.ini before the restore and copied it back to /etc after the restore.

Feel free to close this one.

Thanks!

[pbroste]

Excellent glad that you were able to restore to functioning again, and thanks @gormank for following up with the details too.

Locking,
Perry