Reports Page Hangs Avail.cgi

[hul1]

Hi I'm sure other folks have seen this same issue from tickets I've found but I'm having the issue where we go into the reports page and the site just hangs for a long time. We see on the box the avail.cgi process taking 100% of a core.
I've tried increasing the memory, max_input_vars in php.ini and it hasn't helped. I've also tried disabled the auto-load for reports and that helps me get into the reports section but if I try to do a run it crashes.

max_input_time = 120

; Maximum input variable nesting level
; http://php.net/max-input-nesting-level
;max_input_nesting_level = 64

; How many GET/POST/COOKIE input variables may be accepted
max_input_vars = 10000

; Maximum amount of memory a script may consume (128MB)
; http://php.net/memory-limit
memory_limit = 8192M

Do you have any idea of why this would be happening and how to fix it? It used to work when we first set it up, but maybe now we have a lot more hosts it just is too big? The server has 8 vcpus and 16 GB and we have about 500 hosts and 3000 services.

[ssax]

What is your max_execution_time set to in your /etc/php.ini?

Also, the availability report works off of files on the filesystem, specificially:

Code: Select all

/usr/local/nagios/var/nagios.log
/usr/local/nagios/var/archives/*

Please PM me a copy of your profile, you can download it from Admin > System Profile > Download Profile.

What is the full output of these commands as well:

Code: Select all

ls -l /usr/local/nagios/var
ls -l /usr/local/nagios/var/archives
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagiosxi --table

[hul1]

Okay, my max_execution_time is

Code: Select all

#cat /etc/php.ini | grep max_execution_time
max_execution_time = 60

I'll PM you all the information from the ls and mysql info and the system profile dump.

[ssax]

Please bump your max_execution_time up to like 600 (or higher just for testing) and see if it completes, that's the one that matters for this one (well, and the memory_limit), don't forget to restart httpd so the changes become active:

Code: Select all

service httpd restart

Everything else from the profile looks good.

Let us know the results.

[hul1]

Thanks, but Is there any way we can get nagios not to hang when it runs the report?
For instance, I run

Screen Shot 2019-03-19 at 11.11.21 AM.png

just a simple report of 1 host in the last 24 hours, it takes about 5 minutes to load. During this time, nobody can access nagios or do anything it just hangs.

I also did the php changes to increase timeout, memory_limit, max_execution_time and it's still taking a long time to load as well as hanging the service. It would be okay if we can just have the report run slowly and not hang others from doing things.

[npolovenko]

@hul1, How many hosts and services do you have on this XI in total? How many logs do you have in the archives folder?

/usr/local/nagios/var/archives

I've seen systems where people have years of logs stored in this folder and that was causing reports to be extremely slow because Nagios analyzes past archived logs to figure out whether hosts and services should be in downtime.

Please tail errors in the apache log while running the report. From the command line run:

tail -f /var/log/httpd/*error*

And while this is running try to generate a report in the GUI. Let me know if you see any errors in the console while the report is hanging.

Also, run this database repair script:

mysqlcheck -r -f -uroot -pnagiosxi --all-databases --use_frm

And reboot the server with shutdown -r now

Let me know if this improves the speed.

[hul1]

How many hosts and services do you have on this XI in total? How many logs do you have in the archives folder?
We have 570 hosts and 3725 services
/usr/local/nagios/var/archives
We have 75 logs, it's a fairly new system we just finished setting up so only a few weeks of active data.

I'm running the reports now though and it seems to be much quicker, I can run reports on the entire list and it'll finish in 30 seconds or less when prior it would be several minutes or longer for 1 host. I'm not sure what fixed it, but maybe those changes just took some time to take effect.

Thanks for your help!

[npolovenko]

@hul1, Could be some stuck process consuming too much system resources or database corruption. Glad the performance improved. Are we good to close this thread now or do you have any other questions?

[hul1]

Yes, you can close the thread thanks!