Page 1 of 2
Slow page loads for users
Posted: Tue May 17, 2016 5:07 pm
by acheesem
This happens when the user has 'Can see all objects:' is checked for the user, this has happened recently, and was working fine in the past.
I can grant the user Admin access and page loads shift from 30+ seconds to ~3 seconds, the same result is observed by unticking "Can see all objects" on the user, this isn't related to an individual user account as we can see this behaviour on all accounts with these flags..
I note it is the same behavior to this forum post
https://support.nagios.com/forum/viewto ... 16&t=38417
The accounts are local accounts
Nagios itself is running fine, we've checked for any slow queries / errors on the database side there are no performance issues there that we can see..
On the users browser the hung query appears to most often be through the ajaxhelper.php ajax call
https://nagiosxi/nagiosxi/ajaxhelper.ph ... abled&nsp=####
It follows the user through all the browsers we have tried as well as just masquerading as the user and the problem exists then as well.
any suggestions of where to start looking would be appreciated
cheers
--Aaron
Re: Slow page loads for users
Posted: Wed May 18, 2016 12:05 am
by Box293
What version of XI are you running?
Re: Slow page loads for users
Posted: Wed May 18, 2016 1:23 am
by acheesem
Sorry I should have included more details in the post
Nagios XI 5.2.7
~]# uname -a
Linux rudolph.aut.ac.nz 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.7 (Santiago)
have multiple gearman workers to service the queues
Re: Slow page loads for users
Posted: Wed May 18, 2016 2:01 pm
by tgriep
One thing to try is to add the XI servers host name in the /etc/hosts file on the end of the 127.0.0.1 loopback line.
Sometimes this helps out in issues like this.
How many hosts and service checks are running in this server?
If there are a lot, that could cause the slowdown. Every time a user tries to access a page, the server checks through all of the hosts and services checking to see if they have the correct rights and that does take more time.
What other permissions are you giving the user besides Can see all objects?
Re: Slow page loads for users
Posted: Wed May 18, 2016 4:51 pm
by acheesem
tgriep wrote:One thing to try is to add the XI servers host name in the /etc/hosts file on the end of the 127.0.0.1 loopback line.
Sometimes this helps out in issues like this.
I've added this it didn't make a instant difference
tgriep wrote:
How many hosts and service checks are running in this server?
1000 hosts 10,000 services roughly
The odd thing is if I just turn off the option view all objects, it starts working fine. Or grant admin access. And its only the ajaxhelper call that is slow, though this does look like a wrapper for getting data, so that just hides what is going on.
tgriep wrote:What other permissions are you giving the user besides Can see all objects?
I've turned on and off all the other options for the users and the can see all objects is the one that directly affects the performance. I could try combinations if you like, the user reporting the issue has the following options set
Can see all objects:
Can (re)configure hosts and services:
Can control all objects:
Can see/control monitoring engine:
Can access advanced features:
Just in addition most of the actual performance of NagiosXI seems fine, the system doesn't appear to be under significantly high load when this happens, but we do see apache with several busy threads. The easiest way to experience the fault is to visit the Scheduled Downtime page, not all pages are loading slowly, in fact the front few pages are fine. Scheduled downtime page might start the performance problems for the other pages he visits.
I've just checked and the page after finally loading, doesn't actually contain the correct information, it is coming back blank. So the scheduled Downtime page isn't showing any of his entries (of which this user has a few). But the page has returned to the user and the 'refresh' icon is endlessly spinning (as if you have clicked it and its waiting for a ajax response)
cheers
--Aaron
Re: Slow page loads for users
Posted: Wed May 18, 2016 4:55 pm
by Box293
Does nagiosxi resolve to an internal private IP address, or a external one?
What is the result of this command on your XI server:
You may need to install bind-utils for that command to work
Re: Slow page loads for users
Posted: Wed May 18, 2016 5:09 pm
by acheesem
Box293 wrote:
Does nagiosxi resolve to an internal private IP address, or a external one?
Its a CNAME to the servers hostname which is on a external ip
Box293 wrote:
What is the result of this command on your XI server:
~]# host nagiosxi
nagiosxi.xxx.nz is an alias for rudolph.xxx.nz.
rudolph.xxx.nz has address 156.###.###.###
Re: Slow page loads for users
Posted: Wed May 18, 2016 5:12 pm
by Box293
Can you go to Admin > System Config > Manage System Config
What is in your "Program URL" and "External URL" fields?
Re: Slow page loads for users
Posted: Wed May 18, 2016 5:15 pm
by acheesem
Box293 wrote:Can you go to Admin > System Config > Manage System Config
What is in your "Program URL" and "External URL" fields?
Mine was listed in Admin > System Config > System Settings (if that makes a difference)
Program URL is
https://nagiosxi.xxx.nz/nagiosxi/
External URL is blank
Re: Slow page loads for users
Posted: Wed May 18, 2016 6:08 pm
by acheesem
Following up on this, i've now checked the scheduled host downtime, and there is now a call (maybe as a result of the host file change?)
https://nagiosxi.xxx.nz/nagiosxi/includ ... tails=true
this is getting a 500 internal server error
as I traced it back it links to the statusjson.cgi, unfortunately this is a compiled cgi, i'm not sure where to find the source to start debugging that