Slow page loads for users

[acheesem]

This happens when the user has 'Can see all objects:' is checked for the user, this has happened recently, and was working fine in the past.

I can grant the user Admin access and page loads shift from 30+ seconds to ~3 seconds, the same result is observed by unticking "Can see all objects" on the user, this isn't related to an individual user account as we can see this behaviour on all accounts with these flags..

I note it is the same behavior to this forum post https://support.nagios.com/forum/viewto ... 16&t=38417

The accounts are local accounts

Nagios itself is running fine, we've checked for any slow queries / errors on the database side there are no performance issues there that we can see..

On the users browser the hung query appears to most often be through the ajaxhelper.php ajax call
https://nagiosxi/nagiosxi/ajaxhelper.ph ... abled&nsp=####

It follows the user through all the browsers we have tried as well as just masquerading as the user and the problem exists then as well.

any suggestions of where to start looking would be appreciated

cheers
--Aaron

[Box293]

What version of XI are you running?

[acheesem]

Sorry I should have included more details in the post

Nagios XI 5.2.7

~]# uname -a
Linux rudolph.aut.ac.nz 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.7 (Santiago)

have multiple gearman workers to service the queues

[tgriep]

One thing to try is to add the XI servers host name in the /etc/hosts file on the end of the 127.0.0.1 loopback line.
Sometimes this helps out in issues like this.

How many hosts and service checks are running in this server?

If there are a lot, that could cause the slowdown. Every time a user tries to access a page, the server checks through all of the hosts and services checking to see if they have the correct rights and that does take more time.
What other permissions are you giving the user besides Can see all objects?

[acheesem]

One thing to try is to add the XI servers host name in the /etc/hosts file on the end of the 127.0.0.1 loopback line.
Sometimes this helps out in issues like this.

I've added this it didn't make a instant difference

How many hosts and service checks are running in this server?

1000 hosts 10,000 services roughly

The odd thing is if I just turn off the option view all objects, it starts working fine. Or grant admin access. And its only the ajaxhelper call that is slow, though this does look like a wrapper for getting data, so that just hides what is going on.

What other permissions are you giving the user besides Can see all objects?

I've turned on and off all the other options for the users and the can see all objects is the one that directly affects the performance. I could try combinations if you like, the user reporting the issue has the following options set

Can see all objects:
Can (re)configure hosts and services:
Can control all objects:
Can see/control monitoring engine:
Can access advanced features:


Just in addition most of the actual performance of NagiosXI seems fine, the system doesn't appear to be under significantly high load when this happens, but we do see apache with several busy threads. The easiest way to experience the fault is to visit the Scheduled Downtime page, not all pages are loading slowly, in fact the front few pages are fine. Scheduled downtime page might start the performance problems for the other pages he visits.

I've just checked and the page after finally loading, doesn't actually contain the correct information, it is coming back blank. So the scheduled Downtime page isn't showing any of his entries (of which this user has a few). But the page has returned to the user and the 'refresh' icon is endlessly spinning (as if you have clicked it and its waiting for a ajax response)

cheers
--Aaron

[Box293]

On the users browser the hung query appears to most often be through the ajaxhelper.php ajax call
https://nagiosxi/nagiosxi/ajaxhelper.ph ... abled&nsp=####

Does nagiosxi resolve to an internal private IP address, or a external one?

What is the result of this command on your XI server:

Code: Select all

host nagiosxi

You may need to install bind-utils for that command to work

Code: Select all

yum -y install bind-utils

[acheesem]

Does nagiosxi resolve to an internal private IP address, or a external one?

Its a CNAME to the servers hostname which is on a external ip

What is the result of this command on your XI server:

Code: Select all

host nagiosxi

~]# host nagiosxi
nagiosxi.xxx.nz is an alias for rudolph.xxx.nz.
rudolph.xxx.nz has address 156.###.###.###

[Box293]

Can you go to Admin > System Config > Manage System Config

What is in your "Program URL" and "External URL" fields?

[acheesem]

Can you go to Admin > System Config > Manage System Config

What is in your "Program URL" and "External URL" fields?

Mine was listed in Admin > System Config > System Settings (if that makes a difference)

Program URL is https://nagiosxi.xxx.nz/nagiosxi/
External URL is blank

[acheesem]

Following up on this, i've now checked the scheduled host downtime, and there is now a call (maybe as a result of the host file change?)

https://nagiosxi.xxx.nz/nagiosxi/includ ... tails=true

this is getting a 500 internal server error

as I traced it back it links to the statusjson.cgi, unfortunately this is a compiled cgi, i'm not sure where to find the source to start debugging that