NagiosXI R1.6 - unreasonably low GUI performance

[pzusa]

Hi,

After update from R1.2 to R1.6 NXI is almost unusable - the NXI GUI performance is very low.
For instance making apply - dots don't show as usual - we can see only a few of them and they appear very slowly. This fancy gearwheel spins around very slowly and than completely stops after a few moments. Apply procedure (form the GUI perspective) doesn't complete at all (page times out - no complete info is shown).
Clicking any menu option (especialy from the left menu) or any host/service takes ages to complete.
There is no high load on the server, so this is not about the server being overloaded. Database works just fine (old Nagios GUI works fine too - so DB performance seems not to be an issue).
What is strange - using CCM - when I'm on a particular page (i.e. Services) it works like a charm - I can configure services, move between sub-pages with no lags. But clicking any option from the left menu takes again a lot of time to complete.

Nagios or whole server restarts doesn't help.

Please help - we cannot use this working like that.

Michalux

[mguthrie]

Does the issue appear present on different browsers and workstations as well? (I only ask because some of what you're describing is a slowness on the client-side javascript).

[pzusa]

Tested on workstations:
1. Linux Fedora, Chrome and Firefox
2. Windows 7, Chrome and IE
3. Windows XP, IE

Each time situation is all the same and started just after R1.2->R1.6 upgrade.
I cannot see any increased load on the workstations during NXI pages processing. Actually web browser load is nearly 0 while waiting for the NXI page.
From the network's perspective it looks like the browser sends the request and than it waits for the response for a long time (not packets are beeing sent between the browser/workstation and the server).

I've extended a bit apache logging and got i.e.:
10.x.x.x - [29/Jul/2011:11:14:09 +0200] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_host_status_state_summary_html%22%2C%22args%22%3A%7B%22hostname%22%3A%2210.y.y.y%22%2C%22host_id%22%3A%2293055%22%2C%22display%22%3A%22simple%22%7D%7D&nsp=89059d44109c00117a9437f72ea52866 HTTP/1.1" PID:473 Time:54 Data:783/745/320
where Time is the time in seconds it took to handle the request (54 sec !!)

Michalux

[mguthrie]

Are you using SSL or a Proxy on this system?

[pzusa]

No SSL or proxy, but I think I've pinned out this one.

I've tested R1.5 on our second environment and it behaves just the same - total slowness.
But, both our environments are being accessed from behind the NAT.
So I've tested it from within the same network - and tada - it works just fine.
So the problem is the NAT - but it worked on R1.2 without any problems, so something must have changed after this version, that affected access to the NATted IPs.
The question is what it was ?

Additional info: I've set some name for the NXI server in my workstation's /etc/hosts (pointing to NAT'ed address) and changed it respectively in Nagios Configuration. Next I've set the same name in /etc/hosts on NXI server (pointing both to NAT'ed and and local IP addresses) and .... it started to work properly !
So it looks like trying to open NXI console with NAT'ed IP address in URL makes the problem.

Hope You could fix it somehow.

Take care

Michalux

[mguthrie]

We made some updates to allow for compatibility with SSL usage and different port numbers, there are a lot of local ajax calls that pull data with GET requests. I'm guessing if there's slowness from the NAT it's because the GET requests are all having to go through the NAT.

[pzusa]

We made some updates to allow for compatibility with SSL usage and different port numbers, there are a lot of local ajax calls that pull data with GET requests. I'm guessing if there's slowness from the NAT it's because the GET requests are all having to go through the NAT.

Local calls made to the NAT'ed addresses won't work at all. I've tested it and now (knowing about the local calls) I can see packets originating from the server's local IP address and directed to the NAT'ed IP (just after opening NXI console in web browser) - of course with no response. It won't work that way !
So it looks like, the last changes You've made have broken the NXI server's ability to work behind the NAT

The question is how to fix this without braking SSL support ??
I'd appreciate some solution.

Michalux

[nscott]

How are you forwarding the ports through the NAT? In your VirtualHost are you specifying both IPs that it need be listening on? For instance if your internal IP of the Nagios server is 192.168.1.2 and the external its listening on should be 2.2.2.2 are you specifying it properly in the nagiosxi.conf?

NameVirtualHost 192.168.1.2
NameVirtualHost 2.2.2.2

<VirtualHost 192.168.1.2 2.2.2.2>
....
</VirtualHost>

Also, does the server have an alias or domain name?

[cwscribner]

~Just some input from another XI user~

I have the same slowness problem with the interface. When applying changes, it basically hangs. I've waited to see how long it will take and sometimes its upwards of 10 minutes before it finally accepts the changes. I'm accessing my XI instance via VPN.

[pzusa]

How are you forwarding the ports through the NAT?

The NAT is being performed on the CISCO firewall/router. Only the IP is being NATed - ports stay the same.

In your VirtualHost are you specifying both IPs that it need be listening on? For instance if your internal IP of the Nagios server is 192.168.1.2 and the external its listening on should be 2.2.2.2 are you specifying it properly in the nagiosxi.conf?

NameVirtualHost 192.168.1.2
NameVirtualHost 2.2.2.2

<VirtualHost 192.168.1.2 2.2.2.2>
....
</VirtualHost>

We don't have such config in nagiosxi.conf (I mean VirtualHost and NameVirtualHost). I may try to input something like this and check it out.

Also, does the server have an alias or domain name?

Yes it has, and actually I've found the workaround for this problem (without setting the VirtualHost in nagiosxi.conf).
The solution is to use the FQDN (name + domain name) in the URL instead of a IP and configure both IPs (external and local) pointing to the same name+domain name in /etc/hosts on the NagiosXI server (of course form the end user perspective this name has to resolve to the external IP via DNS or local hosts file).
And it works just fine this way. But it is not the solution, since some users tend to use IPs instead of names, which breaks things up.

OK - I'll try the nagiosxi.conf settings and let You know if that fixes the problem while using IPs not names.

Michalux