Nagios web interface is running slow

[amit.ahuja]

Hi,
We are having slowness issue with Nagios xi web interface. Logging in to the web interface could take 1 to 2 minutes. Applying configuration could take 10 to 15 minutes or even more. Does anyone have any idea about this issue?

Code: Select all

# top
top - 09:50:11 up 28 days, 21:07,  3 users,  load average: 0.35, 0.41, 0.50
Tasks: 264 total,   2 running, 262 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.4%us,  1.2%sy,  0.0%ni, 94.2%id,  0.1%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  32876492k total, 29638488k used,  3238004k free,  2070832k buffers
Swap:  8388604k total,     3272k used,  8385332k free, 21432488k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
17273 root      20   0  586m  25m 8044 S  2.0  0.1  93:38.38 BESClient
26916 nagios    20   0 40620  19m 1488 S  2.0  0.1   0:30.50 nagios
26925 nagios    20   0 64380  15m 1040 S  2.0  0.0   1:43.49 ndo2db
 6513 mysql     20   0 2203m  65m 3792 S  1.0  0.2   1072:48 mysqld
 8550 apache    20   0  446m  35m 4520 R  0.7  0.1  10:57.39 httpd
 7830 root      20   0 3155m  37m 7632 S  0.3  0.1  18:16.30 k08agent
24709 root      20   0 15172 1364  932 R  0.3  0.0   0:00.01 top
26919 nagios    20   0 10016  944  664 S  0.3  0.0   0:02.45 nagios
    1 root      20   0 19360 1396 1176 S  0.0  0.0   0:41.81 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.12 kthreadd
    3 root      RT   0     0    0    0 S  0.0  0.0   1:36.03 migration/0
    4 root      20   0     0    0    0 S  0.0  0.0  12:05.20 ksoftirqd/0
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0
    6 root      RT   0     0    0    0 S  0.0  0.0   0:01.35 watchdog/0
    7 root      RT   0     0    0    0 S  0.0  0.0   1:36.78 migration/1
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/1
    9 root      20   0     0    0    0 S  0.0  0.0  11:36.50 ksoftirqd/1
   10 root      RT   0     0    0    0 S  0.0  0.0   0:01.16 watchdog/1
   11 root      RT   0     0    0    0 S  0.0  0.0   1:33.10 migration/2
   12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/2
   13 root      20   0     0    0    0 S  0.0  0.0  15:08.28 ksoftirqd/2
   14 root      RT   0     0    0    0 S  0.0  0.0   0:01.27 watchdog/2
   15 root      RT   0     0    0    0 S  0.0  0.0   1:29.59 migration/3
   16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/3
   17 root      20   0     0    0    0 S  0.0  0.0   9:08.41 ksoftirqd/3
   18 root      RT   0     0    0    0 S  0.0  0.0   0:01.15 watchdog/3
   19 root      20   0     0    0    0 S  0.0  0.0   4:07.29 events/0
   20 root      20   0     0    0    0 S  0.0  0.0   3:46.54 events/1

Code: Select all

# iostat
Linux 2.6.32-504.36.1.el6.x86_64 (<server name)       07/13/2016      _x86_64_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.72    0.00    3.84    0.18    0.00   88.27

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              16.88         7.17       661.61   17896242 1650912952
sdb             161.96        88.50      4459.04  220824625 11126568696
dm-0              0.18         0.73         1.26    1818210    3141704
dm-1              0.00         0.00         0.00       9232      11584
dm-2              0.00         0.01         0.00      20506        616
dm-3              0.00         0.00         0.00       7288          0
dm-4            559.44        84.89      4434.14  211824130 11064423024
dm-5              1.81         3.59        14.32    8951378   35738696
dm-6             82.73         4.98       661.21   12420506 1649901768
dm-7              0.75         0.07         6.00     166194   14975336
dm-8              0.43         0.59         3.33    1461250    8299504
dm-9              0.08         0.79         0.39    1970282     978632

Thanks.

[tmcdonald]

That IO is pretty high, and I don't recognize BESClient as being anything of ours. A few questions:

• How many hosts and services are you monitoring with this server?
• What other applications are running? Anything involving backups or other disk-heavy tasks? Any graphical environment?
• What Nagios XI version are you using?

[amit.ahuja]

I think BESClient process is owned by whoever maintains this vm.

How many hosts and services are you monitoring with this server?
It is monitoring 266 hosts and 3243 services.

What other applications are running? Anything involving backups or other disk-heavy tasks? Any graphical environment?
This server is used only by Nagios. We have automation script to back up Nagios once every week, other than that, I don't see anything is using disk heavily. There is not graphical environment.

What Nagios XI version are you using?
Nagios XI 5.2.0

[tgriep]

You can try and tweak some PHP settings to see if that helps.
To do this, edit the /etc/php.ini file and change the following options to the values below.

Code: Select all

max_execution_time = 60
max_input_time = 120
memory_limit = 512M

Then restart Apache for the settings take place

Code: Select all

service httpd restart

Also, you can edit the /etc/hosts file and add the server name to the following line.

Code: Select all

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 nagiosserver

Try this and let us know if it helps.

[amit.ahuja]

No, it doesn't seem help, still slow. I notice one more thing here. When I was restarting httpd, it was saying

Code: Select all

Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using <ip> for ServerName

[rkennedy]

No, it doesn't seem help, still slow. I notice one more thing here. When I was restarting httpd, it was saying

Code: Select all

Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using <ip> for ServerName

This is expected, and not much of an issue. Is this the same machine that is having issues with the API not showing up properly?

Could you PM / email over a profile for us to look at? (Admin -> System Profile -> Download Profile)

[amit.ahuja]

Hi rkennedy,

This is a different server. I will send another email for this thread.

[ssax]

Please install iotop and see what is causing that IO wait, that's the reason it's running so slow. What is the speed of your disks? Anything change recently with the disks or the datastore that the VM is stored on? Did they move it to a different tier on the san/nas/etc? I would talk with your VM admins and see what they are seeing in the VM performance statistics and see if they can track down the bottleneck.

Code: Select all

yum install iotop
iotop