Nagios XI web interface slow

[NeilWilson]

Hi , It's a new year so I thought I would have another attempt at getting some help making our nagios server actually load pages in a timely manner....at the moment and of the Views pages take around 1 minute to load.

I spent a long time trying to fix this with nagios support about a year or more ago.. but got nowhere - so thought I would try again!

looking at top the httpd process is pinned at 100% cpu every time a page is loaded.
The server is a VM has 16GB memory and 4 CPU cores , SAN storage array - all flash, IO wait isnt the issue.
The server is only running about 1500 checks...

For example:

top - 12:35:18 up 11 days, 1:13, 1 user, load average: 1.88, 1.78, 1.76
Tasks: 269 total, 3 running, 265 sleeping, 0 stopped, 1 zombie
Cpu(s): 56.7%us, 2.8%sy, 0.0%ni, 38.0%id, 2.4%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 16459032k total, 15096724k used, 1362308k free, 524656k buffers
Swap: 2097148k total, 38796k used, 2058352k free, 3445168k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26719 apache 20 0 1308m 830m 17m R 99.7 5.2 3:09.46 httpd
18111 apache 20 0 1514m 1.0g 18m R 99.1 6.5 5:19.91 httpd
17964 nagios 20 0 109m 72m 1512 S 13.6 0.4 1:55.33 nagios
65496 nagios 20 0 24380 5248 1748 S 1.7 0.0 0:00.05 snmpget
17976 nagios 20 0 63132 2332 1032 S 1.0 0.0 1:22.86 ndo2db
47 root 20 0 0 0 0 S 0.3 0.0 0:38.44 kblockd/1
1595 root 20 0 0 0 0 S 0.3 0.0 6:19.50 flush-253:4
4831 mysql 20 0 2189m 51m 6068 S 0.3 0.3 3:25.69 mysqld
17968 nagios 20 0 10156 1136 688 S 0.3 0.0 0:00.54 nagios
65119 nagios 20 0 327m 33m 8488 S 0.3 0.2 0:00.15 php
65179 postgres 20 0 212m 6956 4480 S 0.3 0.0 0:00.02 postmaster
1 root 20 0 33664 1380 1184 S 0.0 0.0 0:07.28 init

When I enable debugging mode in firefox and look at how long its taking to load each bit of the page its getting stuck for about 20-30 seconds on this url - http://exvnagiosxiprd01/nagiosxi/ajaxhe ... 14ecb09f8d

And then stuck for another 20-30 seconds on this url - http://exvnagiosxiprd01/nagiosxi/ajaxhe ... 14ecb09f8d

I've tried SQL repair several times - it doesnt fix it.
Also tried postgres vacuum, increasing the number of SQL connections , adding RAMdisk, php op caching etc etc..
Saw another forum post that thought it might be dns related so added the entry of my pc to the hosts file on the server and restarted it - that didnt work either..

It's got to be something wrong with the apache config - why else would it be hogging 100% cpu ... I have no idea what though...

[bolson]

Please run the following two commands twice from the XI command line, once while Nagios XI is idle, and again while a page is loading:

Code: Select all

ps aux | awk '{print $6 " " $11}' | sort -nr | head -20
ps aux | awk '{print $3 " " $11}' | sort -nr | sed 1d | head -20

And post the results.

[NeilWilson]

root@exvnagiosxiprd01:~]$ ps aux | awk '{print $6 " " $11}' | sort -nr | head -20
888580 /usr/sbin/httpd
298148 /usr/sbin/httpd
79096 /usr/libexec/mysqld
74892 /usr/local/nagios/bin/nagios
74744 /usr/local/nagios/bin/nagios
71388 /usr/local/nagios/bin/nagios
65876 /usr/sbin/httpd
63748 /usr/sbin/httpd
63244 /usr/sbin/httpd
62844 /usr/sbin/httpd
61908 /usr/sbin/httpd
61648 /usr/sbin/httpd
61396 /usr/sbin/httpd
61300 /usr/sbin/httpd
58608 /usr/sbin/httpd
57824 /usr/sbin/httpd
52820 /usr/sbin/httpd
41624 /usr/bin/ruby
34936 /usr/bin/php
32460 /usr/bin/perl
[root@exvnagiosxiprd01:~]$ ps aux | awk '{print $3 " " $11}' | sort -nr | sed 1d | head -20
12.2 /usr/bin/perl
5.4 /usr/bin/perl
5.2 /usr/bin/perl
4.5 /usr/local/nagios/bin/nagios
4.4 /usr/local/nagios/bin/nagios
2.6 /usr/sbin/httpd
2.3 /usr/sbin/httpd
2.2 /usr/sbin/httpd
2.1 /usr/sbin/httpd
2.1 /usr/sbin/httpd
1.9 /usr/libexec/mysqld
1.8 /usr/sbin/httpd
1.8 /usr/sbin/httpd
1.8 /usr/sbin/httpd
1.8 /usr/sbin/httpd
1.8 /usr/bin/perl
1.7 /usr/sbin/httpd
1.7 /usr/sbin/httpd
1.7 /usr/sbin/httpd
1.7 /usr/bin/perl
[root@exvnagiosxiprd01:~]$

[bolson]

We're these commands run while XI was idle or while you were loading a page? I'm looking for a comparison of resource usage between the two states.

[NeilWilson]

Whilst loading a page:

[root@exvnagiosxiprd01:/home/nswilson]$ ps aux | awk '{print $6 " " $11}' | sort -nr | head -20
128824 /usr/sbin/httpd
104388 /usr/libexec/mysqld
74892 /usr/local/nagios/bin/nagios
74788 /usr/local/nagios/bin/nagios
71388 /usr/local/nagios/bin/nagios
61208 /usr/sbin/httpd
46112 /usr/sbin/httpd
44612 /usr/sbin/httpd
44544 /usr/sbin/httpd
44520 /usr/bin/ruby
44132 /usr/sbin/httpd
43812 /usr/sbin/httpd
43528 /usr/sbin/httpd
35008 /usr/bin/php
26992 /usr/bin/perl
26444 /usr/bin/perl
24324 /usr/bin/php
24048 /usr/bin/php
24024 /usr/bin/php
23716 /usr/bin/php
[root@exvnagiosxiprd01:/home/nswilson]$

[root@exvnagiosxiprd01:/home/nswilson]$ ps aux | awk '{print $3 " " $11}' | sort -nr | sed 1d | head -20
5.0 /usr/bin/perl
4.5 /usr/local/nagios/bin/nagios
4.4 /usr/local/nagios/bin/nagios
2.5 /usr/bin/perl
2.1 /usr/bin/perl
1.6 /usr/libexec/mysqld
1.2 /usr/bin/php
1.0 /usr/local/nagios/bin/ndo2db
1.0 /usr/local/nagios/bin/ndo2db
0.7 /usr/bin/perl
0.6 /usr/bin/php
0.6 /usr/bin/php
0.6 /usr/bin/php
0.5 /usr/bin/php
0.5 /usr/bin/php
0.3 postgres:
0.2 /usr/bin/perl
0.0 xinetd
0.0 vim
0.0 sudo

Idle:

[root@exvnagiosxiprd01:/home/nswilson]$ ps aux | awk '{print $6 " " $11}' | sort -nr | head -20
706328 /usr/sbin/httpd
673328 /usr/sbin/httpd
97896 /usr/libexec/mysqld
74892 /usr/local/nagios/bin/nagios
74788 /usr/local/nagios/bin/nagios
71388 /usr/local/nagios/bin/nagios
61276 /usr/sbin/httpd
46464 /usr/sbin/httpd
44884 /usr/sbin/httpd
44520 /usr/bin/ruby
43860 /usr/sbin/httpd
43720 /usr/sbin/httpd
43120 /usr/sbin/httpd
34852 /usr/bin/php
32436 /usr/bin/perl
31148 /usr/bin/perl
24300 /usr/bin/php
24044 /usr/bin/php
24024 /usr/bin/php
23764 /usr/bin/php
[root@exvnagiosxiprd01:/home/nswilson]$ ps aux | awk '{print $3 " " $11}' | sort -nr | sed 1d | head -20
17.3 /usr/bin/perl
6.4 /usr/bin/perl
4.5 /usr/local/nagios/bin/nagios
4.4 /usr/local/nagios/bin/nagios
3.0 /usr/bin/perl
1.6 /usr/libexec/mysqld
1.2 /usr/bin/perl
1.1 /usr/bin/php
1.0 /usr/local/nagios/bin/ndo2db
1.0 /usr/local/nagios/bin/ndo2db
0.5 /usr/bin/php
0.5 /usr/bin/php
0.5 /usr/bin/php
0.4 /usr/bin/php
0.4 /usr/bin/php
0.3 postgres:
0.2 /usr/bin/perl
0.0 xinetd
0.0 vim
0.0 sudo
[root@exvnagiosxiprd01:/home/nswilson]$

[kyang]

Could you send us your profile? I'd like to see if there is something in your logs.

Nagios XI Profile --> On the XI Home Page click "Admin" > "System Profile" --> "Download Profile" button
Save the profile.zip file and upload it here or PM me.

If you receive a PROFILE BUILD FAILED
Please follow this article,

https://support.nagios.com/kb/article.p ... ategory=44

You can also generate a profile manually using the script at /usr/local/nagiosxi/html/includes/components/profile/getprofile.sh

That should generate a profile in /usr/local/nagiosxi/var/components/ which you can get off the server with an application such as FileZilla.

After you PM the profile please update this thread so we can see it back on our end, unless you post the profile on here. Thanks!


UPDATE: Profile received! Shared with team

[NeilWilson]

I've PM'd you the system profile. Thanks.

[kyang]

I'm still digging through your profile, but I can see that you have two instances of Nagios running.

Code: Select all

nagios   23894     1  4 Jan03 ?        06:29:10 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   23896 23894  0 Jan03 ?        00:02:05 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   23897 23894  0 Jan03 ?        00:02:07 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   23898 23894  0 Jan03 ?        00:02:07 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   23899 23894  0 Jan03 ?        00:02:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   23900 23894  0 Jan03 ?        00:02:06 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   23901 23894  0 Jan03 ?        00:02:06 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   23903 11313  0 Jan03 ?        00:01:50 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios   23904 23903  1 Jan03 ?        01:30:27 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg

Code: Select all

nagios   45506     1  4 Jan02 ?        07:17:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   45508 45506  0 Jan02 ?        00:02:24 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   45509 45506  0 Jan02 ?        00:02:24 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   45510 45506  0 Jan02 ?        00:02:23 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   45511 45506  0 Jan02 ?        00:02:21 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   45512 45506  0 Jan02 ?        00:02:24 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   45514 45506  0 Jan02 ?        00:02:20 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   45517 11313  0 Jan02 ?        00:02:07 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios   45518 45517  1 Jan02 ?        01:43:02 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg

Can you run the following commands in this order. Let us know if anything fails.

NOTE: You ***must*** use mariadb instead of mysqld in the commands below, ***if*** you have mariadb.
# service nagios stop
# service ndo2db stop
# service mysqld stop
# service crond stop
# service httpd stop
# killall -9 nagios
# killall -9 ndo2db
# rm -f /usr/local/nagios/var/rw/nagios.cmd
# rm -f /usr/local/nagios/var/nagios.lock
# rm -f /usr/local/nagios/var/ndo.sock
# rm -f /usr/local/nagios/var/ndo2db.lock
# rm -f /usr/local/nagiosxi/var/reconfigure_nagios.lock
# for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
# service mysqld start
# service ndo2db start
# service nagios start
# service httpd start
# service crond start

Then check it with this, and show us the output.

Code: Select all

ps -ef | head -1 && ps -ef | grep bin/nagios

Also, let us know if that speeds up the interface. Thanks!

[NeilWilson]

ps -ef | head -1 && ps -ef | grep bin/nagios
UID PID PPID C STIME TTY TIME CMD
nagios 60139 1 16 13:43 ? 00:00:07 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 60141 60139 0 13:43 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 60142 60139 0 13:43 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 60143 60139 0 13:43 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 60144 60139 0 13:43 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 60145 60139 0 13:43 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 60146 60139 0 13:43 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
root 60523 57991 0 13:44 pts/0 00:00:00 grep bin/nagios

It's a bit better, still a lag of about 15 seconds to fully load the page though.

[dwhitfield]

In /etc/php.ini, please change the following to the listed numbers (unless the numbers are higher than these):

Code: Select all

    max_execution_time = 60
    max_input_time = 120
    memory_limit = 256M

Also, please add the following line:

Code: Select all

max_input_vars = 10000

After making these changes you'll need to restart the httpd service:

Code: Select all

service httpd restart