httpd memory, server I/O very high

[kotterbein]

It looks like we are having some issues with Memory and I/O wait times on our XI server (2011, R3.2).

right now, httpd is taking up a HUGE amount of the servers memory, and it is killing our check performance:

Code: Select all

Tasks: 272 total,   2 running, 270 sleeping,   0 stopped,   0 zombie
Cpu(s): 13.3%us,  0.6%sy,  0.0%ni, 63.7%id, 22.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:  24682292k total, 17752592k used,  6929700k free,   898168k buffers
Swap: 26738680k total,    40408k used, 26698272k free, 10653280k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
13491 apache    16   0 1899m 1.6g 3640 S  0.0  6.8   0:29.62 httpd
13490 apache    16   0  758m 483m 3640 S  0.0  2.0   0:20.14 httpd
13493 apache    16   0  757m 482m 3644 S  0.0  2.0   0:24.23 httpd
13488 apache    16   0  754m 479m 3640 S  0.0  2.0   0:20.20 httpd
13492 apache    15   0  665m 390m 3656 S  0.0  1.6   0:14.98 httpd
13489 apache    16   0  665m 390m 3640 S  0.0  1.6   0:15.67 httpd
17026 apache    19   0  665m 390m 3632 R 99.8  1.6   0:14.34 httpd
17062 apache    16   0  390m 123m 3400 S  0.0  0.5   0:08.91 httpd
13487 apache    16   0  371m 102m 3640 S  0.0  0.4   0:12.25 httpd
17060 apache    16   0  371m 102m 3616 S  0.0  0.4   0:07.11 httpd
17064 apache    15   0  371m 102m 3616 S  0.0  0.4   0:07.07 httpd
 4953 mysql     15   0  304m  80m 4264 S  0.0  0.3   5:37.50 mysqld
14825 apache    16   0  298m  30m 3656 S  0.0  0.1   0:06.06 httpd
13486 apache    15   0  298m  30m 3632 S  0.0  0.1   0:08.35 httpd
17063 apache    16   0  298m  30m 3568 S  0.0  0.1   0:03.75 httpd
17061 apache    16   0  298m  30m 3576 S  0.0  0.1   0:03.46 httpd
14292 apache    15   0  297m  30m 3592 S  0.0  0.1   0:07.01 httpd
17059 apache    16   0  298m  30m 3564 S  0.0  0.1   0:03.68 httpd

is it "normal" to grab 1.6g of memory? php keeps popping up as well as pulling up to 1.2g, but it then releases it. has anyone seen this behavior before?

[mguthrie]

First, I'd suggest restarting apache.

What is your memory limit set to in your /etc/php.ini file?

Do you have any caching modules installed like memcached or APC?

Code: Select all

yum list installed | grep php 

[kotterbein]

I restarted httpd service earlier and it only took a few minutes for the process to re-stablize at this high consumption rate. I will try again in a bit, but don't want to kick it off just yet.

php memory info:

Code: Select all

;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;

max_execution_time = 30     ; Maximum execution time of each script, in seconds
max_input_time = 60     ; Maximum amount of time each script may spend parsing request data
memory_limit = 128M      ; Maximum amount of memory a script may consume

Code: Select all

[root@ic-nag01 nagios]# yum list installed |grep php
php.x86_64                                  5.1.6-27.el5_5.3           installed
php-cli.x86_64                              5.1.6-27.el5_5.3           installed
php-common.x86_64                           5.1.6-27.el5_5.3           installed
php-devel.x86_64                            5.1.6-27.el5_5.3           installed
php-gd.x86_64                               5.1.6-27.el5_5.3           installed
php-mcrypt.x86_64                           5.1.6-5.el5                installed
php-mssql.x86_64                            5.1.6-5.el5                installed
php-mysql.x86_64                            5.1.6-27.el5_5.3           installed
php-pdo.x86_64                              5.1.6-27.el5_5.3           installed
php-pear.noarch                             1:1.4.9-6.el5              installed
php-pear-HTML-Template-IT.noarch            1.3.0-2.el5                installed
php-pgsql.x86_64                            5.1.6-27.el5_5.3           installed
php-snmp.x86_64                             5.1.6-27.el5_5.3           installed
php-xml.x86_64                              5.1.6-27.el5_5.3           installed

[mguthrie]

The only other times I've seen memory usage for apache this high was when some additional apache modules were installed onto the system. Have you added any other mods to apache since the base install of XI?

[kotterbein]

just attempted this:

This definitely looks better.

Another thing you could do to reduce the load from the httpd process is install an opcode cache, which will reduce compiling every pageload.

To try this run

Code: Select all
yum install php-pecl-apc -y
service httpd restart

[kotterbein]

other modules:

only thing I can think of is the check_interface_table_v3t.pl which serves up it's own interface in conjunction with nagios.

[kotterbein]

after loading php-pecl-apc and restarting apache, it looks to have settled down- page loads are responding much quicker. I will keep an eye on it to see what comes as the day goes on, but so far when I see memory spikes it drops quickly. I still do have quite a high load average on this server, so planned move of checks to passive via NRDS should help level us off a bit.

I'll update later today.

[slansing]

Sounds good Kotterbein let us know, definitely sounds like a strange one.

[kotterbein]

looks a lot cleaner, however the wa time has me a little concerned- bouncing between 0 and 25%

Code: Select all

Tasks: 268 total,   1 running, 267 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.5%us,  0.6%sy,  0.0%ni, 97.8%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  24682292k total, 11565764k used, 13116528k free,  1176740k buffers
Swap: 26738680k total,    40404k used, 26698276k free,  8783572k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4953 mysql     15   0  368m  81m 4264 S  7.0  0.3  22:09.06 mysqld
13594 apache    15   0  336m  34m 3644 S  0.0  0.1   0:26.42 httpd
13596 apache    16   0  336m  34m 3648 S  0.0  0.1   0:26.19 httpd
13595 apache    15   0  332m  31m 3624 S  0.0  0.1   0:30.62 httpd
13597 apache    16   0  332m  31m 3624 S  0.0  0.1   0:26.31 httpd
13598 apache    16   0  332m  30m 3632 S  0.0  0.1   0:26.17 httpd
23102 apache    15   0  332m  30m 3640 S  0.0  0.1   0:15.70 httpd
15105 apache    16   0  332m  30m 3644 S  0.0  0.1   0:21.72 httpd
13601 apache    16   0  332m  30m 3628 S  0.0  0.1   0:21.97 httpd
13599 apache    16   0  332m  30m 3648 S  0.0  0.1   0:17.32 httpd
13600 apache    16   0  326m  25m 3648 S  0.0  0.1   0:13.22 httpd
23015 apache    15   0  326m  25m 3604 S  0.0  0.1   0:06.99 httpd
23101 apache    16   0  326m  24m 3372 S  0.0  0.1   0:07.06 httpd
 2693 nagios    15   0  199m  24m 6056 S  0.0  0.1   0:00.15 php
11474 root      15   0  108m  22m 2928 S  0.0  0.1   0:00.22 mcollectived
 2695 nagios    17   0  194m  20m 6360 S  0.0  0.1   0:00.15 php
 2697 nagios    15   0  194m  20m 6004 S  0.0  0.1   0:00.12 php
 2689 nagios    15   0  194m  19m 6000 S  0.0  0.1   0:00.12 php

Code: Select all

top - 11:14:19 up 2 days,  1:32,  1 user,  load average: 5.62, 6.40, 6.85
Tasks: 273 total,   1 running, 272 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.2%us,  0.7%sy,  0.0%ni, 71.6%id, 24.2%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:  24682292k total, 12645700k used, 12036592k free,  1181312k buffers
Swap: 26738680k total,    40404k used, 26698276k free,  9829792k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4953 mysql     15   0  368m  81m 4264 S  4.0  0.3  22:22.61 mysqld
13596 apache    16   0  336m  34m 3648 S  4.0  0.1   0:27.14 h

looking at offloading to ramdisk the tmp location, check results, and perfdata to see if this cuts down the wa%

[mguthrie]

Yeah, the RAM disk modifications provide a substantial reduction in disk IO.