Really high system load

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
tonyleatwork
Posts: 91
Joined: Mon Jul 07, 2014 8:55 am

Really high system load

Post by tonyleatwork »

Hi -

Like the title says, I'm getting really high system load on my Nagios XI install.

The install was done from the 'official' Nagios XI template (August 2014 build) and has since been upgraded to 2014R2.5 (system profile below). As you can see from the attachment, my system load is ridiculously high.

I have 6 virtual cores assigned to this machine on a brand new Dell Server along with 12GB of ram and 3PAR fiber-based storage.

Code: Select all

 
Output from ps aux

[root@nwd2ng01 ~]# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  19360  1140 ?        Ss   Jan21   0:15 /sbin/init
root         2  0.0  0.0      0     0 ?        S    Jan21   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Jan21   0:07 [migration/0]
root         4  0.0  0.0      0     0 ?        S    Jan21   0:26 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S    Jan21   0:00 [stopper/0]
root         6  0.0  0.0      0     0 ?        S    Jan21   0:01 [watchdog/0]
root         7  0.0  0.0      0     0 ?        S    Jan21   0:07 [migration/1]
root         8  0.0  0.0      0     0 ?        S    Jan21   0:00 [stopper/1]
root         9  0.0  0.0      0     0 ?        S    Jan21   0:06 [ksoftirqd/1]
root        10  0.0  0.0      0     0 ?        S    Jan21   0:00 [watchdog/1]
root        11  0.0  0.0      0     0 ?        S    Jan21   0:06 [migration/2]
root        12  0.0  0.0      0     0 ?        S    Jan21   0:00 [stopper/2]
root        13  0.0  0.0      0     0 ?        S    Jan21   0:07 [ksoftirqd/2]
root        14  0.0  0.0      0     0 ?        S    Jan21   0:01 [watchdog/2]
root        15  0.0  0.0      0     0 ?        S    Jan21   0:06 [migration/3]
root        16  0.0  0.0      0     0 ?        S    Jan21   0:00 [stopper/3]
root        17  0.0  0.0      0     0 ?        S    Jan21   0:06 [ksoftirqd/3]
root        18  0.0  0.0      0     0 ?        S    Jan21   0:00 [watchdog/3]
root        19  0.0  0.0      0     0 ?        S    Jan21   0:06 [migration/4]
root        20  0.0  0.0      0     0 ?        S    Jan21   0:00 [stopper/4]
root        21  0.0  0.0      0     0 ?        S    Jan21   0:06 [ksoftirqd/4]
root        22  0.0  0.0      0     0 ?        S    Jan21   0:01 [watchdog/4]
root        23  0.0  0.0      0     0 ?        S    Jan21   0:06 [migration/5]
root        24  0.0  0.0      0     0 ?        S    Jan21   0:00 [stopper/5]
root        25  0.0  0.0      0     0 ?        S    Jan21   0:06 [ksoftirqd/5]
root        26  0.0  0.0      0     0 ?        S    Jan21   0:00 [watchdog/5]
root        27  0.0  0.0      0     0 ?        S    Jan21   0:35 [events/0]
root        28  0.0  0.0      0     0 ?        S    Jan21   0:38 [events/1]
root        29  0.0  0.0      0     0 ?        S    Jan21   0:50 [events/2]
root        30  0.0  0.0      0     0 ?        S    Jan21   0:40 [events/3]
root        31  0.0  0.0      0     0 ?        S    Jan21   0:49 [events/4]
root        32  0.0  0.0      0     0 ?        S    Jan21   6:32 [events/5]
root        33  0.0  0.0      0     0 ?        S    Jan21   0:00 [cgroup]
root        34  0.0  0.0      0     0 ?        S    Jan21   0:00 [khelper]
root        35  0.0  0.0      0     0 ?        S    Jan21   0:00 [netns]
root        36  0.0  0.0      0     0 ?        S    Jan21   0:00 [async/mgr]
root        37  0.0  0.0      0     0 ?        S    Jan21   0:00 [pm]
root        38  0.0  0.0      0     0 ?        S    Jan21   0:01 [sync_supers]
root        39  0.0  0.0      0     0 ?        S    Jan21   0:02 [bdi-default]
root        40  0.0  0.0      0     0 ?        S    Jan21   0:00 [kintegrityd/0]
root        41  0.0  0.0      0     0 ?        S    Jan21   0:00 [kintegrityd/1]
root        42  0.0  0.0      0     0 ?        S    Jan21   0:00 [kintegrityd/2]
root        43  0.0  0.0      0     0 ?        S    Jan21   0:00 [kintegrityd/3]
root        44  0.0  0.0      0     0 ?        S    Jan21   0:00 [kintegrityd/4]
root        45  0.0  0.0      0     0 ?        S    Jan21   0:00 [kintegrityd/5]
root        46  0.0  0.0      0     0 ?        S    Jan21   1:00 [kblockd/0]
root        47  0.0  0.0      0     0 ?        S    Jan21   0:53 [kblockd/1]
root        48  0.0  0.0      0     0 ?        S    Jan21   0:54 [kblockd/2]
root        49  0.0  0.0      0     0 ?        S    Jan21   0:56 [kblockd/3]
root        50  0.0  0.0      0     0 ?        S    Jan21   0:55 [kblockd/4]
root        51  0.0  0.0      0     0 ?        S    Jan21   0:58 [kblockd/5]
root        52  0.0  0.0      0     0 ?        S    Jan21   0:00 [kacpid]
root        53  0.0  0.0      0     0 ?        S    Jan21   0:00 [kacpi_notify]
root        54  0.0  0.0      0     0 ?        S    Jan21   0:00 [kacpi_hotplug]
root        55  0.0  0.0      0     0 ?        S    Jan21   0:00 [ata_aux]
root        56  0.0  0.0      0     0 ?        S    Jan21   0:00 [ata_sff/0]
root        57  0.0  0.0      0     0 ?        S    Jan21   0:00 [ata_sff/1]
root        58  0.0  0.0      0     0 ?        S    Jan21   0:00 [ata_sff/2]
root        59  0.0  0.0      0     0 ?        S    Jan21   0:00 [ata_sff/3]
root        60  0.0  0.0      0     0 ?        S    Jan21   0:00 [ata_sff/4]
root        61  0.0  0.0      0     0 ?        S    Jan21   0:00 [ata_sff/5]
root        62  0.0  0.0      0     0 ?        S    Jan21   0:00 [ksuspend_usbd]
root        63  0.0  0.0      0     0 ?        S    Jan21   0:00 [khubd]
root        64  0.0  0.0      0     0 ?        S    Jan21   0:00 [kseriod]
root        65  0.0  0.0      0     0 ?        S    Jan21   0:00 [md/0]
root        66  0.0  0.0      0     0 ?        S    Jan21   0:00 [md/1]
root        67  0.0  0.0      0     0 ?        S    Jan21   0:00 [md/2]
root        68  0.0  0.0      0     0 ?        S    Jan21   0:00 [md/3]
root        69  0.0  0.0      0     0 ?        S    Jan21   0:00 [md/4]
root        70  0.0  0.0      0     0 ?        S    Jan21   0:00 [md/5]
root        71  0.0  0.0      0     0 ?        S    Jan21   0:00 [md_misc/0]
root        72  0.0  0.0      0     0 ?        S    Jan21   0:00 [md_misc/1]
root        73  0.0  0.0      0     0 ?        S    Jan21   0:00 [md_misc/2]
root        74  0.0  0.0      0     0 ?        S    Jan21   0:00 [md_misc/3]
root        75  0.0  0.0      0     0 ?        S    Jan21   0:00 [md_misc/4]
root        76  0.0  0.0      0     0 ?        S    Jan21   0:00 [md_misc/5]
root        77  0.0  0.0      0     0 ?        S    Jan21   0:00 [linkwatch]
root        79  0.0  0.0      0     0 ?        S    Jan21   0:00 [khungtaskd]
root        80  0.0  0.0      0     0 ?        S    Jan21   0:17 [kswapd0]
root        81  0.0  0.0      0     0 ?        SN   Jan21   0:00 [ksmd]
root        82  0.0  0.0      0     0 ?        SN   Jan21   1:45 [khugepaged]
root        83  0.0  0.0      0     0 ?        S    Jan21   0:00 [aio/0]
root        84  0.0  0.0      0     0 ?        S    Jan21   0:00 [aio/1]
root        85  0.0  0.0      0     0 ?        S    Jan21   0:00 [aio/2]
root        86  0.0  0.0      0     0 ?        S    Jan21   0:00 [aio/3]
root        87  0.0  0.0      0     0 ?        S    Jan21   0:00 [aio/4]
root        88  0.0  0.0      0     0 ?        S    Jan21   0:00 [aio/5]
root        89  0.0  0.0      0     0 ?        S    Jan21   0:00 [crypto/0]
root        90  0.0  0.0      0     0 ?        S    Jan21   0:00 [crypto/1]
root        91  0.0  0.0      0     0 ?        S    Jan21   0:00 [crypto/2]
root        92  0.0  0.0      0     0 ?        S    Jan21   0:00 [crypto/3]
root        93  0.0  0.0      0     0 ?        S    Jan21   0:00 [crypto/4]
root        94  0.0  0.0      0     0 ?        S    Jan21   0:00 [crypto/5]
root       102  0.0  0.0      0     0 ?        S    Jan21   0:00 [kthrotld/0]
root       103  0.0  0.0      0     0 ?        S    Jan21   0:00 [kthrotld/1]
root       104  0.0  0.0      0     0 ?        S    Jan21   0:00 [kthrotld/2]
root       105  0.0  0.0      0     0 ?        S    Jan21   0:00 [kthrotld/3]
root       106  0.0  0.0      0     0 ?        S    Jan21   0:00 [kthrotld/4]
root       107  0.0  0.0      0     0 ?        S    Jan21   0:00 [kthrotld/5]
root       108  0.0  0.0      0     0 ?        S    Jan21   0:00 [pciehpd]
root       110  0.0  0.0      0     0 ?        S    Jan21   0:00 [kpsmoused]
root       111  0.0  0.0      0     0 ?        S    Jan21   0:00 [usbhid_resume]
root       112  0.0  0.0      0     0 ?        S    Jan21   0:00 [deferwq]
root       143  0.0  0.0      0     0 ?        S    Jan21   0:00 [kdmremove]
root       144  0.0  0.0      0     0 ?        S    Jan21   0:00 [kstriped]
root       294  0.0  0.0      0     0 ?        S    Jan21   0:00 [scsi_eh_0]
root       296  0.0  0.0      0     0 ?        S    Jan21   0:00 [scsi_eh_1]
root       377  0.0  0.0      0     0 ?        S    Jan21   0:12 [mpt_poll_0]
root       378  0.0  0.0      0     0 ?        S    Jan21   0:00 [mpt/0]
root       379  0.0  0.0      0     0 ?        S    Jan21   0:00 [scsi_eh_2]
root       454  0.0  0.0      0     0 ?        S    Jan21   0:00 [kdmflush]
root       456  0.0  0.0      0     0 ?        S    Jan21   0:00 [kdmflush]
root       473  0.0  0.0      0     0 ?        S    Jan21   2:57 [jbd2/dm-0-8]
root       474  0.0  0.0      0     0 ?        S    Jan21   0:00 [ext4-dio-unwr]
nagios     543 13.8  0.0      0     0 ?        Z    23:57   0:08 [che] <defunct>
root       551  0.0  0.0  11036   384 ?        S<s  Jan21   0:00 /sbin/udevd -d
postgres   643  0.0  0.0 217824  6404 ?        Ss   21:59   0:01 postgres: nagio
nagios     729  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
root       779  0.0  0.0      0     0 ?        S    Jan21   0:08 [vmmemctl]
postgres   796  0.0  0.0 217824  6040 ?        Ss   21:59   0:01 postgres: nagio
nagios     820  0.1  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios     826  0.1  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios     850  0.1  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios     903  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios     920  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
root       935  0.0  0.0  11036   400 ?        S<   Jan21   0:00 /sbin/udevd -d
root       966  0.0  0.0      0     0 ?        S    Jan21   0:00 [jbd2/sda1-8]
root       967  0.0  0.0      0     0 ?        S    Jan21   0:00 [ext4-dio-unwr]
root       968  0.0  0.0      0     0 ?        S    Jan21   0:02 [jbd2/sdb1-8]
root       969  0.0  0.0      0     0 ?        S    Jan21   0:00 [ext4-dio-unwr]
root      1003  0.0  0.0      0     0 ?        S    Jan21   0:09 [kauditd]
root      1097  0.1  0.0      0     0 ?        S    Jan21  20:40 [flush-253:0]
root      1549  0.0  0.0  93156   748 ?        S<sl Jan21   0:36 auditd
root      1569  0.0  0.0 249860  5436 ?        Sl   Jan21   0:13 /sbin/rsyslogd
dbus      1582  0.0  0.0  21432   376 ?        Ss   Jan21   0:00 dbus-daemon --s
root      1624  0.0  0.0  66216   620 ?        Ss   Jan21   0:00 /usr/sbin/sshd
root      1633  0.0  0.0  21716   640 ?        Ss   Jan21   0:00 xinetd -stayali
ntp       1657  0.0  0.0  30732  1564 ?        Ss   Jan21   0:00 ntpd -u ntp:ntp
root      1692  0.0  0.0 108168  1292 ?        S    Jan21   0:00 /bin/sh /usr/bi
mysql     1797  1.9  0.4 3285956 55028 ?       Sl   Jan21 372:24 /usr/libexec/my
postgres  1836  0.0  0.0 215872  5352 ?        S    Jan21   1:39 /usr/bin/postma
root      1916  0.0  0.0  80856  2536 ?        Ss   Jan21   0:02 /usr/libexec/po
postfix   1926  0.0  0.0  81108  2708 ?        S    Jan21   0:00 qmgr -l -t fifo
postgres  1927  0.0  0.0 178884   680 ?        Ss   Jan21   0:15 postgres: logge
postgres  1929  0.0  0.1 215992 13864 ?        Ss   Jan21   1:56 postgres: write
postgres  1930  0.0  0.0 215872   960 ?        Ss   Jan21   1:10 postgres: wal w
postgres  1931  0.0  0.0 216160  1180 ?        Ss   Jan21   0:29 postgres: autov
postgres  1932  0.0  0.0 179152   888 ?        Ss   Jan21   1:53 postgres: stats
root      1943  0.0  0.0 116888  1040 ?        Ss   Jan21   0:24 crond
ajaxterm  1971  0.0  0.0 170340  1644 ?        Sl   Jan21   3:19 python /usr/sha
nagios    2079  0.0  0.0  49824   348 ?        Ss   Jan21   0:00 /usr/local/nagi
root      2088  0.0  0.0   4064   500 tty1     Ss+  Jan21   0:00 /sbin/mingetty
root      2090  0.0  0.0   4064   500 tty2     Ss+  Jan21   0:00 /sbin/mingetty
root      2092  0.0  0.0  11032   276 ?        S<   Jan21   0:00 /sbin/udevd -d
root      2093  0.0  0.0   4064   500 tty3     Ss+  Jan21   0:00 /sbin/mingetty
root      2095  0.0  0.0   4064   500 tty4     Ss+  Jan21   0:00 /sbin/mingetty
root      2097  0.0  0.0   4064   500 tty5     Ss+  Jan21   0:00 /sbin/mingetty
root      2099  0.0  0.0   4064   500 tty6     Ss+  Jan21   0:00 /sbin/mingetty
nagios    2118  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    2134  0.0  0.0  37296  2464 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    2217  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    2254  0.0  0.0  39404  2864 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    2285  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    2323  0.0  0.0  33012  2128 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    2509  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    2542  0.0  0.0  37296  2464 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    2623  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    2653  0.0  0.0  37296  2464 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    2973  0.7  0.0  32744 11080 ?        Ss   23:44   0:05 /usr/local/nagi
nagios    2975 50.6  0.0  10124  1208 ?        R    23:44   6:46 /usr/local/nagi
nagios    2976 51.9  0.0  10144  1140 ?        R    23:44   6:57 /usr/local/nagi
nagios    2977 49.9  0.0  10260  1240 ?        R    23:44   6:41 /usr/local/nagi
nagios    2978 52.6  0.0  10176  1144 ?        R    23:44   7:02 /usr/local/nagi
nagios    2979 52.4  0.0  10164  1252 ?        R    23:44   7:01 /usr/local/nagi
nagios    2980 52.2  0.0  10228  1236 ?        R    23:44   6:59 /usr/local/nagi
nagios    2981 57.4  0.0  10264  1272 ?        R    23:44   7:40 /usr/local/nagi
nagios    2982 56.9  0.0  10112  1200 ?        R    23:44   7:37 /usr/local/nagi
nagios    2983 53.1  0.0  10260  1256 ?        R    23:44   7:07 /usr/local/nagi
nagios    2986  0.0  0.0  49824  1220 ?        S    23:44   0:00 /usr/local/nagi
nagios    2988  1.5  0.0  51820  3300 ?        S    23:44   0:12 /usr/local/nagi
nagios    3031  0.0  0.0  31348  8976 ?        S    23:44   0:00 /usr/local/nagi
nagios    3117  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    3151  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    3159  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    3163  0.4  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    3348  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    3420  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    3466  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    3528  0.4  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    3550  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    3596  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    3605  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    3610  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    3611  0.2  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    4124  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    4155  0.0  0.0  39404  2860 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    4386  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    4435  0.0  0.0  37296  2464 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    4524  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    4533  0.0  0.0  33012  2132 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    4537  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    4548  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    4549  0.0  0.0  37296  2468 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    4556  0.0  0.0  37296  2468 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    4563  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    4569  0.0  0.0  37296  2464 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    4570  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    4580  0.0  0.0  37296  2464 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    4708  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    4712  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    4728  0.0  0.0  33012  2128 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    4735  0.0  0.0  37296  2464 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    4832  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    4836  0.0  0.0  39400  2936 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    4848  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    4853  0.0  0.0  37296  2464 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    5157  0.3  0.1 151704 13580 ?        S    23:57   0:00 /usr/bin/perl -
nagios    5177  0.4  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    5183  0.4  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    5246  0.7  0.1 152232 14144 ?        S    23:57   0:00 /usr/bin/perl -
nagios    5259  0.4  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    5274  0.4  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    5577  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    5584  0.0  0.0  33012  2128 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    5667  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    5682  0.0  0.0  37296  2464 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    5695  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    5720  0.0  0.0  37292  2464 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    5764  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    5765  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    5792  0.0  0.0  33012  2128 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    5798  0.0  0.0  33012  2128 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    5888  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    5890  0.0  0.0  33012  2116 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    5933  0.5  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    5962  0.5  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    6030  0.4  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    6034  0.6  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    6036  0.4  0.1 151720 13580 ?        S    23:57   0:00 /usr/bin/perl -
nagios    6050  0.4  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    6071  0.5  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    6086  0.9  0.1 152248 14148 ?        S    23:57   0:00 /usr/bin/perl -
nagios    6089  0.4  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
root      6391  0.0  0.1 335380 13520 ?        Ss   Feb02   0:04 /usr/sbin/httpd
nagios    6464  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    6471  0.0  0.0  39404  2856 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    6499  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    6512  0.0  0.0  37296  2464 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    6514  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    6534  0.0  0.0  39404  2864 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    6581  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    6611  0.0  0.0  39404  2860 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    6683  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    6703  0.0  0.0  33012  2128 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    6755  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    6758  0.0  0.0  33012  2132 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    6858  0.5  0.1 151704 13584 ?        S    23:57   0:00 /usr/bin/perl -
nagios    6912  0.5  0.0      0     0 ?        Z    23:57   0:00 [che] <defunct>
nagios    6916  0.6  0.1 151720 13580 ?        S    23:57   0:00 /usr/bin/perl -
apache    7182  2.0  0.3 465148 44644 ?        S    23:35   0:27 /usr/sbin/httpd
apache    7183  1.5  0.4 478068 57136 ?        S    23:35   0:20 /usr/sbin/httpd
apache    7305  1.3  0.3 466136 45736 ?        S    21:27   2:02 /usr/sbin/httpd
nagios    7600  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    7601  0.0  0.0  39404  2860 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    7659  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    7662  0.0  0.0  33012  2116 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    7697  0.0  0.0 106060  1296 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    7710  0.0  0.0  33012  2128 ?        S    23:57   0:00 /usr/bin/wmic -
nagios    7890  1.7  0.1 152224 14144 ?        S    23:57   0:00 /usr/bin/perl -
nagios    7891  0.0  0.0 106060  1292 ?        S    23:57   0:00 sh -c /usr/bin/
nagios    7892  0.0  0.0  37296  2468 ?        S    23:57   0:00 /usr/bin/wmic -
postgres  7932  0.0  0.0 217796  5832 ?        Ss   21:27   0:02 postgres: nagio
nagios    7934  1.1  0.0      0     0 ?        Z    23:58   0:00 [che] <defunct>
root      7955  0.0  0.0 139776  1608 ?        S    23:58   0:00 CROND
root      7956  0.0  0.0 139776  1608 ?        S    23:58   0:00 CROND
root      7957  0.0  0.0 139776  1608 ?        S    23:58   0:00 CROND
root      7958  0.0  0.0 139776  1608 ?        S    23:58   0:00 CROND
root      7959  0.0  0.0 139776  1608 ?        S    23:58   0:00 CROND
nagios    7961  0.0  0.0 106060  1272 ?        Ss   23:58   0:00 /bin/sh -c /usr
nagios    7962  0.0  0.0 106060  1268 ?        Ss   23:58   0:00 /bin/sh -c /usr
nagios    7963  0.0  0.0 106060  1272 ?        Ss   23:58   0:00 /bin/sh -c /usr
nagios    7964  0.0  0.0 106060  1268 ?        Ss   23:58   0:00 /bin/sh -c /usr
nagios    7967  0.0  0.0 106060  1268 ?        Ss   23:58   0:00 /bin/sh -c /usr
nagios    7970  0.8  0.1 319468 22812 ?        S    23:58   0:00 /usr/bin/php -q
nagios    7973  0.8  0.1 319296 22312 ?        S    23:58   0:00 /usr/bin/php -q
nagios    7975  1.2  0.2 326752 30068 ?        S    23:58   0:00 /usr/bin/php -q
nagios    7976  0.8  0.1 319176 22484 ?        S    23:58   0:00 /usr/bin/php -q
nagios    7977  0.9  0.1 319228 22908 ?        S    23:58   0:00 /usr/bin/php -q
postgres  7980  0.0  0.0 217160  4984 ?        Ss   23:58   0:00 postgres: nagio
postgres  7981  0.0  0.0 217160  4992 ?        Ss   23:58   0:00 postgres: nagio
postgres  7993  0.0  0.0 217300  5348 ?        Ss   23:58   0:00 postgres: nagio
postgres  7997  0.0  0.0 217160  4988 ?        Ss   23:58   0:00 postgres: nagio
postgres  8005  0.0  0.0 217260  5792 ?        Ss   23:58   0:00 postgres: nagio
nagios    8102  2.7  0.0      0     0 ?        Z    23:58   0:00 [che] <defunct>
root      8108  0.0  0.0 110236  1168 pts/0    R+   23:58   0:00 ps aux
postgres  9584  0.0  0.0 217824  6616 ?        Ss   23:35   0:00 postgres: nagio
postgres  9798  0.0  0.0 217760  5640 ?        Ss   23:35   0:00 postgres: nagio
apache   10406  1.6  0.3 465204 44888 ?        S    18:58   4:50 /usr/sbin/httpd
postgres 11399  0.0  0.0 217824  6396 ?        Ss   18:58   0:04 postgres: nagio
postfix  12230  0.0  0.0  80936  3368 ?        S    23:36   0:00 pickup -l -t fi
apache   15386  1.6  0.2 454636 33880 ?        S    23:49   0:08 /usr/sbin/httpd
apache   15597  1.6  0.2 455060 34216 ?        S    23:49   0:08 /usr/sbin/httpd
apache   15817  1.2  0.2 446648 26168 ?        S    23:49   0:06 /usr/sbin/httpd
apache   15818  1.6  0.2 455412 34632 ?        S    23:49   0:08 /usr/sbin/httpd
apache   15819  1.7  0.2 453524 33044 ?        S    23:49   0:08 /usr/sbin/httpd
apache   16245  1.1  0.2 452392 31648 ?        S    23:49   0:05 /usr/sbin/httpd
apache   16550  2.1  0.3 465092 44548 ?        S    23:49   0:10 /usr/sbin/httpd
apache   16552  1.1  0.2 450996 30424 ?        S    23:49   0:05 /usr/sbin/httpd
postgres 16945  0.0  0.0 217760  5652 ?        Ss   23:49   0:00 postgres: nagio
apache   17606  1.7  0.4 479336 58652 ?        S    16:21   8:11 /usr/sbin/httpd
postgres 17839  0.0  0.0 217244  5488 ?        Ss   23:49   0:00 postgres: nagio
postgres 17985  0.0  0.0 217244  5492 ?        Ss   23:49   0:00 postgres: nagio
postgres 18013  0.0  0.0 217244  5484 ?        Ss   23:49   0:00 postgres: nagio
postgres 18188  0.0  0.0 217244  5484 ?        Ss   23:49   0:00 postgres: nagio
postgres 18192  0.0  0.0 217824  6568 ?        Ss   16:21   0:07 postgres: nagio
postgres 18221  0.0  0.0 217244  5484 ?        Ss   23:49   0:00 postgres: nagio
postgres 18483  0.0  0.0 217244  5488 ?        Ss   23:49   0:00 postgres: nagio
postgres 18500  0.0  0.0 217244  5488 ?        Ss   23:49   0:00 postgres: nagio
apache   21276  1.7  0.4 479120 58236 ?        S    16:56   7:20 /usr/sbin/httpd
postgres 21781  0.0  0.0 217760  5900 ?        Ss   16:56   0:07 postgres: nagio
apache   22905  1.2  0.2 452640 31764 ?        S    23:39   0:13 /usr/sbin/httpd
postgres 23319  0.0  0.0 217760  5628 ?        Ss   23:39   0:00 postgres: nagio
nagios   26035  0.0  0.0 368888   960 ?        S    Feb02   0:10 /usr/local/nagi
nagios   30130  0.0  0.0      0     0 ?        Z    23:54   0:00 [che] <defunct>
nagios   30155  0.0  0.0      0     0 ?        Z    23:54   0:00 [che] <defunct>
apache   30221  1.6  0.3 466616 46288 ?        S    16:59   6:56 /usr/sbin/httpd
apache   30263  1.6  0.3 465224 44880 ?        S    17:10   6:48 /usr/sbin/httpd
postgres 30366  0.0  0.0 217244  5908 ?        Ss   17:10   0:07 postgres: nagio
root     30445  0.0  0.0 102504  4820 ?        Ss   15:05   0:00 sshd: root@pts/
apache   30480  1.7  0.4 479120 58288 ?        S    16:59   7:21 /usr/sbin/httpd
root     30515  0.0  0.0 108436  2012 pts/0    Ss   15:05   0:00 -bash
nagios   30552  0.0  0.0 106060  1292 ?        S    23:54   0:00 sh -c /usr/bin/
postgres 30572  0.0  0.0 217796  6444 ?        Ss   16:59   0:07 postgres: nagio
nagios   30592  0.0  0.0  37296  2468 ?        S    23:54   0:00 /usr/bin/wmic -
nagios   30657  0.0  0.0 106060  1288 ?        S    23:54   0:00 sh -c /usr/bin/
nagios   30674  0.0  0.0  37296  2472 ?        S    23:54   0:00 /usr/bin/wmic -
postgres 30853  0.0  0.0 217828  6472 ?        Ss   16:59   0:07 postgres: nagio
nagios   30986  0.0  0.0      0     0 ?        Z    23:54   0:00 [che] <defunct>
nagios   30993  0.0  0.0      0     0 ?        Z    23:54   0:00 [che] <defunct>
nagios   31886  0.0  0.0 106060  1296 ?        S    23:54   0:00 sh -c /usr/bin/
nagios   31897  0.0  0.0  37296  2468 ?        S    23:54   0:00 /usr/bin/wmic -
apache   32142  1.6  0.4 479120 58288 ?        S    21:59   1:55 /usr/sbin/httpd
nagios   32398  0.0  0.0      0     0 ?        Z    23:55   0:00 [che] <defunct>
apache   32689  1.6  0.3 465144 44832 ?        S    21:59   1:57 /usr/sbin/httpd
[root@nwd2ng01 ~]#

Code: Select all

Nagios XI Installation Profile
Download Profile	
System:
Nagios XI Version : 2014R2.5
nwd2ng01.corp.analog.com 2.6.32-504.3.3.el6.x86_64 x86_64
CentOS release 6.6 (Final)
Gnome is not installed
Apache Information
PHP Version: 5.3.3
Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0
Server Name: nwd2ng01.corp.analog.com
Server Address: 10.64.52.120
Server Port: 80
Date/Time
PHP Timezone: America/New_York
PHP Time: Tue, 03 Feb 2015 23:55:34 -0500
System Time: Tue, 03 Feb 2015 23:55:34 -0500
Nagios XI Data
License ends in: MSTNQS

nagios (pid 2973) is running...
NPCD running (pid 26035).
ndo2db (pid 2079) is running...
CPU Load 15: 26.16
Total Hosts: 511
Total Services: 3369
Function 'get_base_uri' returns: http://nwd2ng01.corp.analog.com/nagiosxi/
Function 'get_base_url' returns: http://nwd2ng01.corp.analog.com/nagiosxi/
Function 'get_backend_url(internal_call=false)' returns: http://nwd2ng01.corp.analog.com/nagiosxi/includes/components/profile/profile.php
Function 'get_backend_url(internal_call=true)' returns: http://localhost/nagiosxi/backend/
Ping Test localhost
Running:

/bin/ping -c 3 localhost 2>&1 

PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.017 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.016 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.014 ms

--- localhost ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.014/0.015/0.017/0.004 ms
Test wget To localhost
WGET From URL: http://localhost/nagiosxi/includes/components/ccm/
Running:

/usr/bin/wget http://localhost/nagiosxi/includes/components/ccm/ 

--2015-02-03 23:55:36-- http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: "/usr/local/nagiosxi/tmp/ccm_index.tmp"

0K ........ 699M=0s

2015-02-03 23:55:36 (699 MB/s) - "/usr/local/nagiosxi/tmp/ccm_index.tmp" saved [8385]

Network Settings

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:50:56:9f:52:ef brd ff:ff:ff:ff:ff:ff

    inet 10.64.52.120/24 brd 10.64.52.255 scope global eth0

    inet6 fe80::250:56ff:fe9f:52ef/64 scope link 

       valid_lft forever preferred_lft forever


Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

10.64.52.0      *               255.255.255.0   U     0      0        0 eth0

link-local      *               255.255.0.0     U     1002   0        0 eth0

default         10.64.52.1      0.0.0.0         UG    0      0        0 eth0


You do not have the required permissions to view the files attached to this post.
tonyleatwork
Posts: 91
Joined: Mon Jul 07, 2014 8:55 am

Re: Really high system load

Post by tonyleatwork »

Update, I just went through the performance 'tuning' exercise of creating a ramdisk. Cant tell if it's helping or not, load is up and down - mostly up.
Fred Kroeger
Posts: 588
Joined: Wed Oct 19, 2011 11:36 pm
Location: Perth, Western Australia
Contact:

Re: Really high system load

Post by Fred Kroeger »

Have you tried running the Nagios VM with a smaller number of CPU's - eg: 2 or 3 ?
This is a quick test to see if your ESX Host is the bottleneck.
tonyleatwork
Posts: 91
Joined: Mon Jul 07, 2014 8:55 am

Re: Really high system load

Post by tonyleatwork »

Fred Kroeger wrote:Have you tried running the Nagios VM with a smaller number of CPU's - eg: 2 or 3 ?
This is a quick test to see if your ESX Host is the bottleneck.
We did - initially the VM had 2 virtual CPUs and grew to 4, then 6 to address the contention issues we saw. The last increase (from 4 to 6) was because I had a high number of <defunct> checks zombifying (redundant, I know).

Now my defunct list is hovering around a 'manageable' 40-50 and getting reaped. Before I was getting all sorts defunct processes and WPROC errors in syslog.
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Really high system load

Post by lmiltchev »

tonyleatwork, do you mind opening a new support ticket in our email ticketing system? Email us the "profile.zip" at [email protected].

Admin->System Profile->Download Profile

Paste a link to this post in your email and type "Really high system load" in the emails subject field. Thanks!
Be sure to check out our Knowledgebase for helpful articles and solutions!
Locked