find process flooding my system

[ytlcomm]

Hi,

My system got flooded with this 'find' process. I have no idea how to solve this problem. Load average is ridiculously high. Please help.

12-5-2012 11-47-51 PM.png

- Faiz

[scottwilkerson]

I would start by killing the find processes.

Code: Select all

killall -9 find

It is possible because the load is so high it cannot complete this standard maintenance task

[ytlcomm]

I have kill it several time and it coming back again.

[mguthrie]

Can you show the output from:

Code: Select all

ps aux | grep php

[ytlcomm]

Code: Select all

# ps aux | grep php
nagios     635  0.0  0.0   9228   608 ?        Ss   10:38   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios     643  0.0  0.0 222708  5176 ?        S    10:38   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios    1201  0.0  0.0   9228   592 ?        Ss   10:29   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios    1206  0.0  0.0 222708  3016 ?        S    10:29   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios    1361  0.0  0.0   9228   616 ?        Ss   10:43   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios    1362  0.0  0.0 222708  2948 ?        S    10:43   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios    2457  0.0  0.0   9228   580 ?        Ss   10:10   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios    2465  0.0  0.0 222708  1360 ?        S    10:10   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios    3006  0.0  0.0   9228   580 ?        Ss   10:11   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios    3011  0.0  0.0 222708  1340 ?        S    10:11   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios    5073  0.0  0.0   9228   580 ?        Ss   10:12   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios    5081  0.0  0.0 222708  1332 ?        S    10:12   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios    5446  0.0  0.0   9228   624 ?        Ss   10:16   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios    5454  0.0  0.0 222708  5704 ?        S    10:16   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios    5706  0.0  0.0   9228   600 ?        Ss   10:49   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios    5711  0.0  0.1 222708 10260 ?        S    10:49   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios    5904  0.0  0.0   9228   580 ?        Ss   10:18   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios    5910  0.0  0.0 222708  2184 ?        S    10:18   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios    6458  0.0  0.0   9228   600 ?        Ss   10:44   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios    6469  0.0  0.0 222708  6132 ?        S    10:44   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios    8413  0.0  0.0   9228   692 ?        Ss   10:24   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios    8425  0.0  0.0 222708  7248 ?        S    10:24   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios    8535  0.0  0.0   9228   580 ?        Ss   10:13   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios    8542  0.0  0.0 222708  1284 ?        S    10:13   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios    9070  0.0  0.0   9228   640 ?        Ss   10:20   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios    9078  0.0  0.1 222708 12688 ?        S    10:20   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   10423  0.0  0.0   9228   580 ?        Ss   10:21   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   10429  0.0  0.0 222708  1812 ?        S    10:21   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   10818  0.0  0.0   9228   588 ?        Ss   10:30   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   10823  0.0  0.0 222708  2748 ?        S    10:30   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   10906  0.0  0.0   9228   584 ?        Ss   10:35   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   10921  0.0  0.0 222708  4644 ?        S    10:35   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   12065  0.0  0.0   9228   596 ?        Ss   10:31   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   12072  0.0  0.0 222708  1568 ?        S    10:31   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   12458  0.0  0.0   9228   596 ?        Ss   10:45   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   12467  0.0  0.0 222668  3400 ?        S    10:45   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   13797  0.0  0.0   9228   584 ?        Ss   10:32   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   13801  0.0  0.1 222708 15168 ?        S    10:32   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   14387  0.0  0.0   9228   600 ?        Ss   10:46   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   14394  0.0  0.0 222708  3160 ?        S    10:46   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   15174  0.0  0.0   9228   596 ?        Ss   10:36   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   15178  0.0  0.0 222708  2852 ?        S    10:36   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   15578  0.0  0.0   9228   700 ?        Ss   10:22   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   15589  0.0  0.1 222708 13328 ?        S    10:22   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   15643  0.0  0.0   9228   580 ?        Ss   10:04   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   15648  0.0  0.0 222708  1284 ?        S    10:04   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   15896  0.0  0.0   9228   604 ?        Ss   10:25   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   15904  0.0  0.0 222708  2236 ?        S    10:25   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   16701  0.0  0.0   9228   596 ?        Ss   10:26   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   16709  0.0  0.1 222708 10556 ?        S    10:26   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   17518  0.0  0.0   9228   580 ?        Ss   10:08   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   17524  0.0  0.0 222708  1284 ?        S    10:08   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   17983  0.0  0.0   9228   628 ?        Ss   10:39   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   17992  0.0  0.0 222708  6056 ?        S    10:39   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
root     18354  0.0  0.0 103248   860 pts/4    S+   12:07   0:00 grep php
nagios   19545  0.0  0.0   9228   580 ?        Ss   10:05   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   19553  0.0  0.0 222708  1288 ?        S    10:05   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   19892  0.0  0.0   9228   580 ?        Ss   10:14   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   19896  0.0  0.0 222708  1284 ?        S    10:14   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   20142  0.0  0.0   9228   580 ?        Ss   10:06   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   20149  0.0  0.0 222708  1284 ?        S    10:06   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   20450  0.0  0.0   9228   596 ?        Ss   10:33   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   20453  0.0  0.0 222708  3080 ?        S    10:33   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   20562  0.0  0.0   9228   580 ?        Ss   10:17   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   20573  0.0  0.0 222708  1284 ?        S    10:17   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   20922  0.0  0.0   9228   668 ?        Ss   10:54   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   20930  0.0  0.1 222708 14912 ?        S    10:54   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   21064  0.0  0.0   9228   596 ?        Ss   10:27   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   21066  0.0  0.0 222708  3632 ?        S    10:27   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   21132  0.0  0.0   9228   684 ?        Ss   10:37   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   21139  0.0  0.1 222708 11896 ?        S    10:37   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   21794  0.0  0.0   9228   584 ?        Ss   10:19   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   21800  0.0  0.0 222708  4764 ?        S    10:19   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   24515  0.0  0.0   9228   592 ?        Ss   10:48   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   24527  0.0  0.0 222708  4076 ?        S    10:48   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   27377  0.0  0.0   9228   604 ?        Ss   10:28   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   27381  0.0  0.0 222708  3176 ?        S    10:28   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   27619  0.0  0.0   9228   580 ?        Ss   10:07   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   27624  0.0  0.0 222708  1300 ?        S    10:07   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   28940  0.0  0.0   9228   580 ?        Ss   10:15   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   28948  0.0  0.0 222708  1284 ?        S    10:15   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   29246  0.0  0.0   9228   620 ?        Ss   10:40   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   29255  0.0  0.0 222708  4924 ?        S    10:40   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   30531  0.0  0.0   9228   604 ?        Ss   10:34   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   30539  0.0  0.0 222708  4492 ?        S    10:34   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   31151  0.0  0.0   9228   616 ?        Ss   10:23   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   31154  0.0  0.2 222708 19224 ?        S    10:23   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   31317  0.0  0.0   9228   608 ?        Ss   10:41   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   31322  0.0  0.0 222708  5188 ?        S    10:41   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   32506  0.0  0.0   9228   608 ?        Ss   10:42   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   32517  0.0  0.0 222708  5388 ?        S    10:42   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php
nagios   32548  0.0  0.0   9228   580 ?        Ss   10:09   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
nagios   32553  0.0  0.0 222708  1392 ?        S    10:09   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php

Please help...

[mguthrie]

Yikes. Yeah that's the problem. For the moment, run the following:

Code: Select all

service crond stop
killall php
service crond start

Can you manually run the following cron job and see what kind of output you get?

Code: Select all

php /usr/local/nagiosxi/cron/cleaner.php

[ytlcomm]

Its showing some out put of 'No such file or directory'. Should I run it endlessly? Find process accumulating... I've commented cleaner.php in cron file for temporary.

12-7-2012 10-15-43 AM.png

[mguthrie]

How many files do you have in the following two directories?

Code: Select all

ls -f /etc/nagiosql/backup/hosts | wc -l
ls -f /etc/nagiosql/backup/hosts | wc -l

I'm wondering if there's just too many files in there where it gets stuck on an endless loop...

[ytlcomm]

Code: Select all

# ls -f /etc/nagiosql/backup/hosts | wc -l
310975

Yeah, it's quite a lot of files there. But I guess we don't have that many hosts.

[mguthrie]

I would go ahead and remove those backup config files in that directory. We added a feature recently where you can restore a database snapshot of the configs, which is far more effective than keeping the old .cfg files around. I'm betting once that directory is cleaned up that issue will disappear.