Nagioxi CPU consumption strongly increased

[Frédéric GRANAT]

Hi,
I noticed that, Since the middle of August, the CPU consumption of the server hosting Nagios XI increased strongly as you can see.

Please help me to solve that problem.

Rgds,

Frederic

[bwallace]

We'll need some more info in order to troubleshoot. What are the system specs - number of CPUs, drive space,etc?

While the system is in a high CPU state, please obtain the following and post here:
- screenshots of 'System Status' and 'Monitoring Engine Status' under Admin > System Information
- top output
- Around the time of the spike, what is being recorded in the event log?
- Home > Monitoring Process > Event Log
- nagios.log files from when spikes occur could be helpful as well.

* Don't forget to scrub sensitive data prior to posting *

[Frédéric GRANAT]

Code: Select all

[root@nagiosxi ~]# cat /proc/cpuinfo | grep processor | wc -l
4
[root@nagiosxi ~]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      29329476  23385300   4430576  85% /
/dev/sda1               101086     31686     64181  34% /boot
tmpfs                  1553548         0   1553548   0% /dev/shm

- screenshots of 'System Status' and 'Monitoring Engine Status' under Admin > System Information
=> Please see the attached files

- Top output

Code: Select all

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
17221 nagios    25   0  3080  820  592 R 86.5  0.0 514:50.10 nagios
17220 nagios    25   0  3080  828  592 R 85.2  0.0 497:59.53 nagios
24717 postgres  16   0 21992  11m  10m R 33.9  0.4   2:11.16 postmaster
24737 postgres  16   0 23012  12m  10m R 33.6  0.4   2:13.74 postmaster
17651 nagios    18   0 14464  10m 1976 S  5.3  0.3   0:00.16 check_wmi_plus.
24632 apache    15   0 66360  25m 5448 S  5.3  0.8   0:23.08 httpd
17652 nagios    17   0 11992 7820 1960 R  4.0  0.3   0:00.12 check_wmi_plus.
 2511 apache    15   0 65876  25m 5608 S  3.7  0.8   1:26.05 httpd
24676 apache    15   0 65640  25m 5448 S  3.3  0.8   0:23.72 httpd
 2639 postgres  17   0 22240  11m  10m R  1.7  0.4   7:16.61 postmaster
  651 root      10  -5     0    0    0 S  0.3  0.0  10:53.22 kjournald
17218 nagios    15   0 11808 3964 1372 S  0.3  0.1  13:51.37 nagios

- Around the time of the spike, what is being recorded in the event log?
=> The CPU consumption is constant
- Home > Monitoring Process > Event Log
=> Please see the attached file

- nagios.log files from when spikes occur could be helpful as well.
=> where could I find that ?

[ssax]

Please post the output of these commands:

Code: Select all

ps aux |grep nagios.cfg
ipcs -q

Do you see any crashed tables messages in /var/log/mysqld.log or /var/log/mariadb/mariadb.log? If so, please run a repair:

Code: Select all

service nagios stop
killall -9 nagios
service ndo2db stop
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
service mysqld restart
cd /usr/local/nagiosxi/scripts
./repair_databases.sh

This will automatically start the services back up so go ahead and test after that.

Are the cron jobs running, are you seeing anything in /var/log/cron? What about in /var/log/messages, /var/log/httpd/error_log, or /var/log/httpd/ssl_error_log, are you seeing anything that could be related?


Thank you

[Frédéric GRANAT]

Code: Select all

[root@nagiosxi services]# ps aux |grep nagios.cfg
root     11806  0.0  0.0   4024   776 pts/0    S+   08:41   0:00 grep nagios.cfg
nagios   17218  0.3  0.1  11808  3964 ?        Ss   Aug26  16:20 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   17232  0.0  0.0  11192  2636 ?        S    Aug26   0:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
[root@nagiosxi services]# ipcs -q

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x61000002 0          nagios     600        0            0
0x3e000002 65537      nagios     600        0            0
0xe3000002 98306      nagios     600        0            0
0xe6000002 131075     nagios     600        0            0
0x91000002 163844     nagios     600        0            0
0x6a000002 294917     nagios     600        0            0
[root@nagiosxi services]#

Do you see any crashed tables messages in /var/log/mysqld.log
=> Yes

[root@nagiosxi services]# cd /var/log/mariadb
-bash: cd: /var/log/mariadb: No such file or directory
=> mariadb doesn't exist

please run a repair:
=> Done, but the CPU climbed at the same level than before repair

are you seeing anything in /var/log/cron?
=> Yes

What about in /var/log/messages
=> It's not empty but I don't know how to collect usefull information in it

What about in /var/log/httpd/error_log
=> Same thing

What about in /var/log/httpd/ssl_error_log
=> Same thing

[tmcdonald]

Let's restart cron:

service crond restart

then check the logs and post the output here:

Code: Select all

tail -20 /var/log/messages
tail -20 /var/log/httpd/error_log
tail -20 /var/log/httpd/ssl_error_log

Let the system catch up for a bit, then run:

Code: Select all

tail -20 /usr/local/nagiosxi/var/eventman.log
tail -20 /usr/local/nagiosxi/var/cmdsubsys.log

and post that output as well.

[Frédéric GRANAT]

Here it is :

Code: Select all

[root@nagiosxi scripts]# service crond restart
Stopping crond:                                            [  OK  ]
Starting crond:                                            [  OK  ]
[root@nagiosxi scripts]# tail -20 /var/log/messages
Aug 31 08:32:33 nagiosxi nagios: SERVICE ALERT: ws-xa17.cg.ahp;CPU Usage;OK;SOFT;2;OK (Sample Period 360 sec) - Average CPU Utilisation 6.41%
Aug 31 08:32:41 nagiosxi nagios: SERVICE ALERT: ws-xa09.cg.ahp;CPU Usage;OK;SOFT;2;OK (Sample Period 365 sec) - Average CPU Utilisation 3.58%
Aug 31 08:33:38 nagiosxi nagios: SERVICE ALERT: ws-xa03.cg.ahp;Uptime;OK;SOFT;2;OK - System Uptime is  05:29:18 (329min).
Aug 31 08:35:23 nagiosxi nagios: SERVICE ALERT: WS-APP29-PP.cg.ahp;CPU Usage;CRITICAL;SOFT;1;CRITICAL (Sample Period 271 sec) - [Triggered by _AvgCPU>90] - Average CPU Utilisation 99.99%
Aug 31 08:36:00 nagiosxi nagios: SERVICE ALERT: ws-xa02.cg.ahp;CPU Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host 172.16.20.17 might just be really busy, it might not even be running Windows.
Aug 31 08:36:10 nagiosxi nagios: SERVICE ALERT: ws-xa20.cg.ahp;Services;CRITICAL;SOFT;1;CRITICAL - [Triggered by _NumBad>0] - Found 54 Services(s), 53 OK and 1 with problems (17 excluded). 'Spouleur d’impression' (Spooler) is Stopped.
Aug 31 08:36:18 nagiosxi nagios: SERVICE ALERT: ws-xa04.cg.ahp;Uptime;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host 172.16.20.14 might just be really busy, it might not even be running Windows.
Aug 31 08:36:20 nagiosxi nagios: SERVICE ALERT: WS-APP29-PP.cg.ahp;CPU Usage;CRITICAL;SOFT;2;CRITICAL (Sample Period 27 sec) - [Triggered by _AvgCPU>90] - Average CPU Utilisation 99.99%
Aug 31 08:36:57 nagiosxi nagios: SERVICE ALERT: ws-xa04.cg.ahp;Uptime;OK;SOFT;2;OK - System Uptime is  05:35:05 (335min).
Aug 31 08:37:00 nagiosxi nagios: SERVICE ALERT: ws-xa20.cg.ahp;Services;OK;SOFT;2;OK - Found 54 Services(s), 54 OK and 0 with problems (17 excluded).
Aug 31 08:37:08 nagiosxi nagios: SERVICE ALERT: ws-xa02.cg.ahp;CPU Usage;OK;SOFT;2;OK (Sample Period 383 sec) - Average CPU Utilisation 15.96%
Aug 31 08:38:08 nagiosxi nagios: SERVICE ALERT: WS-APP29-PP.cg.ahp;CPU Usage;OK;SOFT;3;OK (Sample Period 104 sec) - Average CPU Utilisation 0.86%
Aug 31 08:40:42 nagiosxi nagios: SERVICE ALERT: ws-xa12.cg.ahp;CPU Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host 172.16.20.21 might just be really busy, it might not even be running Windows.
Aug 31 08:40:54 nagiosxi nagios: SERVICE ALERT: ws-xa12.cg.ahp;CPU Usage;OK;SOFT;2;OK (Sample Period 360 sec) - Average CPU Utilisation 2.22%
Aug 31 08:41:27 nagiosxi nagios: SERVICE ALERT: ws-xa11.cg.ahp;CPU Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host 172.16.20.20 might just be really busy, it might not even be running Windows.
Aug 31 08:42:07 nagiosxi nagios: SERVICE ALERT: ws-xa11.cg.ahp;CPU Usage;OK;SOFT;2;OK (Sample Period 357 sec) - Average CPU Utilisation 15.41%
Aug 31 08:43:42 nagiosxi nagios: SERVICE ALERT: WS-APP29-PP.cg.ahp;CPU Usage;CRITICAL;SOFT;1;CRITICAL (Sample Period 170 sec) - [Triggered by _AvgCPU>90] - Average CPU Utilisation 99.99%
Aug 31 08:43:47 nagiosxi nagios: SERVICE ALERT: ws-xa08.cg.ahp;CPU Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host 172.16.20.16 might just be really busy, it might not even be running Windows.
Aug 31 08:44:27 nagiosxi nagios: SERVICE ALERT: ws-xa08.cg.ahp;CPU Usage;OK;SOFT;2;OK (Sample Period 349 sec) - Average CPU Utilisation 6.17%
Aug 31 08:44:36 nagiosxi nagios: SERVICE ALERT: WS-APP29-PP.cg.ahp;CPU Usage;OK;SOFT;2;OK (Sample Period 54 sec) - Average CPU Utilisation 4.64%
[root@nagiosxi scripts]# tail -20 /var/log/httpd/error_log
[Tue Aug 30 09:34:24 2016] [error] [client 172.16.1.125] PHP Notice:  Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://nagiosxi.cg.ahp/nagiosxi/config/
PHP Warning:  PHP Startup: SourceGuardian: Unable to initialize module
Module compiled with module API=20060613
PHP    compiled with module API=20100525
These options need to match
 in Unknown on line 0
[Tue Aug 30 09:34:53 2016] [error] [client 172.16.1.125] PHP Notice:  Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://nagiosxi.cg.ahp/nagiosxi/includes/components/ccm/xi-index.php
[Tue Aug 30 09:34:58 2016] [error] [client 172.16.1.125] PHP Notice:  Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://nagiosxi.cg.ahp/nagiosxi/includes/components/ccm/xi-index.php
[Tue Aug 30 09:36:24 2016] [error] [client 172.16.1.125] PHP Notice:  Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://nagiosxi.cg.ahp/nagiosxi/includes/components/ccm/?cmd=view&type=host
[Tue Aug 30 09:38:20 2016] [error] [client 172.16.1.125] PHP Notice:  Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://nagiosxi.cg.ahp/nagiosxi/includes/components/ccm/index.php
[Tue Aug 30 09:41:03 2016] [error] [client 172.16.1.125] PHP Notice:  Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://nagiosxi.cg.ahp/nagiosxi/includes/components/ccm/index.php
[Tue Aug 30 09:47:07 2016] [error] [client 172.16.1.125] PHP Notice:  Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://nagiosxi.cg.ahp/nagiosxi/includes/components/ccm/index.php?cmd=modify&type=host&id=582&page=1&returnUrl=index.php%3Fcmd%3Dview%26type%3Dhost%26page%3D1
[Tue Aug 30 09:48:43 2016] [error] [client 172.16.1.125] PHP Notice:  Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://nagiosxi.cg.ahp/nagiosxi/includes/components/ccm/index.php?type=host&page=1
[Tue Aug 30 16:13:08 2016] [error] [client 172.16.6.9] File does not exist: /var/www/html/itassistant
[Tue Aug 30 16:13:08 2016] [error] [client 172.16.6.9] File does not exist: /var/www/html/xmldata
[Wed Aug 31 06:11:03 2016] [error] [client 172.16.6.9] File does not exist: /var/www/html/itassistant
[Wed Aug 31 06:11:03 2016] [error] [client 172.16.6.9] File does not exist: /var/www/html/xmldata
[Wed Aug 31 07:55:34 2016] [error] [client 172.16.100.25] PHP Warning:  ldap_bind(): Unable to bind to server: Can't contact LDAP server in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 714, referer: http://nagiosxi/nagiosxi/login.php?redirect=/nagiosxi/index.php%3f&noauth=1
[Wed Aug 31 07:55:44 2016] [error] [client 172.16.100.25] PHP Notice:  Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://nagiosxi/nagiosxi/login.php?redirect=/nagiosxi/index.php%3f&noauth=1
[Wed Aug 31 07:55:50 2016] [error] [client 172.16.100.25] PHP Notice:  Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://nagiosxi/nagiosxi/index.php?
[root@nagiosxi scripts]# tail -20 /var/log/httpd/ssl_error_log
[Sun Aug 28 04:02:14 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Sun Aug 28 04:02:14 2016] [warn] RSA server certificate CommonName (CN) `localhost.localdomain' does NOT match server name!?
[root@nagiosxi scripts]# tail -20 /usr/local/nagiosxi/var/eventman.log
[root@nagiosxi scripts]# tail -20 /usr/local/nagiosxi/var/cmdsubsys.log

[lmiltchev]

Are you still seeing the red dots in the "System Status" windows?

What is the output of the following commands?

Code: Select all

ps -ef | grep cron
tail -50 /var/log/cron
chage nagios -l
chage apache -l
tail /var/log/mysqld.log
// or "tail /var/log/mariadb/mariadb.log" in case you are using mariadb

[Frédéric GRANAT]

Hi,
Are you still seeing the red dots in the "System Status" windows?
=> Yes (starting from database maintenance)

What is the output of the following commands?
Here it is :

Code: Select all

[root@nagiosxi scripts]# ps -ef | grep cron
root     19695 18580  0 10:01 pts/0    00:00:00 grep cron
root     26384     1  0 Aug31 ?        00:00:00 crond
[root@nagiosxi scripts]# tail -50 /var/log/cron
Sep  1 09:59:02 nagiosxi crond[18077]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Sep  1 09:59:04 nagiosxi crond[18046]: (nagios) MAIL (mailed 153 bytes of output but got status 0x0001 )
Sep  1 09:59:04 nagiosxi crond[18044]: (nagios) MAIL (mailed 143 bytes of output but got status 0x0001 )
Sep  1 09:59:04 nagiosxi crond[18055]: (nagios) MAIL (mailed 145 bytes of output but got status 0x0001 )
Sep  1 09:59:08 nagiosxi crond[18062]: (nagios) MAIL (mailed 147 bytes of output but got status 0x0001 )
Sep  1 09:59:08 nagiosxi crond[18054]: (nagios) MAIL (mailed 145 bytes of output but got status 0x0001 )
Sep  1 09:59:08 nagiosxi crond[18065]: (nagios) MAIL (mailed 143 bytes of output but got status 0x0001 )
Sep  1 09:59:14 nagiosxi crond[18052]: (nagios) MAIL (mailed 153 bytes of output but got status 0x0001 )
Sep  1 09:59:14 nagiosxi crond[18049]: (nagios) MAIL (mailed 135 bytes of output but got status 0x0001 )
Sep  1 10:00:01 nagiosxi crond[18579]: (nobody) CMD (/usr/sbin/vnstat.cron)
Sep  1 10:00:01 nagiosxi crond[18582]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1)
Sep  1 10:00:01 nagiosxi crond[18589]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Sep  1 10:00:01 nagiosxi crond[18591]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1)
Sep  1 10:00:01 nagiosxi crond[18592]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Sep  1 10:00:01 nagiosxi crond[18595]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Sep  1 10:00:01 nagiosxi crond[18598]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Sep  1 10:00:01 nagiosxi crond[18602]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Sep  1 10:00:01 nagiosxi crond[18612]: (root) CMD (/usr/bin/mrtg /etc/mrtg/mrtg.cfg)
Sep  1 10:00:01 nagiosxi crond[18613]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Sep  1 10:00:01 nagiosxi crond[18616]: (root) CMD (/usr/lib/sa/sa1 1 1)
Sep  1 10:00:01 nagiosxi crond[18617]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Sep  1 10:00:01 nagiosxi crond[18618]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Sep  1 10:00:04 nagiosxi crond[18581]: (nagios) MAIL (mailed 145 bytes of output but got status 0x0001 )
Sep  1 10:00:04 nagiosxi crond[18604]: (nagios) MAIL (mailed 147 bytes of output but got status 0x0001 )
Sep  1 10:00:05 nagiosxi crond[18599]: (nagios) MAIL (mailed 145 bytes of output but got status 0x0001 )
Sep  1 10:00:07 nagiosxi crond[18584]: (nagios) MAIL (mailed 143 bytes of output but got status 0x0001 )
Sep  1 10:00:07 nagiosxi crond[18587]: (nagios) MAIL (mailed 153 bytes of output but got status 0x0001 )
Sep  1 10:00:07 nagiosxi crond[18588]: (nagios) MAIL (mailed 135 bytes of output but got status 0x0001 )
Sep  1 10:00:13 nagiosxi crond[18594]: (nagios) MAIL (mailed 153 bytes of output but got status 0x0001 )
Sep  1 10:00:13 nagiosxi crond[18585]: (nagios) MAIL (mailed 143 bytes of output but got status 0x0001 )
Sep  1 10:00:13 nagiosxi crond[18605]: (nagios) MAIL (mailed 143 bytes of output but got status 0x0001 )
Sep  1 10:00:25 nagiosxi crond[18596]: (nagios) MAIL (mailed 145 bytes of output but got status 0x0001 )
Sep  1 10:01:01 nagiosxi crond[19195]: (root) CMD (run-parts /etc/cron.hourly)
Sep  1 10:01:01 nagiosxi crond[19200]: (nagios) CMD (/usr/local/nagiosxi/cron/recurringdowntime.pl > /usr/local/nagiosxi/var/recurringdowntime.log 2>&1)
Sep  1 10:01:01 nagiosxi crond[19204]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Sep  1 10:01:02 nagiosxi crond[19220]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Sep  1 10:01:02 nagiosxi crond[19224]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Sep  1 10:01:02 nagiosxi crond[19226]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Sep  1 10:01:02 nagiosxi crond[19229]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Sep  1 10:01:02 nagiosxi crond[19233]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Sep  1 10:01:02 nagiosxi crond[19235]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Sep  1 10:01:02 nagiosxi crond[19240]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Sep  1 10:01:05 nagiosxi crond[19210]: (nagios) MAIL (mailed 145 bytes of output but got status 0x0001 )
Sep  1 10:01:05 nagiosxi crond[19208]: (nagios) MAIL (mailed 153 bytes of output but got status 0x0001 )
Sep  1 10:01:05 nagiosxi crond[19232]: (nagios) MAIL (mailed 143 bytes of output but got status 0x0001 )
Sep  1 10:01:08 nagiosxi crond[19223]: (nagios) MAIL (mailed 145 bytes of output but got status 0x0001 )
Sep  1 10:01:08 nagiosxi crond[19228]: (nagios) MAIL (mailed 147 bytes of output but got status 0x0001 )
Sep  1 10:01:08 nagiosxi crond[19207]: (nagios) MAIL (mailed 135 bytes of output but got status 0x0001 )
Sep  1 10:01:14 nagiosxi crond[19205]: (nagios) MAIL (mailed 153 bytes of output but got status 0x0001 )
Sep  1 10:01:14 nagiosxi crond[19202]: (nagios) MAIL (mailed 143 bytes of output but got status 0x0001 )
[root@nagiosxi scripts]# chage nagios -l
Last password change                                    : Mar 14, 2013
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7
[root@nagiosxi scripts]# chage apache -l
Last password change                                    : Jan 08, 2010
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7
[root@nagiosxi scripts]# tail /var/log/mysqld.log
160830  8:47:04 InnoDB: Completed initialization of buffer pool
160830  8:47:04 InnoDB: highest supported file format is Barracuda.
160830  8:47:04  InnoDB: Waiting for the background threads to start
160830  8:47:05 InnoDB: 1.1.8 started; log sequence number 1588791
160830  8:47:05 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
160830  8:47:05 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
160830  8:47:05 [Note] Server socket created on IP: '0.0.0.0'.
160830  8:47:05 [Note] Event Scheduler: Loaded 0 events
160830  8:47:05 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.28'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Atomicorp
[root@nagiosxi scripts]#

[lmiltchev]

It seems like something is causing crond to "hang". Do you have any "custom" entries? What is the output of the following command?

Code: Select all

cat /etc/cron.d/nagiosxi

Try stopping crond:

Code: Select all

service crond stop
killall crond

then start it:

Code: Select all

service crond start

Make sure it is running. You should be able to see the php scripts in the "ps" output - example:

Code: Select all

[root@localhost ~]# ps -ef | grep cron | grep -v grep
root      1499     1  0 Aug26 ?        00:00:08 crond
nagios   26012 26008  0 09:31 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios   26013 26011  0 09:31 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios   26016 26012  0 09:31 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
nagios   26017 26007  0 09:31 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios   26018 26009  0 09:31 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios   26019 26013  0 09:31 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios   26020 26010  0 09:31 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios   26024 26017  0 09:31 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios   26025 26018  0 09:31 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios   26026 26020  0 09:31 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php