Page 1 of 1

Nagios Not performing Service check and false alerts

Posted: Wed Feb 04, 2015 2:44 pm
by sarfarosh
Hello team,

We are facing quite strange issue at one of our client place,around 5024 hosts are been added with 18586 service, all ubuntu server monitored through SNMP, load distribution is been done through mod_gearman following http://assets.nagios.com/downloads/nagi ... ios_XI.pdf.

Checks for hosts seems to be working fine, but with services nagios is not performing checks,services for Host which are down are showing as OK,and Services are showing down for hosts which are UP and matrix fetch-able through snmpwalk command line from both nagiosxi & mod_gearman_worker servers, we have tried cross verifying all setting in nagiosxi, nagios.cfg parameters everything looks fine for us, also check_time defined are also fine, below are some of the error we are seeing in

apache error log

Code: Select all

[Wed Feb 04 16:55:09 2015] [error] [client 10.171.4.35] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 341, referer: http://10.187.212.31/nagiosxi//includes/page-home-main.php?&=
[Wed Feb 04 16:55:09 2015] [error] [client 10.171.4.35] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 353, referer: http://10.187.212.31/nagiosxi//includes/page-home-main.php?&=
[Wed Feb 04 16:55:09 2015] [error] [client 10.171.4.35] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 354, referer: http://10.187.212.31/nagiosxi//includes/page-home-main.php?&=
[Wed Feb 04 16:55:09 2015] [error] [client 10.171.4.35] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 355, referer: http://10.187.212.31/nagiosxi//includes/page-home-main.php?&=
[Wed Feb 04 16:55:09 2015] [error] [client 10.171.4.35] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 356, referer: http://10.187.212.31/nagiosxi//includes/page-home-main.php?&=
[Wed Feb 04 16:55:37 2015] [error] [client 10.171.4.35] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:55:37 2015] [error] [client 10.171.4.35] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:56:07 2015] [error] [client 10.171.4.35] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:56:07 2015] [error] [client 10.171.4.35] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:56:37 2015] [error] [client 10.171.4.35] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:56:37 2015] [error] [client 10.171.4.35] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:57:07 2015] [error] [client 10.171.4.35] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:57:07 2015] [error] [client 10.171.4.35] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:57:37 2015] [error] [client 10.171.4.35] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:57:37 2015] [error] [client 10.171.4.35] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:58:07 2015] [error] [client 10.171.4.35] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:58:07 2015] [error] [client 10.171.4.35] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:58:37 2015] [error] [client 10.171.4.35] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:58:37 2015] [error] [client 10.171.4.35] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:59:07 2015] [error] [client 10.171.4.35] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 16:59:07 2015] [error] [client 10.171.4.35] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-servicestatus.inc.php on line 425, referer: http://10.187.212.31/nagiosxi/includes/components/xicore/status.php?show=services
[Wed Feb 04 17:40:55 2015] [error] [client ::1] PHP Warning:  mysql_pconnect(): Too many connections in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383
[Wed Feb 04 18:31:57 2015] [error] [client ::1] PHP Notice:  Undefined index: ccm_language in /usr/local/nagiosxi/html/includes/components/ccm/includes/common_functions.inc.php on line 710
[Wed Feb 04 18:31:57 2015] [error] [client ::1] PHP Notice:  Undefined index: language in /usr/local/nagiosxi/html/includes/components/ccm/includes/common_functions.inc.php on line 710
[Wed Feb 04 18:32:04 2015] [error] [client ::1] PHP Notice:  Undefined index: ccm_language in /usr/local/nagiosxi/html/includes/components/ccm/includes/common_functions.inc.php on line 710
[Wed Feb 04 18:32:04 2015] [error] [client ::1] PHP Notice:  Undefined index: language in /usr/local/nagiosxi/html/includes/components/ccm/includes/common_functions.inc.php on line 710


Perfdata.log

Code: Select all

2015-02-04 18:14:11 [20429] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1423052542.perfdata.service-PID-20429 deleted
2015-02-04 18:14:11 [20429] [0] *** process_perfdata.pl terminated on signal ALRM
2015-02-04 18:14:11 [20425] [0] *** TIMEOUT: Timeout after 60 Sec. ****
2015-02-04 18:14:11 [20425] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-02-04 18:14:11 [20425] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-02-04 18:14:11 [20425] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1423052530.perfdata.service-PID-20425 deleted
2015-02-04 18:14:11 [20425] [0] *** process_perfdata.pl terminated on signal ALRM
2015-02-04 18:15:12 [20644] [0] *** TIMEOUT: Timeout after 60 Sec. ****
2015-02-04 18:15:12 [20644] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-02-04 18:15:12 [20644] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-02-04 18:15:12 [20644] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1423052572.perfdata.service-PID-20644 deleted
2015-02-04 18:15:12 [20644] [0] *** process_perfdata.pl terminated on signal ALRM
2015-02-04 18:15:13 [20642] [0] *** TIMEOUT: Timeout after 60 Sec. ****
2015-02-04 18:15:13 [20642] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-02-04 18:15:13 [20642] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-02-04 18:15:13 [20642] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1423052558.perfdata.service-PID-20642 deleted
2015-02-04 18:15:13 [20642] [0] *** process_perfdata.pl terminated on signal ALRM
2015-02-04 18:15:16 [20649] [0] *** TIMEOUT: Timeout after 60 Sec. ****
2015-02-04 18:15:16 [20649] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-02-04 18:15:16 [20649] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-02-04 18:15:16 [20649] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1423052587.perfdata.service-PID-20649 deleted
2015-02-04 18:15:16 [20649] [0] *** process_perfdata.pl terminated on signal ALRM
2015-02-04 18:18:33 [21391] [0] *** TIMEOUT: Timeout after 60 Sec. ****
2015-02-04 18:18:33 [21391] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-02-04 18:18:33 [21391] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-02-04 18:18:33 [21391] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1423052767.perfdata.service-PID-21391 deleted
2015-02-04 18:18:33 [21391] [0] *** process_perfdata.pl terminated on signal ALRM
2015-02-04 18:18:35 [21387] [0] *** TIMEOUT: Timeout after 60 Sec. ****
2015-02-04 18:18:35 [21387] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-02-04 18:18:35 [21387] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-02-04 18:18:35 [21387] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1423052752.perfdata.service-PID-21387 deleted
2015-02-04 18:18:35 [21387] [0] *** process_perfdata.pl terminated on signal ALRM
2015-02-04 18:28:04 [16482] [0] *** TIMEOUT: Timeout after 60 Sec. ****
2015-02-04 18:28:04 [16482] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-02-04 18:28:04 [16482] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-02-04 18:28:04 [16482] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1423052828.perfdata.service-PID-16482 deleted
2015-02-04 18:28:04 [16482] [0] *** process_perfdata.pl terminated on signal ALRM
2015-02-04 18:28:05 [16488] [0] *** TIMEOUT: Timeout after 60 Sec. ****
2015-02-04 18:28:05 [16488] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-02-04 18:28:05 [16488] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-02-04 18:28:05 [16488] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1423052857.perfdata.service-PID-16488 deleted
2015-02-04 18:28:05 [16488] [0] *** process_perfdata.pl terminated on signal ALRM
2015-02-04 18:28:07 [16486] [0] *** TIMEOUT: Timeout after 60 Sec. ****
2015-02-04 18:28:07 [16486] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-02-04 18:28:07 [16486] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-02-04 18:28:07 [16486] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1423052842.perfdata.service-PID-16486 deleted
2015-02-04 18:28:07 [16486] [0] *** process_perfdata.pl terminated on signal ALRM
2015-02-04 18:29:07 [18046] [0] *** TIMEOUT: Timeout after 60 Sec. ****
2015-02-04 18:29:07 [18046] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-02-04 18:29:07 [18046] [0] *** TIMEOUT: Please check your process_perfdata.cfg
2015-02-04 18:29:07 [18046] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1423052872.perfdata.service-PID-18046 deleted
2015-02-04 18:29:07 [18046] [0] *** process_perfdata.pl terminated on signal ALRM

Code: Select all

Feb  4 18:20:24 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:24 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:25 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:25 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:26 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:26 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:27 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:27 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:28 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:28 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:29 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:33 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:34 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:34 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:35 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:39 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:40 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:40 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:41 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:41 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:42 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:42 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:43 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:43 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:44 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:47 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:48 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:50 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:51 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:55 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:56 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:20:56 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:20:57 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:00 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:01 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:02 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:03 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:04 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:05 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:06 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:07 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:07 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:08 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:09 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:10 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:12 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:13 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:13 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:14 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:17 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:18 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:22 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:23 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:29 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:30 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:36 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:37 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:37 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:38 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:38 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:39 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:42 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:43 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:44 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:45 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:46 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:47 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:48 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:50 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:50 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:51 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:51 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:52 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:52 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:53 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:53 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:54 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:56 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:57 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:58 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:21:59 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:21:59 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:22:00 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:22:00 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:22:01 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:22:01 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:22:02 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:22:06 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:22:07 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:22:07 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:22:08 cctnsdcems ndo2db: Message sent to queue.
Feb  4 18:22:08 cctnsdcems ndo2db: Warning: queue send error, retrying...
Feb  4 18:22:09 cctnsdcems ndo2db: Message sent to queue.
attached is nagios.cfg & service.cfg for 1 of the services which is not changing state

also mysql Tables are also getting corrupted even now & then, beklow is mysqld.log extract

Code: Select all

150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
150204 16:59:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed


extract from timeperiod.cfg

Code: Select all

###############################################################################
#
# Timeperiod configuration file
#
# Created by: Nagios Core Config Manager 2.3.1
# Date:	      2015-02-03 22:33:47
# Version:    Nagios 3.x config file
#
# --- DO NOT EDIT THIS FILE BY HAND --- 
# Nagios CCM will overwrite all manual settings during the next update if you 
# would like to edit files manually, place them in the 'static' directory or 
# import your configs into the CCM by placing them in the 'import' directory.
#
###############################################################################

define timeperiod {
	timeperiod_name               		24x7
	alias                         		24 Hours A Day, 7 Days A Week
	sunday                        		00:00-24:00
	monday                        		00:00-24:00
	tuesday                       		00:00-24:00
	wednesday                     		00:00-24:00
	thursday                      		00:00-24:00
	friday                        		00:00-24:00
	saturday                      		00:00-24:00
	}	

define timeperiod {
	timeperiod_name               		24x7_sans_holidays
	alias                         		24x7 Sans Holidays
	use                           		us-holidays
	sunday                        		00:00-24:00
	monday                        		00:00-24:00
	tuesday                       		00:00-24:00
	wednesday                     		00:00-24:00
	thursday                      		00:00-24:00
	friday                        		00:00-24:00
	saturday                      		00:00-24:00
	}	

define timeperiod {
	timeperiod_name               		cctns-user1_notification_times
	alias                         		Notification Times for cctns-user1
	monday                        		00:00-24:00
	tuesday                       		00:00-24:00
	wednesday                     		00:00-24:00
	thursday                      		00:00-24:00
	friday                        		00:00-24:00
	saturday                      		00:00-24:00
	sunday                        		00:00-24:00
	}	

define timeperiod {
	timeperiod_name               		cctnsnoc_notification_times
	alias                         		Notification Times for cctnsnoc
	monday                        		00:00-24:00
	tuesday                       		00:00-24:00
	wednesday                     		00:00-24:00
	thursday                      		00:00-24:00
	friday                        		00:00-24:00
	saturday                      		00:00-24:00
	sunday                        		00:00-24:00
	}	

define timeperiod {
	timeperiod_name               		dcnoc_notification_times
	alias                         		Notification Times for dcnoc
	monday                        		00:00-24:00
	tuesday                       		00:00-24:00
	wednesday                     		00:00-24:00
	thursday                      		00:00-24:00
	friday                        		00:00-24:00
	saturday                      		00:00-24:00
	sunday                        		00:00-24:00
	}	

define timeperiod {
	timeperiod_name               		nagiosadmin_notification_times
	alias                         		Notification Times for nagiosadmin
	tuesday                       		00:00-24:00
	monday                        		00:00-24:00
	wednesday                     		00:00-24:00
	thursday                      		00:00-24:00
	saturday                      		00:00-24:00
	sunday                        		00:00-24:00
	friday                        		00:00-24:00
	}	

define timeperiod {
	timeperiod_name               		none
	alias                         		No Time Is A Good Time
	}	

define timeperiod {
	timeperiod_name               		test_notification_times
	alias                         		Notification Times for test
	monday                        		00:00-24:00
	tuesday                       		00:00-24:00
	wednesday                     		00:00-24:00
	thursday                      		00:00-24:00
	friday                        		00:00-24:00
	saturday                      		00:00-24:00
	sunday                        		00:00-24:00
	}	

define timeperiod {
	timeperiod_name               		us-holidays
	alias                         		U.S. Holidays
	name                          		us-holidays
	january 1                     		00:00-00:00
	monday 1 september            		00:00-00:00
	july 4                        		00:00-00:00
	thursday -1 november          		00:00-00:00
	december 25                   		00:00-00:00
	}	

define timeperiod {
	timeperiod_name               		workhours
	alias                         		Normal Work Hours
	monday                        		09:00-17:00
	tuesday                       		09:00-17:00
	wednesday                     		09:00-17:00
	thursday                      		09:00-17:00
	friday                        		09:00-17:00
	}	

define timeperiod {
	timeperiod_name               		xi_timeperiod_24x7
	alias                         		24x7
	name                          		xi_timeperiod_24x7
	saturday                      		00:00-24:00
	friday                        		00:00-24:00
	thursday                      		00:00-24:00
	wednesday                     		00:00-24:00
	tuesday                       		00:00-24:00
	monday                        		00:00-24:00
	sunday                        		00:00-24:00
	}	

###############################################################################
#
# Timeperiod configuration file
#
# END OF FILE
#
###############################################################################
please help me to hit a right button to resolve it

Re: Nagios Not performing Service check and false alerts

Posted: Wed Feb 04, 2015 4:39 pm
by cmerchant
You should clean up the database errors:
150204 16:59:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_objects' is marked as crashed and last (automatic?) repair failed
Refer to this document:

http://assets.nagios.com/downloads/nagi ... tabase.pdf

Re: Nagios Not performing Service check and false alerts

Posted: Thu Feb 05, 2015 2:16 am
by sarfarosh
I have already tried repairing it, it didn't helped so had to restore mysql backup of last week which had cleared this error as if now, but still services are not changing state even after "Schedule a forced immediate check" which are actually down & showing UP in Nagios

Re: Nagios Not performing Service check and false alerts

Posted: Thu Feb 05, 2015 9:48 am
by scottwilkerson
Lets try repairing the table a different way, run the following from the command line

Code: Select all

echo "REPAIR TABLE nagios_objects;"|mysql -pnagiosxi nagios