Schedule a forced immediate check not working

[rajasegar]

For a test, can you disable the gearman worker that is running on the XI server so the XI server will do the local checks instead of the gearman worker?
This may increase the performance of the XI server.

We added gearman because local server had issues handling the scheduling.
Not taking any chances on the production box sending out invalid alerts due to this.

[jolson]

Try navigating to Service Details -> (select a service) -> Advanced, and then select the link "See this service in Nagios Core." See if you can access that url and submit the command correctly/quickly. Core may output more verbose errors.

[rajasegar]

Try navigating to Service Details -> (select a service) -> Advanced, and then select the link "See this service in Nagios Core." See if you can access that url and submit the command correctly/quickly. Core may output more verbose errors.

No error messages.

Does anyone have any clue? This happens so often. We have to restart the Gearman services to solve this problem.
Just restarting Nagios wont solve it.

[abrist]

This is very odd. Gearman should not effect the command.cgi at all, as when you submit a command it is done through a an ajax call to the command cgi. You can even see this behavior by watch the requests from your browser console/network log. Do you see any errors in the apache logs at /var/log/httpd/error_log?

[rajasegar]

This is very odd. Gearman should not effect the command.cgi at all, as when you submit a command it is done through a an ajax call to the command cgi. You can even see this behavior by watch the requests from your browser console/network log. Do you see any errors in the apache logs at /var/log/httpd/error_log?

See attached log. See a bunch of errors in it.
This happened yesterday 06/04/2014 around 12:30 - 12:55pm

Code: Select all

[Mon Apr 06 12:22:42 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined variable: ac_needed_js_inject in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 176, referer: http://10.17.19.235/nagiosxi/includes/components/ccm/xi-index.php
[Mon Apr 06 12:22:42 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined variable: sync_table_status in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 196, referer: http://10.17.19.235/nagiosxi/includes/components/ccm/xi-index.php
[Mon Apr 06 12:23:39 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined variable: ac_needed_js_inject in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 176, referer: http://10.17.19.235/nagiosxi/includes/components/ccm/?cmd=view&type=service
[Mon Apr 06 12:23:39 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined variable: sync_table_status in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 196, referer: http://10.17.19.235/nagiosxi/includes/components/ccm/?cmd=view&type=service
[Mon Apr 06 12:23:43 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined index: template_name in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/common_settings.php on line 53, referer: http://10.17.19.235/nagiosxi/includes/components/ccm/index.php
[Mon Apr 06 12:24:09 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined index: template_name in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/common_settings.php on line 53, referer: http://10.17.19.235/nagiosxi/includes/components/ccm/index.php?type=service&page=1
[Mon Apr 06 12:25:01 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined variable: ac_needed_js_inject in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 176, referer: http://10.17.19.235/nagiosxi/includes/components/ccm/index.php?type=service&page=1
[Mon Apr 06 12:25:01 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined variable: sync_table_status in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 196, referer: http://10.17.19.235/nagiosxi/includes/components/ccm/index.php?type=service&page=1
[Mon Apr 06 12:25:32 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined index: template_name in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/common_settings.php on line 53, referer: http://10.17.19.235/nagiosxi/includes/components/ccm/index.php
[Mon Apr 06 12:25:47 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined index: template_name in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/common_settings.php on line 53, referer: http://10.17.19.235/nagiosxi/includes/components/ccm/index.php?type=service&page=1
[Mon Apr 06 12:26:25 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined variable: ac_needed_js_inject in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 176, referer: http://10.17.19.235/nagiosxi/includes/components/ccm/index.php?type=service&page=1
[Mon Apr 06 12:26:25 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined variable: sync_table_status in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 196, referer: http://10.17.19.235/nagiosxi/includes/components/ccm/index.php?type=service&page=1
[Mon Apr 06 12:39:59 2015] [error] [client ::1] PHP Notice:  Undefined index: language in /usr/local/nagiosxi/html/includes/components/ccm/includes/common_functions.inc.php on line 710
[Mon Apr 06 12:39:59 2015] [error] [client ::1] PHP Notice:  Undefined index: language in /usr/local/nagiosxi/html/includes/components/ccm/includes/common_functions.inc.php on line 711
[Mon Apr 06 12:40:15 2015] [error] [client 10.17.38.5] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/opscreen/merlin.php on line 142, referer: http://10.17.19.235/nagiosxi/includes/components/opscreen/opscreen.php
[Mon Apr 06 12:40:15 2015] [error] [client 10.17.38.5] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/opscreen/merlin.php on line 144, referer: http://10.17.19.235/nagiosxi/includes/components/opscreen/opscreen.php
[Mon Apr 06 12:40:15 2015] [error] [client 10.17.38.5] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/opscreen/merlin.php on line 185, referer: http://10.17.19.235/nagiosxi/includes/components/opscreen/opscreen.php
[Mon Apr 06 12:40:15 2015] [error] [client 10.17.38.5] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/opscreen/merlin.php on line 187, referer: http://10.17.19.235/nagiosxi/includes/components/opscreen/opscreen.php
[Mon Apr 06 12:40:23 2015] [error] [client 10.17.44.60] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/opscreen/merlin.php on line 142, referer: http://10.17.19.235/nagiosxi/includes/components/opscreen/opscreen.php
[Mon Apr 06 12:40:23 2015] [error] [client 10.17.44.60] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/opscreen/merlin.php on line 144, referer: http://10.17.19.235/nagiosxi/includes/components/opscreen/opscreen.php
[Mon Apr 06 12:40:23 2015] [error] [client 10.17.44.60] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/opscreen/merlin.php on line 185, referer: http://10.17.19.235/nagiosxi/includes/components/opscreen/opscreen.php
[Mon Apr 06 12:40:23 2015] [error] [client 10.17.44.60] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/opscreen/merlin.php on line 187, referer: http://10.17.19.235/nagiosxi/includes/components/opscreen/opscreen.php
[Mon Apr 06 12:40:24 2015] [error] [client 10.17.44.34] PHP Warning:  arsort() expects parameter 1 to be array, null given in /usr/local/nagiosxi/html/includes/components/latestalerts/latestalerts.inc.php on line 517, referer: http://10.17.19.235/nagiosxi//includes/page-home-main.php?&=
[Mon Apr 06 12:40:24 2015] [error] [client 10.17.44.34] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/latestalerts/latestalerts.inc.php on line 518, referer: http://10.17.19.235/nagiosxi//includes/page-home-main.php?&=
[Mon Apr 06 12:40:24 2015] [error] [client 10.17.44.34] PHP Notice:  Undefined variable: c in /usr/local/nagiosxi/html/includes/components/latestalerts/latestalerts.inc.php on line 521, referer: http://10.17.19.235/nagiosxi//includes/page-home-main.php?&=
[Mon Apr 06 12:40:26 2015] [error] [client 10.17.38.5] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/opscreen/merlin.php on line 142, referer: http://10.17.19.235/nagiosxi/includes/components/opscreen/opscreen.php
[Mon Apr 06 12:40:26 2015] [error] [client 10.17.38.5] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/opscreen/merlin.php on line 144, referer: http://10.17.19.235/nagiosxi/includes/components/opscreen/opscreen.php
[Mon Apr 06 12:40:26 2015] [error] [client 10.17.38.5] PHP Warning:  Division by zero in /usr/local/nagiosxi/html/includes/components/opscreen/merlin.php on line 185, referer: http://10.17.19.235/nagiosxi/includes/components/opscreen/opscreen.php

error.zip

[abrist]

If you can submit commands from core, but not XI, you may have an issue with the cmdsubsys cron. What is the output of:

Code: Select all

ps -aef | grep cron
service crond status
cat /etc/cron.d/nagiosxi
tail -25 /var/log/cron
ls -ld /home/nagios
grep nag /etc/group

[rajasegar]

If you can submit commands from core, but not XI, you may have an issue with the cmdsubsys cron. What is the output of:

Code: Select all

ps -aef | grep cron
service crond status
cat /etc/cron.d/nagiosxi
tail -25 /var/log/cron
ls -ld /home/nagios
grep nag /etc/group

It was the same problem from core.
Here is the information requested.

Code: Select all

[root@nagiosprodxi1 ~]# ps -aef | grep cron
root      2535     1  0 Mar31 ?        00:00:47 crond
nagios   11532 11513  0 08:29 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios   11533 11519  0 08:29 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios   11535 11532  1 08:29 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios   11536 11533  1 08:29 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios   11538 11514  0 08:29 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios   11539 11516  0 08:29 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios   11541 11515  0 08:29 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios   11545 11541  1 08:29 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios   11546 11538  1 08:29 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
nagios   11562 11539  1 08:29 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios   23085  7042  2 08:29 ?        00:00:00 /usr/local/nagios/libexec/check_nrpe -u -H 172.29.37.89 -t 60 -c rs_check_service -a cron
root     23092 22252  0 08:29 pts/0    00:00:00 grep cron
[root@nagiosprodxi1 ~]# service crond status
crond (pid  2535) is running...
[root@nagiosprodxi1 ~]# cat /etc/cron.d/nagiosxi
# /etc/cron.d/nagiosxi: crontab fragment for nagiosxi

# Backup MySQL & PostgreSQL Databases
0   7 * * * root   /root/scripts/automysqlbackup
0   8 * * * root   /root/scripts/autopostgresqlbackup

*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
01  * * * * nagios /usr/local/nagiosxi/cron/recurringdowntime.pl > /usr/local/nagiosxi/var/recurringdowntime.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1

[root@nagiosprodxi1 ~]# tail -25 /var/log/cron
Apr  8 08:26:01 nagiosprodxi1 CROND[10511]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Apr  8 08:27:01 nagiosprodxi1 CROND[26685]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Apr  8 08:27:01 nagiosprodxi1 CROND[26686]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Apr  8 08:27:01 nagiosprodxi1 CROND[26688]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Apr  8 08:27:01 nagiosprodxi1 CROND[26694]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Apr  8 08:27:01 nagiosprodxi1 CROND[26695]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Apr  8 08:27:01 nagiosprodxi1 CROND[26693]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Apr  8 08:27:01 nagiosprodxi1 CROND[26699]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Apr  8 08:27:01 nagiosprodxi1 CROND[26698]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Apr  8 08:28:01 nagiosprodxi1 CROND[28272]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Apr  8 08:28:01 nagiosprodxi1 CROND[28273]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Apr  8 08:28:01 nagiosprodxi1 CROND[28274]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Apr  8 08:28:01 nagiosprodxi1 CROND[28271]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Apr  8 08:28:01 nagiosprodxi1 CROND[28276]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Apr  8 08:28:01 nagiosprodxi1 CROND[28277]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Apr  8 08:28:01 nagiosprodxi1 CROND[28278]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Apr  8 08:28:01 nagiosprodxi1 CROND[28275]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Apr  8 08:29:01 nagiosprodxi1 CROND[11531]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Apr  8 08:29:01 nagiosprodxi1 CROND[11533]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Apr  8 08:29:01 nagiosprodxi1 CROND[11532]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Apr  8 08:29:01 nagiosprodxi1 CROND[11538]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Apr  8 08:29:01 nagiosprodxi1 CROND[11537]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Apr  8 08:29:01 nagiosprodxi1 CROND[11540]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Apr  8 08:29:01 nagiosprodxi1 CROND[11541]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Apr  8 08:29:01 nagiosprodxi1 CROND[11539]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
[root@nagiosprodxi1 ~]# ls -ld /home/nagios
drwx------. 30 nagios users 4096 Apr  8 08:14 /home/nagios
[root@nagiosprodxi1 ~]# grep nag /etc/group
nagios:x:501:nagios,apache,my02390,mycp1z2k
nagcmd:x:502:nagios,apache,my02390,mycp1z2k
[root@nagiosprodxi1 ~]#

[abrist]

Can you post the config for the User's associated contact? Go to the ccm --> contacts --> click the disk icon next to the user's contact and post the relevant config block for the contact.

[rajasegar]

Can you post the config for the User's associated contact? Go to the ccm --> contacts --> click the disk icon next to the user's contact and post the relevant config block for the contact.

The person doing the schedule immediate checks is Admin users in XI.
There is no user in the contact with the same user name.
It was working fine since day 1. Just 1 month back it became very slow and sometimes shows failed to execute.

Code: Select all

define contact {
	contact_name                  		INFRA_MON_Raja_Segar
	alias                         		Raja Segar
	email                         		[email protected]
	pager                         		+60122345678
	use                           		generic-contact
	}	

[tgriep]

Can you run the following in a shell when you Schedule an immediate check with the user account you are having problems with and post the output back here?

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log