Large amount of service checks

[CFT6Server]

We are configuring hosts with large amount of service checks. I noticed that once it reaches a certain amount of checks, I am starting to see odd behaviors. For example, when I add checks up to a certain point, the services doesn't show up at all for that host even though they show up in CCM under services. Also when Applying Configuration, it fails the first time, and second time takes a long time but applies successfully. But again all services for that host do not show up and we are left with a blank services page for the host.

We have tested this by disabling all the services checks for the host and then add the services in a bit at a time. It seems to start behaving oddly after enabling about 500+ service checks. (We list by 500 per page and there are 4 pages). Are we hitting some limit in the background somewhere? The minute I deactivate the most recent service checks, the services comes back.

[jdalrymple]

Probably start with

Code: Select all

tail /var/log/httpd/error_log

After trying to load the detail page that's failing.

[CFT6Server]

So I was enabling them slowly and eventually they were all enabled without a hitch. Looks like I can just modify too much at once.

Here's the output anyways.

Code: Select all

[Wed Aug 19 13:02:38 2015] [error] [client 10.242.13.10] PHP Notice:  Undefined variable: sync_table_status in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 196, referer: https://10.242.102.100/nagiosxi/includes/components/ccm/index.php?cmd=view&type=service&page=8
[Wed Aug 19 13:02:39 2015] [error] [client 10.242.13.10] PHP Notice:  Undefined variable: ac_needed_js_inject in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 176, referer: https://10.242.102.100/nagiosxi/includes/components/ccm/index.php?cmd=view&type=service&page=7
[Wed Aug 19 13:02:39 2015] [error] [client 10.242.13.10] PHP Notice:  Undefined variable: sync_table_status in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 196, referer: https://10.242.102.100/nagiosxi/includes/components/ccm/index.php?cmd=view&type=service&page=7
[Wed Aug 19 13:02:59 2015] [error] [client ::1] PHP Notice:  Undefined index: language in /usr/local/nagiosxi/html/includes/components/ccm/includes/common_functions.inc.php on line 710
[Wed Aug 19 13:02:59 2015] [error] [client ::1] PHP Notice:  Undefined index: language in /usr/local/nagiosxi/html/includes/components/ccm/includes/common_functions.inc.php on line 711
[Wed Aug 19 13:04:07 2015] [error] [client 10.242.13.10] PHP Notice:  Undefined variable: ac_needed_js_inject in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 176, referer: https://10.242.102.100/nagiosxi/includes/components/ccm/xi-index.php
[Wed Aug 19 13:04:07 2015] [error] [client 10.242.13.10] PHP Notice:  Undefined variable: sync_table_status in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 196, referer: https://10.242.102.100/nagiosxi/includes/components/ccm/xi-index.php
[Wed Aug 19 13:04:10 2015] [error] [client 10.242.13.10] PHP Notice:  Undefined variable: ac_needed_js_inject in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 176, referer: https://10.242.102.100/nagiosxi/includes/components/ccm/?cmd=view&type=service
[Wed Aug 19 13:04:10 2015] [error] [client 10.242.13.10] PHP Notice:  Undefined variable: sync_table_status in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 196, referer: https://10.242.102.100/nagiosxi/includes/components/ccm/?cmd=view&type=service

The first 3 lines comes up often, but the ones afterwards only start when I add just over 800 service checks.

I also viewed the cmdsubsys.log and looks like it applied successfully without errors.

[jdalrymple]

If you look back awayas in the error_log do you see any php timeouts or limits exceeded? I wonder if apply config was timing out midstream of updating because of a php.ini setting?