Custom Python Check Returns "UNKNOWN" exit state only in UI

[[email protected]]

Custom Python Check Returns "UNKNOWN" exit state only in UI

Problem Case
I have a custom python script that is being used to gather results from the Broadcom WSS Status Page API.

The script functions as expected when being executed from the command line and when being sanity checked with the "Run Check Command" option within CCM.

RunTestCommand-CCM.PNG

The check however fails to return the expected state for the telemetry evaluation (as seen in CLI and Test) when being executed as normal within the UI, always an UNKNOWN.

XIUI-CheckResult.PNG

When being executed from the command line or from the run test command option I get complete execution as seen in my debug logs

[2021-04-13T14:06:21] pid=40101; level="DEBUG"; name="check_wss"; function="<module>"; line="140"; message="meta-host="https://70h1x50k6t63.statuspage.io/" meta-endpoint="component" meta-critical="major" meta-perfdata="True" meta-debug="True"";
[2021-04-13T14:06:21] pid=40101; level="DEBUG"; name="check_wss"; function="wssComponentStatus"; line="73"; message="request="<PreparedRequest [GET]>" status_code="200" result-json="{'page': {'id': '70h1x50k6t63', 'name': 'Symantec Web Security Service', 'url': 'https://wss.status.broadcom.com', 'time_zone': 'Etc/UTC', 'updated_at': '2021-04-11T23:23:37.674Z'}, 'components': [{'id': 'pkp9wcnd6ch5', 'name': 'Amsterdam (GNLAM1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T16:01:12.705Z', 'updated_at': '2021-04-11T22:30:37.778Z', 'position': 1, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'fwqqrbst39c6', 'name': 'Data Center Selection\xa0 Services', 'status': 'operational', 'created_at': '2019-02-20T21:25:56.689Z', 'updated_at': '2020-12-09T16:56:41.715Z', 'position': 1, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'xr7gk3px5jf7', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'mnjlckw7gs09', 'name': 'API Services', 'status': 'operational', 'created_at': '2019-02-20T21:25:08.684Z', 'updated_at': '2021-04-09T22:00:31.355Z', 'position': 1, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'dqsl1b9wn0sm', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'yjkbtyf4wmrl', 'name': 'Buenos Aires (GARBA1)', 'status': 'under_maintenance', 'created_at': '2020-04-07T04:40:47.010Z', 'updated_at': '2021-04-11T22:30:37.733Z', 'position': 1, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'cq77r33hflrk', 'name': 'Auckland (GNZAU1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:52:56.604Z', 'updated_at': '2021-04-11T22:30:37.757Z', 'position': 1, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'v4h3zsqs9jp9', 'name': 'Beijing (PEK1)', 'status': 'under_maintenance', 'created_at': '2018-12-22T01:40:27.856Z', 'updated_at': '2021-04-11T22:30:37.821Z', 'position': 2, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'cmt2mg2t4pys', 'name': 'Data Centers - Americas', 'status': 'under_maintenance', 'created_at': '2018-04-17T15:24:04.201Z', 'updated_at': '2018-10-05T14:49:59.425Z', 'position': 2, 'description': None, 'showcase': False, 'start_date': None, 'group_id': None, 'page_id': '70h1x50k6t63', 'group': True, 'only_show_if_degraded': False, 'components': ['yjkbtyf4wmrl', 'w2lvsb6d12z5', 'lyc8zm1vkfsc', 'zqkcvv3y1c43', 'm7prq2nlfcqq', '427xrj2vcpd1', '64h92ndsgltx', '6zt77y0t47cy', 'pj8tjlwg94gm', '038l2k1tsyy5', 'zgy3h27sjbq9', 'lkmmy1xgfcx9', '8v7pkwq2kb1k', 'g45c895njndl']}, {'id': 'w2lvsb6d12z5', 'name': 'Des Moines (GUSDM1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T16:03:03.471Z', 'updated_at': '2021-04-11T22:30:37.799Z', 'position': 2, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '9rjlt42fyk9l', 'name': 'Mobile Device Enrollment Service', 'status': 'operational', 'created_at': '2019-02-20T21:26:16.518Z', 'updated_at': '2020-10-03T19:02:41.655Z', 'position': 2, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'xr7gk3px5jf7', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '1hcblp7vkz7c', 'name': 'Cloud Portal', 'status': 'operational', 'created_at': '2018-10-05T14:48:36.642Z', 'updated_at': '2021-04-09T22:00:31.374Z', 'position': 2, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'dqsl1b9wn0sm', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'm8g5g9rx652q', 'name': 'Bucharest (GROBU1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:55:56.732Z', 'updated_at': '2021-04-11T22:30:37.890Z', 'position': 3, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'cgdrzz6x5khj', 'name': 'Global Intelligence Network', 'status': 'operational', 'created_at': '2019-02-20T21:26:34.438Z', 'updated_at': '2020-02-21T20:00:42.051Z', 'position': 3, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'xr7gk3px5jf7', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'x8vxpx8b70r1', 'name': 'Data Centers - Europe, Middle East & Africa', 'status': 'under_maintenance', 'created_at': '2018-04-17T15:28:53.220Z', 'updated_at': '2018-10-05T14:50:38.325Z', 'position': 3, 'description': None, 'showcase': False, 'start_date': None, 'group_id': None, 'page_id': '70h1x50k6t63', 'group': True, 'only_show_if_degraded': False, 'components': ['pkp9wcnd6ch5', 'm8g5g9rx652q', 'pvgy937m92jy', '63yncnyf8p1v', '45jhtrr1dfzd', 'kpc5jhy6vmdz', 'c08y79s2fgdz', '0vzz6yc765z7', 'rznpzynzb0fd', '4tctk9m9fg58', 'hfy1hb3c0dyp', '7rg4g62pzdcn', 'z5y70t89zfb1', 'v31vpv2cwmk9', 'n8485qtp1hkj', 'kgkn3hsypkhx', '5k1xdw6ggm3n', '3d64cvzzl14v', 'sg9fpzfc65cd', 'z9hhqp1vd823']}, {'id': '5b9qfwjyvgjn', 'name': 'Log Transfer Services', 'status': 'operational', 'created_at': '2019-02-20T21:24:56.601Z', 'updated_at': '2021-04-09T22:00:31.402Z', 'position': 3, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'dqsl1b9wn0sm', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'lyc8zm1vkfsc', 'name': 'Des Moines (GUSDM2)', 'status': 'under_maintenance', 'created_at': '2020-04-16T16:02:18.685Z', 'updated_at': '2021-04-11T22:30:37.868Z', 'position': 3, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'bzkf131ytjn3', 'name': 'Mumbai (GINMU2)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:58:33.535Z', 'updated_at': '2021-04-11T22:30:37.847Z', 'position': 3, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'zqkcvv3y1c43', 'name': 'Des Moines (GUSDM3)', 'status': 'under_maintenance', 'created_at': '2020-04-15T04:19:44.589Z', 'updated_at': '2021-04-11T22:30:37.910Z', 'position': 4, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'ckbq0wph249w', 'name': 'Anti-Malware Scanning', 'status': 'operational', 'created_at': '2019-02-20T21:26:53.607Z', 'updated_at': '2020-02-21T20:00:42.837Z', 'position': 4, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'xr7gk3px5jf7', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '91k1ys6nf3bq', 'name': 'Data Centers - Asia Pacific', 'status': 'under_maintenance', 'created_at': '2018-04-20T07:52:38.630Z', 'updated_at': '2018-10-05T14:50:20.358Z', 'position': 4, 'description': None, 'showcase': False, 'start_date': None, 'group_id': None, 'page_id': '70h1x50k6t63', 'group': True, 'only_show_if_degraded': False, 'components': ['cq77r33hflrk', 'v4h3zsqs9jp9', 'bzkf131ytjn3', '5stfb974v7b5', '6c57pg8s2pds', 'wnpvz6d6gzhq', 'pt92krtx8373', 'hzrcdjnql9yb', 'q59y3yqd0hy0', 'vz0831741pvt', '2n0ggh88t04h', 'ymrx2crlq9b1', 'lcqdgm79f6zq']}, {'id': 'f5fqd316q6v9', 'name': 'Management UI', 'status': 'operational', 'created_at': '2019-02-20T21:23:44.892Z', 'updated_at': '2021-04-09T22:00:31.449Z', 'position': 4, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'dqsl1b9wn0sm', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '5stfb974v7b5', 'name': 'Hong Kong (GCNHK1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T16:00:12.558Z', 'updated_at': '2021-04-11T22:30:37.995Z', 'position': 5, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'dqsl1b9wn0sm', 'name': 'Portal & Reporting', 'status': 'operational', 'created_at': '2019-02-20T21:21:39.653Z', 'updated_at': '2019-02-20T21:21:39.653Z', 'position': 5, 'description': None, 'showcase': False, 'start_date': None, 'group_id': None, 'page_id': '70h1x50k6t63', 'group': True, 'only_show_if_degraded': False, 'components': ['mnjlckw7gs09', '1hcblp7vkz7c', '5b9qfwjyvgjn', 'f5fqd316q6v9', 'd3c5pg227z74', '3gr49d3gw3nw', '6zrlf3gtzgnb', 'fywmpztq50w1']}, {'id': '5qwkbm0gdcds', 'name': 'Malware Analysis Sandbox', 'status': 'operational', 'created_at': '2019-02-20T21:27:10.468Z', 'updated_at': '2020-07-10T12:00:54.954Z', 'position': 5, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'xr7gk3px5jf7', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'd3c5pg227z74', 'name': 'PAC File Management System (PFMS)', 'status': 'operational', 'created_at': '2019-03-27T15:32:56.159Z', 'updated_at': '2020-11-08T00:00:29.796Z', 'position': 5, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'dqsl1b9wn0sm', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'm7prq2nlfcqq', 'name': 'Las Vegas (GUSLV1)', 'status': 'under_maintenance', 'created_at': '2020-11-06T10:46:50.640Z', 'updated_at': '2021-04-11T22:30:37.945Z', 'position': 5, 'description': None, 'showcase': True, 'start_date': '2020-11-06', 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'pvgy937m92jy', 'name': 'Copenhagen (GDKCP1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:55:39.144Z', 'updated_at': '2021-04-11T22:30:37.975Z', 'position': 5, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'xr7gk3px5jf7', 'name': 'Global Services', 'status': 'operational', 'created_at': '2019-02-20T21:25:56.679Z', 'updated_at': '2019-02-20T21:25:56.679Z', 'position': 6, 'description': None, 'showcase': False, 'start_date': None, 'group_id': None, 'page_id': '70h1x50k6t63', 'group': True, 'only_show_if_degraded': False, 'components': ['fwqqrbst39c6', '9rjlt42fyk9l', 'cgdrzz6x5khj', 'ckbq0wph249w', '5qwkbm0gdcds', '6dq8xkgf9dbh', 'cbypxqjf2dfk', '7bs739wm76rs', 'tl38ngwk2rlr', '5xgcbmrwh344']}, {'id': '427xrj2vcpd1', 'name': 'Mexico City (GMXMC1)', 'status': 'under_maintenance', 'created_at': '2020-04-14T23:56:02.829Z', 'updated_at': '2021-04-11T22:30:38.018Z', 'position': 6, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '3gr49d3gw3nw', 'name': 'Reporting Services and Portal Dashboards', 'status': 'operational', 'created_at': '2019-02-20T21:24:17.717Z', 'updated_at': '2021-03-05T02:00:34.449Z', 'position': 6, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'dqsl1b9wn0sm', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '6dq8xkgf9dbh', 'name': 'DLP Scan Interconnect', 'status': 'operational', 'created_at': '2019-02-20T21:27:35.814Z', 'updated_at': '2021-02-10T00:00:29.552Z', 'position': 6, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'xr7gk3px5jf7', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '6zrlf3gtzgnb', 'name': 'Symantec Secure Login', 'status': 'operational', 'created_at': '2019-02-20T21:22:51.755Z', 'updated_at': '2021-03-05T02:00:34.474Z', 'position': 7, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'dqsl1b9wn0sm', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '64h92ndsgltx', 'name': 'Columbia (GUSCO1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:58:08.237Z', 'updated_at': '2021-04-11T22:30:38.059Z', 'position': 7, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '63yncnyf8p1v', 'name': 'Dubai (GAEDX1)', 'status': 'under_maintenance', 'created_at': '2020-04-14T02:43:21.435Z', 'updated_at': '2021-04-11T22:30:38.039Z', 'position': 7, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'cbypxqjf2dfk', 'name': 'CASB Gateway Interconnect', 'status': 'operational', 'created_at': '2019-02-20T21:27:51.344Z', 'updated_at': '2021-03-20T00:00:30.442Z', 'position': 7, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'xr7gk3px5jf7', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'fywmpztq50w1', 'name': 'Software Download Licensing', 'status': 'operational', 'created_at': '2019-02-20T21:24:31.624Z', 'updated_at': '2021-02-26T23:00:32.174Z', 'position': 8, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'dqsl1b9wn0sm', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '6zt77y0t47cy', 'name': 'Montreal (GCAMO1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:55:23.907Z', 'updated_at': '2021-04-11T22:30:38.101Z', 'position': 8, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '7bs739wm76rs', 'name': 'Web-Isolation Interconnect', 'status': 'operational', 'created_at': '2019-02-20T21:28:06.742Z', 'updated_at': '2021-04-11T11:00:31.483Z', 'position': 8, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'xr7gk3px5jf7', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '6c57pg8s2pds', 'name': 'Mumbai (GINMU1)', 'status': 'under_maintenance', 'created_at': '2020-04-15T12:27:54.746Z', 'updated_at': '2021-04-11T22:30:38.078Z', 'position': 8, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'tl38ngwk2rlr', 'name': 'WebPulse', 'status': 'operational', 'created_at': '2019-08-15T01:50:04.940Z', 'updated_at': '2020-02-21T20:00:45.515Z', 'position': 9, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'xr7gk3px5jf7', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'wnpvz6d6gzhq', 'name': 'Osaka (GJPOS1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:56:57.309Z', 'updated_at': '2021-04-11T22:30:38.121Z', 'position': 9, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'pj8tjlwg94gm', 'name': 'Washington, DC (GUSAS2)', 'status': 'under_maintenance', 'created_at': '2020-04-16T16:01:43.343Z', 'updated_at': '2021-04-11T22:30:38.138Z', 'position': 9, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '45jhtrr1dfzd', 'name': 'Dublin (GIEDU1)', 'status': 'under_maintenance', 'created_at': '2020-04-20T11:19:05.515Z', 'updated_at': '2021-04-11T22:30:38.175Z', 'position': 10, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '038l2k1tsyy5', 'name': 'Los Angeles (GUSLA1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:54:04.657Z', 'updated_at': '2021-04-11T22:30:38.156Z', 'position': 10, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '5xgcbmrwh344', 'name': 'System for Cross-domain Identity Management', 'status': 'operational', 'created_at': '2020-12-07T13:11:11.813Z', 'updated_at': '2020-12-12T04:00:31.349Z', 'position': 10, 'description': None, 'showcase': True, 'start_date': '2020-12-07', 'group_id': 'xr7gk3px5jf7', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'kpc5jhy6vmdz', 'name': 'Frankfurt (GDEFR1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T16:02:05.472Z', 'updated_at': '2021-04-11T22:30:38.236Z', 'position': 11, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'zgy3h27sjbq9', 'name': 'Sao Paulo (GBRSP1)', 'status': 'under_maintenance', 'created_at': '2020-04-13T16:59:42.469Z', 'updated_at': '2021-04-11T22:30:38.215Z', 'position': 11, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'pt92krtx8373', 'name': 'Seoul (GKRSE1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:47:52.525Z', 'updated_at': '2021-04-11T22:30:38.195Z', 'position': 11, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'lkmmy1xgfcx9', 'name': 'Portland (GUSPO1)', 'status': 'under_maintenance', 'created_at': '2020-04-11T04:41:43.974Z', 'updated_at': '2021-04-11T22:30:38.255Z', 'position': 12, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'c08y79s2fgdz', 'name': 'Helsinki (GFIHE1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:53:40.425Z', 'updated_at': '2021-04-11T22:30:38.276Z', 'position': 13, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '8v7pkwq2kb1k', 'name': 'Toronto (GCATO1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:48:44.182Z', 'updated_at': '2021-04-11T22:30:38.320Z', 'position': 13, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'hzrcdjnql9yb', 'name': 'Shanghai (SHA1)', 'status': 'under_maintenance', 'created_at': '2018-04-20T07:53:57.937Z', 'updated_at': '2021-04-11T22:30:38.298Z', 'position': 13, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'g45c895njndl', 'name': 'Washington, DC (GUSAS1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T16:00:38.400Z', 'updated_at': '2021-04-11T22:30:38.385Z', 'position': 14, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'cmt2mg2t4pys', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '0vzz6yc765z7', 'name': 'Johannesburg (JNB2)', 'status': 'under_maintenance', 'created_at': '2018-04-17T16:13:33.012Z', 'updated_at': '2021-04-11T22:30:38.340Z', 'position': 14, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'q59y3yqd0hy0', 'name': 'Singapore (GSGRS1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T16:00:24.053Z', 'updated_at': '2021-04-11T22:30:38.364Z', 'position': 14, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'rznpzynzb0fd', 'name': 'London (GGBLR)', 'status': 'under_maintenance', 'created_at': '2020-04-15T04:20:36.003Z', 'updated_at': '2021-04-11T22:30:38.404Z', 'position': 15, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'vz0831741pvt', 'name': 'Sydney (GAUSY1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:56:34.972Z', 'updated_at': '2021-04-11T22:30:38.431Z', 'position': 16, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '4tctk9m9fg58', 'name': 'Madrid (GESMA1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:56:09.281Z', 'updated_at': '2021-04-11T22:30:38.456Z', 'position': 17, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '2n0ggh88t04h', 'name': 'Taipei (GTWTA1)', 'status': 'under_maintenance', 'created_at': '2020-04-14T01:19:20.771Z', 'updated_at': '2021-04-11T22:30:38.479Z', 'position': 18, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'hfy1hb3c0dyp', 'name': 'Middlesex (GGBLO1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T16:02:49.158Z', 'updated_at': '2021-04-11T22:30:38.500Z', 'position': 19, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'ymrx2crlq9b1', 'name': 'Tokyo (GJPTK)', 'status': 'under_maintenance', 'created_at': '2020-04-16T16:00:51.251Z', 'updated_at': '2021-04-11T22:30:38.523Z', 'position': 20, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '7rg4g62pzdcn', 'name': 'Milan (GITMI1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:59:55.580Z', 'updated_at': '2021-04-11T22:30:38.543Z', 'position': 21, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'lcqdgm79f6zq', 'name': 'Wellington (GNZWL)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:45:41.645Z', 'updated_at': '2021-04-11T22:30:38.566Z', 'position': 22, 'description': None, 'showcase': True, 'start_date': None, 'group_id': '91k1ys6nf3bq', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'z5y70t89zfb1', 'name': 'Munich (GDEMU)', 'status': 'under_maintenance', 'created_at': '2020-04-16T16:02:34.195Z', 'updated_at': '2021-04-11T22:30:38.585Z', 'position': 23, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'v31vpv2cwmk9', 'name': 'Oslo (GNOOS1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:55:00.397Z', 'updated_at': '2021-04-11T22:30:38.604Z', 'position': 25, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'n8485qtp1hkj', 'name': 'Paris (GFRPA1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:56:22.145Z', 'updated_at': '2021-04-11T22:30:38.627Z', 'position': 27, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'kgkn3hsypkhx', 'name': 'Stockholm (GSESK1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T16:01:30.663Z', 'updated_at': '2021-04-11T22:30:38.644Z', 'position': 30, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '5k1xdw6ggm3n', 'name': 'Tel Aviv (GILTA1)', 'status': 'under_maintenance', 'created_at': '2020-04-10T17:26:08.861Z', 'updated_at': '2021-04-11T22:30:38.662Z', 'position': 31, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': '3d64cvzzl14v', 'name': 'Turin (GITTU1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:49:04.362Z', 'updated_at': '2021-04-11T22:30:38.683Z', 'position': 33, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'sg9fpzfc65cd', 'name': 'Zurich (GCHZU1)', 'status': 'under_maintenance', 'created_at': '2020-04-16T15:54:44.544Z', 'updated_at': '2021-04-11T22:30:38.703Z', 'position': 34, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}, {'id': 'z9hhqp1vd823', 'name': 'Dubai (GAEDX2)', 'status': 'under_maintenance', 'created_at': '2020-04-23T05:36:38.078Z', 'updated_at': '2021-04-11T22:30:38.725Z', 'position': 36, 'description': None, 'showcase': True, 'start_date': None, 'group_id': 'x8vxpx8b70r1', 'page_id': '70h1x50k6t63', 'group': False, 'only_show_if_degraded': False}]}"";
[2021-04-13T14:06:21] pid=40101; level="DEBUG"; name="check_wss"; function="<module>"; line="304"; message="output-msg="OK: (20 of 70) components for report Operational. (50 of 70) components report Under Maintenance. (Data Center Selection  Services=operational,API Services=operational,Mobile Device Enrollment Service=operational,Cloud Portal=operational,Global Intelligence Network=operational,Log Transfer Services=operational,Anti-Malware Scanning=operational,Management UI=operational,Portal & Reporting=operational,Malware Analysis Sandbox=operational,PAC File Management System (PFMS)=operational,Global Services=operational,Reporting Services and Portal Dashboards=operational,DLP Scan Interconnect=operational,Symantec Secure Login=operational,CASB Gateway Interconnect=operational,Software Download Licensing=operational,Web-Isolation Interconnect=operational,WebPulse=operational,System for Cross-domain Identity Management=operational)| component_total=70; operational=20; degraded_performance=0; partial_outage=0; major_outage=0; maintenance=50;"";

When being executed as normal from the UI however I only get 1 line (with PID) and it never finishes.

[2021-04-13T14:11:48] pid=53966; level="DEBUG"; name="check_wss"; function="<module>"; line="140"; message="meta-host="https://70h1x50k6t63.statuspage.io" meta-endpoint="unresolved" meta-critical="critical" meta-perfdata="True" meta-debug="True"";
[2021-04-13T14:16:47] pid=67325; level="DEBUG"; name="check_wss"; function="<module>"; line="140"; message="meta-host="https://70h1x50k6t63.statuspage.io" meta-endpoint="unresolved" meta-critical="critical" meta-perfdata="True" meta-debug="True"";

ownership is correct and can be executed by the nagios user
script is marked executable
Works CLI
Works Run Check Command
Fails in UI...

Any help greatly appreciated, I'm stumped at this point.
--SN

[[email protected]]

UPDATE

I went a little deeper on the logging and I was finally able to get an error to work with. This error only occurs when the check is run when

• Triggered via the normal execution for the service check
  Executed with the "force an immediate check" option

From the command line execution:

Code: Select all

./check_wss.py --host 'https://70h1x50k6t63.statuspage.io/' -e 'unresolved' -c 'warning' -p -d

OK: There are (0) unresolved warning severity incidents for Symantec Web Security Service| incident_total=0; warning=0;

The logs show no issues with running the script or connecting to the API endpoint URL.

tail -f /usr/local/nagios/var/check_wss.log

Code: Select all

[2021-04-14T14:01:35] pid=62816; level="DEBUG"; name="check_wss"; function="<module>"; line="170"; message="uir="<Response [200]>"";
[2021-04-14T14:01:35] pid=62816; level="DEBUG"; name="check_wss"; function="<module>"; line="175"; message="uir_json="{'page': {'id': '70h1x50k6t63', 'name': 'Symantec Web Security Service', 'url': 'https://wss.status.broadcom.com', 'time_zone': 'Etc/UTC', 'updated_at': '2021-04-11T23:23:37.674Z'}, 'incidents': []}"";
[2021-04-14T14:01:35] pid=62816; level="DEBUG"; name="check_wss"; function="<module>"; line="181"; message="pageName="Symantec Web Security Service"";
[2021-04-14T14:01:35] pid=62816; level="DEBUG"; name="check_wss"; function="<module>"; line="186"; message="pageID="70h1x50k6t63"";
[2021-04-14T14:01:35] pid=62816; level="DEBUG"; name="check_wss"; function="<module>"; line="192"; message="incidents="[]"";
[2021-04-14T14:01:35] pid=62816; level="DEBUG"; name="check_wss"; function="<module>"; line="206"; message="incCount="0" critCount="0"";
[2021-04-14T14:01:35] pid=62816; level="DEBUG"; name="check_wss"; function="<module>"; line="223"; message="output-msg="OK: There are (0) unresolved warning severity incidents for Symantec Web Security Service| incident_total=0; warning=0;"";

When executed through the UI or on the expected timed interval with XI (and only when) I receive the error message for the API call.

Code: Select all

Error="('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))"

tail -f /usr/local/nagios/var/check_wss.log

Code: Select all

[2021-04-14T14:11:40] pid=89961; level="DEBUG"; name="check_wss"; function="<module>"; line="155"; message="meta-host="https://70h1x50k6t63.statuspage.io" meta-endpoint="unresolved" meta-critical="critical" meta-perfdata="True" meta-debug="True"";
[2021-04-14T14:11:40] pid=89961; level="ERROR"; name="check_wss"; function="broadcomUnsresolvedIncidents"; line="68"; message="Error="('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))"";
[2021-04-14T14:11:40] pid=89961; level="ERROR"; name="check_wss"; function="<module>"; line="230"; message="UNKNOWN: Unable to retrieve the unresolved incident count for Symantec Web Security Service.";

Here is the function that is being called.

Code: Select all

#GET LIST OF UNRESOLVED INCIDENTS FROM THE API
def broadcomUnsresolvedIncidents():
    headers = {
    "Content-Type": "Application/Json",
    "Accept-Encoding": "Application/Json"
    }

    #SLEEP TO TRY TO FIX UNRESOLVED
    time.sleep(0.01)
    
    try:
        r=requests.get(meta.host+"/api/v2/incidents/unresolved.json", headers=headers)
        r.raise_for_status()
        if meta.debug:
            logMsg = ('request="{}" status_code="{}" result-json="{}"'.format(r.request,r.status_code,r.json()))
            logger.debug(logMsg)
    except requests.exceptions.RequestException as e:
        logMsg = ('Error="{}"'.format(e))
        logger.error(logMsg)
    finally:
        return r

I can see no reason why this would properly execute via the command line and "Run Check Command" option in CCM and Fail in the XI interface.

[dchurch]

There's a number of reasons it could fail, but I'm thinking there's some missing part of your environment.

Do you need to go thru a proxy server to get the check to work?

What's the output of the following command (from the command line):

Code: Select all

env |grep -i 'http\|proxy'

[[email protected]]

There's a number of reasons it could fail, but I'm thinking there's some missing part of your environment.

Do you need to go thru a proxy server to get the check to work?

What's the output of the following command (from the command line):

Code: Select all

env |grep -i 'http\|proxy'

No proxies and can reproduce on multiple systems

[[email protected]]

There's a number of reasons it could fail, but I'm thinking there's some missing part of your environment.

Do you need to go thru a proxy server to get the check to work?

What's the output of the following command (from the command line):

Code: Select all

env |grep -i 'http\|proxy'

I can reproduce on multiple systems one with and one without proxy give the same error

[[email protected]]

I'm more confused now.

When developing for Nagios there has always been the checklist for expected operations;
1. The goal of you script, return 1 line of text (143 max char for correct interface display) followed by the exit state for nagios (2, 1, 0, 3)
2. The script can be executed on command line by the "nagios" user and functions as expected
3. Once configured in Nagios, the script can be executed via the "Run Check Command" button in CCM

If all these are done then NagiosXI should have no issues, well until now... The only place I can get the script to fail is in NagiosXI when executed outside of CCM.

My NagiosXI Command

Code: Select all

define command {
    command_name    check_wss
    command_line    python3 /usr/local/nagios/libexec/check_wss.py --host "$ARG1$" -e "$ARG2$" -c "$ARG3$" $ARG4$
}

Command as defined in NagiosXI

commandDef.PNG

NagiosXI Service Definition

ServiceDef.PNG

Nagios CCM "Run Check Command" Output

RunTestCommand-CCM.PNG

NagiosXI UI Execution

UI-Execute.PNG

Any thoughts other than proxy?
--SN

[dchurch]

I've noticed that the command is executed NOT using the python3 interpreter when executed via the "Run Check Command" dialog.

Are you sure the one that starts with "python3" is the only command in your system that's defining the check_wss.py script? What's the bangline of that script? If it's executing using /usr/bin/python, it'll have different behavior than if executed using /usr/bin/python3 since /usr/bin/python is Python 2.x.



Can you please post the output from the following commands?

Code: Select all

pip3 show requests certifi urllib3
sudo -u nagios pip3 show requests certifi urllib3

Also, it may help if I had the entire source code for the check_wss.py file, so if you could PM that to me that would be helpful.

[[email protected]]

BANG
#!/usr/bin/python3

I have tried running with python3 and without.

I have tried with $USER1$/check_wss.py and the direct path shown

pip3 show requests certifi urllib3

Code: Select all

Name: requests
Version: 2.24.0
Summary: Python HTTP for Humans.
Home-page: https://requests.readthedocs.io
Author: Kenneth Reitz
Author-email: [email protected]
License: Apache 2.0
Location: /usr/local/lib/python3.6/site-packages
Requires: idna, certifi, urllib3, chardet
---
Name: certifi
Version: 2020.6.20
Summary: Python package for providing Mozilla's CA Bundle.
Home-page: https://certifiio.readthedocs.io/en/latest/
Author: Kenneth Reitz
Author-email: [email protected]
License: MPL-2.0
Location: /usr/local/lib/python3.6/site-packages
Requires:
---
Name: urllib3
Version: 1.25.11
Summary: HTTP library with thread-safe connection pooling, file post, and more.
Home-page: https://urllib3.readthedocs.io/
Author: Andrey Petrov
Author-email: [email protected]
License: MIT
Location: /usr/local/lib/python3.6/site-packages
Requires:

sudo -u nagios pip3 show requests certifi urllib3

Code: Select all

Name: requests
Version: 2.24.0
Summary: Python HTTP for Humans.
Home-page: https://requests.readthedocs.io
Author: Kenneth Reitz
Author-email: [email protected]
License: Apache 2.0
Location: /usr/local/lib/python3.6/site-packages
Requires: idna, chardet, certifi, urllib3
---
Name: certifi
Version: 2020.6.20
Summary: Python package for providing Mozilla's CA Bundle.
Home-page: https://certifiio.readthedocs.io/en/latest/
Author: Kenneth Reitz
Author-email: [email protected]
License: MPL-2.0
Location: /usr/local/lib/python3.6/site-packages
Requires:
---
Name: urllib3
Version: 1.25.11
Summary: HTTP library with thread-safe connection pooling, file post, and more.
Home-page: https://urllib3.readthedocs.io/
Author: Andrey Petrov
Author-email: [email protected]
License: MIT
Location: /usr/local/lib/python3.6/site-packages
Requires:

[vtrac]

Hi,
Hope you are having a good day ...

Can you try as suggested in the URL below? .... install "requests[security]" and also put your "requests.get" call in the "while" loop. Of course, we don't want a while loop that runs forever, maybe a count=5:
https://stackoverflow.com/questions/418 ... et-by-peer

Also, could you try adding "timeout" to your "request.get" command as well:
Example:

Code: Select all

requests.get(url, timeout=2.50)

Also, on your Nagios XI command prompt, please run this:

Code: Select all

nmap -Pn -p 443,80 70h1x50k6t63.statuspage.io

Just want to see that your XI is able to communicate with both port 80 and 443.


Best Regards,
Vinh

[vtrac]

Also, can you please upload your whole script?


Regards,
Vinh