Nagios XI host and service detail screens blank

[murdock]

Hi Nagios,

Sometime today and for no obvious reason clicking on any host or service in the "Home" tab has started displaying blank screens.

I've rewritten configs and restarted, no change.

A cursory check of logs, KBs, and forum articles have revealed nothing. I have a profile.zip and screencaps ready for a DM.

I do see suspicious activity in ssl_error_log, but I think it is unrelated. Specifically, certain users are making erroneous API calls but it does not seem to be impacting system load.

Rob * U of Illinois

[gsmith]

Hi Rob,

Please send me the profile.zip and screencaps in a PM, and
I'll take a look.

Thanks!

[murdock]

The forum software is behaving very oddly: it only allows me to attach one file at a time, and one message is stuck in the "outbox".

I sent 3 DMs, were you able to receive them?

Rob

[murdock]

I have sent 3 DMs twice (6 total) today; of the first 3, 1 got stuck in my outbox, and the other 2 got sent to me. I just re-sent the 3 messages, and those 3 are all in my Outbox now -- I do not recall the forum software working like this previously.

Anyway, please let me know if you received the last 3 messages -- or if I should send them another way. If you got them twice, I apologize.

Rob

[murdock]

Can someone from Nagios confirm that the profile.zip and 2 screencaps I sent yesterday (twice) were received?

Rob

[murdock]

Having not heard anything back, and given that next week is an extended holiday for some, I'd like to escalate this case to a ticket -- I'll follow the appropriate steps.

Rob

[gsmith]

Hey Rob,

I received the System Profile and screen captures. I am taking a look now.

If you opened a ticket already please reply to this thread with the ticket number and I will close
this out and continue from the ticket.

Thanks!

[murdock]

Hi, ticket 477713 was still open from last week, where we upgraded XI and fixed AD. Earlier today in that ticket I asked if this issue (where host and service detail screens are blank) is associated with last weeks work, or if it should be handled under a new, separate ticket -- please let me know.

I did remove the debug line Sean mentioned; and we have not run out of disk space [due to excessive logging].

My hope is that we can return our production instance to service before what will be a long holiday next week (many people -- myself included -- will be out for the entire week).

Having done some further digging, it looks like basic Core functions are mostly working, although I have scattered reports of people thinking they should have received notifications but didn't. Passive monitoring appears to be working as well. But we can't see any host or service detail in the GUI; and I can't figure out with my very basic web developer -type skills (nothing jumping out at me in the Apache logs, poking around in Firefox debugger, etc.).

Rob

[gsmith]

Hi Rob,

From looking at your System Profile I am seeing a ton of timeouts:

Code: Select all

00005: [1637007925] SERVICE ALERT: localhost;Course Explorer-DB Connections;CRITICAL;HARD;1;(Service check timed out after 66.43 seconds)
00009: [1637007925] SERVICE ALERT: odosfile.odos.illinois.edu;Drive C: Disk Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host 128.174.45.48 might just be really busy, it might not even be running Windows.
00016: [1637007925] SERVICE ALERT: s_0197_002_aces;Memory Usage;CRITICAL;SOFT;1;(Service check timed out after 66.49 seconds)
00028: [1637007925] SERVICE ALERT: api.rokwire.illinois.edu;DNS Resolution;CRITICAL;SOFT;1;CRITICAL - Plugin timed out while executing system call
00055: [1637007925] SERVICE ALERT: uie-print.extension.illinois.edu;Ethernet Bandwidth - Inbound;CRITICAL;SOFT;1;(Service check timed out after 66.43 seconds)
00071: [1637007925] SERVICE ALERT: urbmunki3.admin.uillinois.edu;Disk Usage on /dev/mapper/main_vg-storage_lv;CRITICAL;HARD;1;(Service check timed out after 66.67 seconds)
00083: [1637007925] SERVICE ALERT: worksets.hathitrust.org;Disk Usage on /dev/sda1;CRITICAL;SOFT;1;(Service check timed out after 66.93 seconds)
00107: [1637007925] SERVICE ALERT: ADMIN-ZRDS1;Disk Usage on D:/;CRITICAL;SOFT;1;(Service check timed out after 66.83 seconds)
00115: [1637007925] SERVICE ALERT: ADMIN-ZWEB3B;Memory Usage;CRITICAL;SOFT;1;(Service check timed out after 68.04 seconds)
00124: [1637007925] SERVICE ALERT: cctv_SNIPE;Memory Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host snipe.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00125: [1637007925] SERVICE ALERT: cctv_COOT;Drive C: Disk Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host coot.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00126: [1637007925] SERVICE ALERT: cic-it-db13-wmi-sql;SQLServer;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host cic-it-db13.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00128: [1637007925] SERVICE ALERT: cctv_NIGHTJAR;Memory Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host nightjar.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00129: [1637007925] SERVICE ALERT: cctv_WREN;CPU Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host 172.21.150.12 might just be really busy, it might not even be running Windows.
00135: [1637007925] SERVICE ALERT: splunk-urbana-uf-7;Memory Usage;CRITICAL;SOFT;1;(Service check timed out after 64.90 seconds)
00141: [1637007925] SERVICE ALERT: vpn4g-4.gw.illinois.edu;Combined Uplink Bandwidth;CRITICAL;HARD;5;(Service check timed out after 65.00 seconds)
00147: [1637007925] SERVICE ALERT: ADMIN-ZDAT3;Service status for: MSSQL_ADMIN;CRITICAL;SOFT;1;(Service check timed out after 65.10 seconds)
00153: [1637007925] SERVICE ALERT: ADMIN-ZRDS1;Service status for: Schedule;CRITICAL;SOFT;1;(Service check timed out after 65.78 seconds)
00159: [1637007925] SERVICE ALERT: cctv_COOT;Drive D: Disk Usage;CRITICAL;SOFT;1;(Service check timed out after 65.37 seconds)
00165: [1637007925] SERVICE ALERT: cctv_HERON;Drive K: Disk Usage;CRITICAL;SOFT;1;(Service check timed out after 65.61 seconds)
00176: [1637007925] SERVICE ALERT: cctv_MAGPIE;Page File Usage;CRITICAL;SOFT;1;(Service check timed out after 65.53 seconds)
00182: [1637007925] SERVICE ALERT: cctv_NIGHTJAR;Page File Usage;CRITICAL;SOFT;1;(Service check timed out after 65.54 seconds)
00184: [1637007925] SERVICE ALERT: cic-it-w14-wmi;Page File Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host cic-it-w14.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00187: [1637007925] SERVICE ALERT: cic-it-f12-wmi;Page File Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host cic-it-f12.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00194: [1637007925] SERVICE ALERT: cctv_VIREO;Drive C: Disk Usage;CRITICAL;SOFT;1;(Service check timed out after 65.33 seconds)
00200: [1637007925] SERVICE ALERT: odosfile.odos.illinois.edu;CloudBerry Backup Service;CRITICAL;SOFT;1;(Service check timed out after 65.33 seconds)
00206: [1637007925] SERVICE ALERT: FAA-WWW-SEDAC;Disk Usage on E:/;CRITICAL;SOFT;1;(Service check timed out after 64.75 seconds)
00212: [1637007925] SERVICE ALERT: core1-1.gw.uiuc.edu;UIC2 Response;CRITICAL;SOFT;1;(Service check timed out after 64.42 seconds)
00218: [1637007925] SERVICE ALERT: ACES-REMOTE-01;Memory Usage;CRITICAL;SOFT;1;(Service check timed out after 64.25 seconds)
00229: [1637007925] SERVICE ALERT: ACF022-32;CPU Usage;CRITICAL;HARD;1;(Service check timed out after 64.06 seconds)
00231: [1637007925] SERVICE ALERT: cctv_TROGON;Drive C: Disk Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host trogon.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00232: [1637007925] SERVICE ALERT: cic-it-f12-wmi;Drive F: Disk Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host cic-it-f12.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00233: [1637007925] SERVICE ALERT: cic-it-w14-wmi;CPU Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host cic-it-w14.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00234: [1637007925] SERVICE ALERT: cctv_PETREL;Drive C: Disk Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host petrel.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00235: [1637007925] SERVICE ALERT: cctv_TANAGER;CPU Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host Tanager.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00241: [1637007925] SERVICE ALERT: localhost;Course Explorer-Memory-RDS;CRITICAL;HARD;1;(Service check timed out after 60.37 seconds)
00247: [1637007925] SERVICE ALERT: libeasysch19.library.illinois.edu;Ethernet0 Bandwidth - Inbound;CRITICAL;SOFT;1;(Service check timed out after 61.06 seconds)
00254: [1637007925] SERVICE ALERT: localhost;Service Status - ndo2db;CRITICAL;HARD;1;(Service check timed out after 63.84 seconds)
00262: [1637007925] SERVICE ALERT: s_0197_001_aces;Ethernet Bandwidth - Inbound;CRITICAL;SOFT;1;(Service check timed out after 65.01 seconds)
00270: [1637007925] SERVICE ALERT: worksets.hathitrust.org;Disk Usage on /dev/mapper/htrcvirtuoso--vg-root;CRITICAL;HARD;5;(Service check timed out after 66.89 seconds)
00276: [1637007925] SERVICE ALERT: xnatdev.beckman.illinois.edu;ens192 Bandwidth - Inbound;CRITICAL;SOFT;1;(Service check timed out after 67.19 seconds)
00282: [1637007925] SERVICE ALERT: ADMIN-XDAT3;Memory Usage;CRITICAL;SOFT;1;(Service check timed out after 67.66 seconds)
00288: [1637007925] SERVICE ALERT: cctv_TOWHEE;CPU Usage;CRITICAL;SOFT;1;(Service check timed out after 67.51 seconds)
00295: [1637007925] SERVICE ALERT: ACF022-00;CPU Usage;CRITICAL;SOFT;1;(Service check timed out after 67.42 seconds)
00297: [1637007925] SERVICE ALERT: cctv_SWIFT;CPU Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host Swift.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00306: [1637007925] SERVICE ALERT: ACF022-36;Service Status: KeyAccess;CRITICAL;HARD;1;(Service check timed out after 67.48 seconds)
00316: [1637007925] SERVICE ALERT: ACF029-14;Memory Usage;CRITICAL;HARD;1;(Service check timed out after 67.41 seconds)
00326: [1637007925] SERVICE ALERT: CITES-VEEAM-R1;Disk Usage on P:/;CRITICAL;SOFT;1;(Service check timed out after 67.59 seconds)
00328: [1637007925] SERVICE ALERT: cctv_MAGPIE;Drive C: Disk Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host magpie.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00335: [1637007925] SERVICE ALERT: cctv_LOON;Drive C: Disk Usage;CRITICAL;SOFT;1;(Service check timed out after 67.59 seconds)
00337: [1637007925] SERVICE ALERT: cctv_ROADRUNNER;Drive C: Disk Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host roadrunner.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00344: [1637007925] SERVICE ALERT: cic-it-admin14-wmi;Drive C: Disk Usage;CRITICAL;SOFT;1;(Service check timed out after 67.56 seconds)
00350: [1637007925] SERVICE ALERT: cctv_SHRIKE;Drive C: Disk Usage;CRITICAL;SOFT;1;(Service check timed out after 67.56 seconds)
00356: [1637007925] SERVICE ALERT: cic-it-db15-wmi;Drive G: Disk Usage;CRITICAL;SOFT;1;(Service check timed out after 67.56 seconds)
00358: [1637007925] SERVICE ALERT: cic-it-f12-wmi;Drive R: Disk Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host cic-it-f12.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00367: [1637007925] SERVICE ALERT: cctv_TROGON;Drive D: Disk Usage;CRITICAL;SOFT;1;(Service check timed out after 67.45 seconds)
00373: [1637007925] SERVICE ALERT: cctv_WYVERN;Page File Usage;CRITICAL;SOFT;1;(Service check timed out after 67.35 seconds)
00379: [1637007925] SERVICE ALERT: FAA-FILES;Disk Usage on H:/;CRITICAL;SOFT;1;(Service check timed out after 67.19 seconds)
00387: [1637007925] SERVICE ALERT: crocosaurus.ad.uillinois.edu;Drive C: Disk Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host crocosaurus.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00388: [1637007925] SERVICE ALERT: ACF024-13;Swap Usage;UNKNOWN;HARD;1;UNKNOWN: An error occured connecting to API. (Connection error: '[Errno 111] Connection refused')
00394: [1637007925] SERVICE ALERT: publish.illinois.edu;Shibboleth login check;CRITICAL;SOFT;1;(Service check timed out after 60.25 seconds)
00401: [1637007925] SERVICE ALERT: s_1233_001_aces;Local Area Connection_ 2 Bandwidth - Inbound;CRITICAL;SOFT;1;(Service check timed out after 61.23 seconds)
00407: [1637007925] SERVICE ALERT: shiny.citl.illinois.edu;Disk Usage on /;CRITICAL;HARD;1;(Service check timed out after 63.15 seconds)
00413: [1637007925] SERVICE ALERT: splunk-deployment-test.machinedata.illinois.edu;Service status for: amazon-ssm-agent;CRITICAL;SOFT;1;(Service check timed out after 64.08 seconds)
00421: [1637007925] SERVICE ALERT: cctv_ROC;CPU Usage;CRITICAL;SOFT;1;(Service check timed out after 63.26 seconds)
00423: [1637007925] SERVICE ALERT: cctv_FINCH;CPU Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host finch.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00430: [1637007925] SERVICE ALERT: ADMIN-ZPRN1;Service status for: Spooler;CRITICAL;SOFT;1;(Service check timed out after 64.64 seconds)
00436: [1637007925] SERVICE ALERT: cctv_PHOENIX;CPU Usage;CRITICAL;SOFT;1;(Service check timed out after 64.59 seconds)
00442: [1637007925] SERVICE ALERT: cctv_LOON;CPU Usage;CRITICAL;SOFT;1;(Service check timed out after 64.52 seconds)
00450: [1637007925] SERVICE ALERT: ACF022-43;CPU Usage;CRITICAL;HARD;1;(Service check timed out after 65.16 seconds)
00456: [1637007925] SERVICE ALERT: vpn4g-2.gw.illinois.edu;Port-channel1 Bandwidth;CRITICAL;SOFT;1;(Service check timed out after 65.66 seconds)
00462: [1637007925] SERVICE ALERT: CITES-VEEAM-R1;Disk Usage on H:/;CRITICAL;SOFT;1;(Service check timed out after 66.78 seconds)
00468: [1637007925] SERVICE ALERT: ACF029-23;Service Status: KeyAccess;CRITICAL;HARD;1;(Service check timed out after 67.22 seconds)
00476: [1637007925] SERVICE ALERT: cctv_BULLFINCH;Drive K: Disk Usage;CRITICAL;SOFT;1;(Service check timed out after 67.22 seconds)
00478: [1637007925] SERVICE ALERT: cctv_CONURE;Drive C: Disk Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host conure.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00483: [1637007925] SERVICE ALERT: cctv_KEA;Memory Usage;UNKNOWN;HARD;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host kea.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00488: [1637007925] SERVICE ALERT: cctv_COOT;Page File Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host coot.ad.uillinois.edu might just be really busy, it might not even be running Windows.
00491: [1637007925] SERVICE ALERT: conflictresolution.illinois.edu;DNS Resolution;CRITICAL;SOFT;1;CRITICAL - Plugin timed out while executing system call
00500: [1637007925] SERVICE ALERT: cctv_HERON;Memory Usage;CRITICAL;SOFT;1;(Service check timed out after 67.16 seconds)

Could you please take a peek at /usr/local/nagios/var/nagios.log and see if it is still going on?

Thanks!

[murdock]

Hi, I watched the log last night; yes, we're getting a lot of timeouts... but that is typical; I'd talked with Sean about it last week.

Long story short, I've been deactivating abandoned probes; we're down about 200 from ~9400 to ~9200. I've got Admins that are responsible for their own probes, but they've been ignoring them -- so now I'm deactivating them.

We were up and running with lots more probe last week; the main issue now is host and service details are not displayed in the web interface. I would think getting rid of bad probes would improve the situation, not make it worse...

Rob