Page 1 of 2

Service Status not displaying

Posted: Tue Aug 02, 2011 6:17 pm
by rseiwert
Running 1.5 right now. I have a couple of critical services but when I click to show critical services or even the all service problems the swirly just sits and spins and I cannot see what is alerting.

Re: Service Status not displaying

Posted: Wed Aug 03, 2011 9:24 am
by rseiwert
not sure how to bump but the seeing 9 critical services and not being able to see what is currently alerting is a critical problem.

Re: Service Status not displaying

Posted: Wed Aug 03, 2011 9:28 am
by lmiltchev
Do you have any other issues besides not being able to see critical services? Have you tried restarting nagios? Do you have any messages in your error logs that can give us a little bit more information?

Re: Service Status not displaying

Posted: Wed Aug 03, 2011 11:54 am
by rseiwert
I am sure it's some check's result code that is causing the serverside php script to crash. Have discovered so far that core works. It's not isolated to "critical" services, clicked on the OK services of one host and it didn't display. I have tried deleting some of the new checks and hosts but haven't made any progress in determining what is causing XI's php to crash.
lmiltchev wrote:Do you have any other issues besides not being able to see critical services? Have you tried restarting nagios? Do you have any messages in your error logs that can give us a little bit more information?
Being a linux noob I an not even sure where to look for error logs related to this. 25 years ago I was a C programmer working in Zenix on Altos but that was a long time ago. I would assume errors would be in /var/logs but been looking for php and nagios error logs for a while now with no luck. Any suggestions on where to find a clue to what is causing XI to fail?

Re: Service Status not displaying

Posted: Wed Aug 03, 2011 12:02 pm
by rseiwert
BTW I am running the standard Windows VM image upgraded from 1.3->1.4->1.5. Very minimal custom checks.

Re: Service Status not displaying

Posted: Wed Aug 03, 2011 12:11 pm
by rseiwert
From error log in /var/log/httpd (silly me I was looking for apache).

[Wed Aug 03 12:40:33 2011] [notice] child pid 430 exit signal Segmentation fault (11)
[Wed Aug 03 12:41:03 2011] [notice] child pid 29600 exit signal Segmentation fault (11)
[Wed Aug 03 12:41:33 2011] [notice] child pid 1214 exit signal Segmentation fault (11)
[Wed Aug 03 13:09:55 2011] [notice] child pid 1910 exit signal Segmentation fault (11)
[Wed Aug 03 13:10:23 2011] [notice] child pid 1507 exit signal Segmentation fault (11)
[Wed Aug 03 13:10:26 2011] [notice] child pid 871 exit signal Segmentation fault (11)

Re: Service Status not displaying

Posted: Wed Aug 03, 2011 1:14 pm
by lmiltchev
The "segmentation fault" is a bug in previous versions of Nagios XI. It is caused by exceeding the memory limit in one of the PHP built-in functions because of a very long output strings. This is supposed to be fixed in R1.6.

Since you are currently using R1.5, I would suggest give R1.6 a try and see if this is going to fix your problem.

Before you proceed with an upgrade, please, review our documentation on the upgrade process:

http://assets.nagios.com/downloads/nagi ... ctions.pdf

Re: Service Status not displaying

Posted: Thu Aug 04, 2011 10:27 am
by rseiwert
I upgraded to 1.6 and the problem persists.

[root@nagios httpd]# tail error_log
[Thu Aug 04 11:12:25 2011] [notice] child pid 6550 exit signal Segmentation fault (11)
[Thu Aug 04 11:12:54 2011] [notice] child pid 19900 exit signal Segmentation fault (11)
[Thu Aug 04 11:13:25 2011] [notice] child pid 6547 exit signal Segmentation fault (11)
[Thu Aug 04 11:13:55 2011] [notice] child pid 20524 exit signal Segmentation fault (11)

XI is basically unusable. Also trying to build a path back to XI from Core. I set the notes_url on the hostgroup "All Hosts" to
http://nagios.vca.com/nagiosxi/includes ... =$HOSTNAME$ but the link doesn't show up in core. Since this doesn't seem to work anymore I guess I need to use SQL to mass update the hosts Notes_URL and then rewrite the config? It should work off the hostgroups and I shouldn't have to set this on each host.

Also last night had something very interesting happen after the update. All services were last checked two hours in the future with the next check scheduled two hours and 5 mins (or something like) in the future. I was wondering how to tell Nagios to recheck everything immediately. Or schedule an immediate check for everything not OK? Basically had alot of false positives masking to conditions for several hours last not.

How to I solve this... My monitoring system needs to report problems, not have false positives, and not make more work than checking everything by hand.

Re: Service Status not displaying

Posted: Thu Aug 04, 2011 10:59 am
by rseiwert
My assumption that notes_url is inherited from the hostgroup is invalid? Using the template seems to have worked.

Re: Service Status not displaying

Posted: Thu Aug 04, 2011 11:46 am
by rseiwert
Using the link from core back to the services I was able to isolate the "sit and spin" issue to one service. As I thought it is the application log. I am including the massive response here to document...

CRITICAL - [Triggered by _ItemCount>10] - 24 event(s) of at least Severity Level "Error", were recorded in the last 1 hours from the Application Event Log. (List is on next line. Fields shown are - Logfile:TimeGenerated:Type:SourceName:Message)
Application:20110804163632.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804163632.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804163132.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804163132.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804162632.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804162632.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804162132.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804162132.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804161632.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804161632.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804161132.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804161132.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804160632.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804160632.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804160132.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804160132.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804155632.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804155632.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804155132.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804155132.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804154632.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804154632.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804154131.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.
Application:20110804154131.000000-000:Error:SideBySide:Activation context generation failed for "C:\Windows\system32\conhost.exe". Dependent Assembly Microsoft.Windows.SystemCompatible,processorArchitecture="amd64",publicKeyToken="6595b64144ccf1df",type="win32",version="6.0.7600.16816" could not be found. Please use sxstrace.exe for detailed diagnosis.