*BIG BUG*

[niebais]

OS: Centos 5.5
Nagios Version: 2011R1.2

Symptoms:
when clicking on the button: Open Service Problems or All Service Problems nothing appears when it shows errors.

How to duplicate:
Errors will appear in the "Service Status Summary"

I click on "Open Service Problems" or "All Service Problems" and no services appear. It is just white and blank with nothing and the page is fully loaded.

I attached the image so you can see what is happening.

Please help me get this one fixed quickly, it is serious.

[niebais]

Ok,
So here's something that jolted it back into submission, but it happened on Friday and I'm worried because it happened again today.

I did the following:
service nagios stop
service ndo2db stop
service mysqld stop
cd /usr/local/nagiosxi/scripts
./repairmysql.sh nagios *

The script came up with some DB errors, plus mentioned that I shouldn't use the -q option in some places. So here's what I modified the script to (line 44):
$cmd --safe-recover -r $t - then I reran it.

Next I killed ALL the Nagios daemons and I stopped postgresql.
Then I started up Nagios and applied the configuration. Things came back after that.

[mguthrie]

If you see this occur again, run:

tail -f /var/log/httpd/error_log

and then load that page.

We'll run some tests on our end and see if we can recreate this.

[niebais]

Thanks, I hope it doesn't happen again.

[mguthrie]

If there was corruption in one of the tables, the backend call for the xml may have timed out, returning false, and caused the blank page. So far we haven't recreated anything similar unless we stop the mysqld process. If this issue shows up again let us know though.

[niebais]

Thanks I will, so far it's been stable since I went through the "DB cleanup document" for Nagios XI. I had to truncate the logentries table and another one, but ever since then the load dropped on our server as well and I don't even know what I'm missing yet.

[admin]

Thanks for the report on this!

[niebais]

Ok,
This problem keeps happening once per week. Here's what I see in the logs when this problem happens:

Code: Select all

[Fri Jun 17 08:40:40 2011] [error] [client 10.35.42.161] PHP Warning:  DOMDocument::load(http://myserver/nagiosxi/backend/?cmd=getservicestatus&username=myhelpgroup&ticket=sugqs3qv6vlsd7j3o3c7990biruhr94n7jf0l8vo7ilio6ke8tih5gpmbl56t6r2) [<a href='domdocument.load'>domdocument.load</a>]: failed to open stream: HTTP request failed!  in /usr/local/nagiosxi/html/includes/components/helpdeskmap/alerts.php on line 148, referer: http://myserver/nagiosxi/includes/components/helpdeskmap/
[Fri Jun 17 08:40:40 2011] [error] [client 10.35.42.161] PHP Warning:  DOMDocument::load() [<a href='domdocument.load'>domdocument.load</a>]: I/O warning : failed to load external entity "http://myserver/nagiosxi/backend/?cmd=getservicestatus&username=myhelpgroup&ticket=sugqs3qv6vlsd7j3o3c7990biruhr94n7jf0l8vo7ilio6ke8tih5gpmbl56t6r2" in /usr/local/nagiosxi/html/includes/components/helpdeskmap/alerts.php on line 148, referer: http://myserver/nagiosxi/includes/components/helpdeskmap/
[Fri Jun 17 08:40:41 2011] [notice] child pid 8901 exit signal Segmentation fault (11)
[Fri Jun 17 08:40:41 2011] [error] [client 10.35.42.161] PHP Notice:  Undefined index:  2556 in /usr/local/nagiosxi/html/includes/components/helpdeskmap/alerts.php on line 114, referer: http://myserver/nagiosxi/includes/components/helpdeskmap/
[Fri Jun 17 08:40:41 2011] [error] [client 10.35.42.161] PHP Notice:  Undefined index:  2556 in /usr/local/nagiosxi/html/includes/components/helpdeskmap/alerts.php on line 114, referer: http://myserver/nagiosxi/includes/components/helpdeskmap/
[Fri Jun 17 08:40:42 2011] [notice] child pid 10649 exit signal Segmentation fault (11)

It appears to be related to the database getting out of sync somehow. However, it segfaults when this occurs. Not a great scenario. Any ideas on how to prevent this from ocurring in the future. It also looks like it might be related to the backendapi component that we have in place.

[mguthrie]

We did just release version 2011R1.4 yesterday, which has some revisions to the nagios init script. I'm wondering if that might take care of this issue.

[niebais]

I'll update it on this coming Tuesday. I hope it does