Internal server error on web interface when clicking "host"

yaoyao · Post by **yaoyao** » Mon Jul 28, 2014 2:54 pm

I haven't checked web interface for some time. Nagios alerts were sent out fine. Today, when I added another host to monitor, I found that web interface stopped working. If I clicked host or service group more, http would exit as well. Not sure what was the cause.

Error_log shows:

[Mon Jul 28 14:25:49 2014] [notice] Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/0.9.8x DAV/2 PHP/5.3.10 configured -- resuming normal operations
[Mon Jul 28 14:36:50 2014] [error] [client 172.16.64.49] Premature end of script headers: status.cgi, referer: http://sunmon/nagios/side.php
[Mon Jul 28 14:36:51 2014] [notice] caught SIGTERM, shutting down
[Mon Jul 28 14:37:50 2014] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Jul 28 14:37:52 2014] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Jul 28 14:37:52 2014] [notice] Digest: generating secret for digest authentication ...
[Mon Jul 28 14:37:52 2014] [notice] Digest: done
[Mon Jul 28 14:37:53 2014] [notice] Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/0.9.8x DAV/2 PHP/5.3.10 configured -- resuming normal operations
[Mon Jul 28 14:50:46 2014] [error] [client 172.16.64.49] Premature end of script headers: status.cgi, referer: http://sunmon/nagios/side.php
[Mon Jul 28 14:50:47 2014] [notice] caught SIGTERM, shutting down
[Mon Jul 28 14:50:48 2014] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Jul 28 14:50:49 2014] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Jul 28 14:50:49 2014] [notice] Digest: generating secret for digest authentication ...
[Mon Jul 28 14:50:49 2014] [notice] Digest: done
[Mon Jul 28 14:50:50 2014] [notice] Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/0.9.8x DAV/2 PHP/5.3.10 configured -- resuming normal operations

Access log shows:

172.16.64.49 - - [28/Jul/2014:14:50:40 -0500] "GET /nagios/main.php HTTP/1.1" 200 5001
172.16.64.49 - - [28/Jul/2014:14:50:40 -0500] "GET /nagios/rss-corefeed.php HTTP/1.1" 200 499
172.16.64.49 - - [28/Jul/2014:14:50:40 -0500] "GET /nagios/rss-newsfeed.php HTTP/1.1" 200 572
172.16.64.49 - - [28/Jul/2014:14:50:46 -0500] "GET /nagios/cgi-bin/status.cgi?hostgroup=all&style=hostdetail HTTP/1.1" 500 531
127.0.0.1 - - [28/Jul/2014:14:53:53 -0500] "GET / HTTP/1.0" 200 44

sreinhardt · Post by **sreinhardt** » Mon Jul 28, 2014 3:13 pm

Sounds like php thinks that side.php is ending without closing properly. What OS, version, and architecture are you on? Also what nagios version and how was it installed(source, rpm, other)?

yaoyao · Post by **yaoyao** » Mon Jul 28, 2014 3:15 pm

Solaris 10 on a V240

yaoyao · Post by **yaoyao** » Mon Jul 28, 2014 3:18 pm

nagios 3.5 and compiled from source.

tmcdonald · Post by **tmcdonald** » Tue Jul 29, 2014 11:02 am

I would argue it's actually the CGIs erroring out:

Code: Select all

[Mon Jul 28 14:50:46 2014] [error] [client 172.16.64.49] Premature end of script headers: status.cgi, referer: http://sunmon/nagios/side.php

[28/Jul/2014:14:50:46 -0500] "GET /nagios/cgi-bin/status.cgi?hostgroup=all&style=hostdetail HTTP/1.1" 500 531

What happens if you run it manually from the CLI?

Code: Select all

root@nagiosxi: /usr/local/nagios/libexec
$ REQUEST_METHOD="GET" QUERY_STRING="hostgroup=all&style=hostdetail" REMOTE_USER="nagiosadmin" /usr/local/nagios/sbin/status.cgi > /tmp/status.txt

yaoyao · Post by **yaoyao** » Tue Jul 29, 2014 12:03 pm

The output looks normal if I ran it manually.

Cache-Control: no-store
Pragma: no-cache
Refresh: 90
Last-Modified: Tue, 29 Jul 2014 16:59:28 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-type: text/html

<html>
<head>
<link rel="shortcut icon" href="/nagios/images/favicon.ico" type="image/ico">
<title>
Current Network Status
</title>
<link rel='stylesheet' type='text/css' href='/nagios/stylesheets/common.css' />
<link rel='stylesheet' type='text/css' href='/nagios/stylesheets/status.css' />
<script type='text/javascript' src='/nagios/js/jquery-1.7.1.min.js'></script>
<script type='text/javascript'>
$(document).ready(function() { $('#top_page_numbers').append($('#bottom_page_numbers').html() ); });function set_limit(url) {
this.location = url+'&limit='+$('#limit').val();
}</script>
</head>
<body class='status'>


<table class='headertable'>
<tr>
<td align=left valign=top width=33%>
<TABLE CLASS='infoBox' BORDER=1 CELLSPACING=0 CELLPADDING=0>
<TR><TD CLASS='infoBox'>
<DIV CLASS='infoBoxTitle'>Current Network Status</DIV>
Last Updated: Tue Jul 29 11:59:28 CDT 2014<BR>
Updated every 90 seconds<br>
Nagios® Core™ 3.5.0 - <A HREF='http://www.nagios.org' TARGET='_new' CLASS='homepageURL'>www.nagios.org</A><BR>
Logged in as <i>nagiosadmin</i><BR>
</TD></TR>
</TABLE>
<table class='linkBox'>
<tr><td class='linkBox'>
<a href='status.cgi?hostgroup=all&style=detail'>View Service Status Detail For All Host Groups</a><br>
<a href='status.cgi?hostgroup=all&style=overview'>View Status Overview For All Host Groups</a><br>
<a href='status.cgi?hostgroup=all&style=summary'>View Status Summary For All Host Groups</a><br>
<a href='status.cgi?hostgroup=all&style=grid'>View Status Grid For All Host Groups</a><br>
</td></tr>
</table>
</td>
<td align=center valign=top width=33%>
<td class='serviceTotals'>0</td>
<td class='serviceTotalsCRITICAL'>1</td>
<td class='serviceTotals'>0</td>
</tr>
</table>
</td></tr><tr><td align='center'>
<table class='serviceTotals'>
<tr>
<th class='serviceTotals'><a class='serviceTotals' href='status.cgi?hostgroup=all&style=detail&servicestatustypes=28&hoststatustypes=15'><em>All Problems</em></a></th>
<th class='serviceTotals'><a class='serviceTotals' href='status.cgi?hostgroup=all&style=detail&hoststatustypes=15'><em>All Types</em></a></th>
</tr><tr>
<td class='serviceTotalsPROBLEMS'>1</td>
<td class='serviceTotals'>241</td>
</tr>
</table>
</td></tr>
</table>
</div>
</td>
<td align=right valign=bottom>
</td>
</tr>
</table>
<table class='pageTitle' border='0' width='100%'>
<tr>
<td valign=top align=left width=33%>
</td><td valign=top align=center width=33%>
<div align='center' class='statusTitle'>Host Status Details For All Host Groups</div>
<br></td>
<td valign=top align=right width=33%></td>
</tr>
</table>
<div id='pagelimit'>
<div id='result_limit'>
<label for='limit'>Limit Results: </label>
<select onchange='set_limit("status.cgi?hostgroup=all&style=hostdetail")' name='limit' id='limit'>
<option value='50'>50</option>
<option selected='selected' value='100'>100</option>
<option value='250'>250</option>
<option value='1000'>1000</option>
<option value='0'>All</option>
</select></div>
<div id='top_page_numbers'></div>
</div>
<div align='center'>
<table border=0 class='status' width='100%'>
<tr>
<th class='status'>Host <a href='status.cgi?hostgroup=all&style=hostdetail&sorttype=1&sortoption=1'><IMG SRC='/nagios/images/up.gif' border=0 ALT='Sort by host name (ascending)' TITLE='Sort by host name (ascending)'></a><a href='status.cgi?hostgroup=all&style=hostdetail&sorttype=2&sortoption=1'><IMG SRC='/nagios/images/down.gif' border=0 ALT='Sort by host name (descending)' TITLE='Sort by host name (descending)'></a></th><th class='status'>Status <a href='status.cgi?hostgroup=all&style=hostdetail&sorttype=1&sortoption=8'><IMG SRC='/nagios/images/up.gif' border=0 ALT='Sort by host status (ascending)' TITLE='Sort by host status (ascending)'></a><a href='status.cgi?hostgroup=all&style=hostdetail&sorttype=2&sortoption=8'><IMG SRC='/nagios/images/down.gif' border=0 ALT='Sort by host status (descending)' TITLE='Sort by host status (descending)'></a></th><th class='status'>Last Check <a href='status.cgi?hostgroup=all&style=hostdetail&sorttype=1&sortoption=4'><IMG SRC='/nagios/images/up.gif' border=0 ALT='Sort by last check time (ascending)' TITLE='Sort by last check time (ascending)'></a><a href='status.cgi?hostgroup=all&style=hostdetail&sorttype=2&sortoption=4'><IMG SRC='/nagios/images/down.gif' border=0 ALT='Sort by last check time (descending)' TITLE='Sort by last check time (descending)'></a></th><th class='status'>Duration <a href='status.cgi?hostgroup=all&style=hostdetail&sorttype=1&sortoption=6'><IMG SRC='/nagios/images/up.gif' border=0 ALT='Sort by state duration (ascending)' TITLE='Sort by state duration (ascending)'></a><a href='status.cgi?hostgroup=all&style=hostdetail&sorttype=2&sortoption=6'><IMG SRC='/nagios/images/down.gif' border=0 ALT='Sort by state duration time (descending)' TITLE='Sort by state duration time (descending)'></a></th><th class='status'>Status Information</th>
</tr>
<tr>
<td class='statusHOSTUP'><table border=0 WIDTH='100%' cellpadding=0 cellspacing=0>
<tr>
<td align='left'>
<table border=0 cellpadding=0 cellspacing=0>
<tr>
<td align=left valign=center class='statusHOSTUP'><a href='extinfo.cgi?type=1&host=apollo' title='172.16.8.3'>apollo</a> </td>
</tr>
</table>
</td>
<td align=right valign=center>
<table border=0 cellpadding=0 cellspacing=0>
<tr>
<td><a href='status.cgi?host=apollo'><img src='/nagios/images/status2.gif' border=0 alt='View Service Details For This Host' title='View Service Details For This Host'></a></td>
</tr>
</table>
</td>
</tr>
</table>
</td>
<td class='statusHOSTUP'>UP</td>
<td class='statusEven' nowrap>07-29-2014 11:58:46</td>
<td class='statusEven' nowrap> 3d 0h 6m 3s</td>
<td class='statusEven' valign='center'>PING OK - Packet loss = 0%, RTA = 0.37 ms </td>
</tr>
<tr>
<td class='statusHOSTUP'><table border=0 WIDTH='100%' cellpadding=0 cellspacing=0>
<tr>
<td align='left'>
<table border=0 cellpadding=0 cellspacing=0>
<tr>
<td align=left valign=center class='statusHOSTUP'><a href='extinfo.cgi?type=1&host=aqua' title='172.16.8.122'>aqua</a> </td>
</tr>
</table>
</td>
<td align=right valign=center>
<table border=0 cellpadding=0 cellspacing=0>
<tr>
<td><a href='status.cgi?host=aqua'><img src='/nagios/images/status2.gif' border=0 alt='View Service Details For This Host' title='View Service Details For This Host'></a></td>
</tr>
</table>
</td>
</tr>
</table>
</td>
<td class='statusHOSTUP'>UP</td>
<td class='statusOdd' nowrap>07-29-2014 11:54:16</td>
<td class='statusOdd' nowrap> 5d 16h 19m 33s</td>
<td class='statusOdd' valign='center'>PING OK - Packet loss = 0%, RTA = 0.47 ms </td>
</tr>
<tr>
<td class='statusHOSTUP'><table border=0 WIDTH='100%' cellpadding=0 cellspacing=0>
<tr>
<td align='left'>
<table border=0 cellpadding=0 cellspacing=0>
<tr>
<td align=left valign=center class='statusHOSTUP'><a href='extinfo.cgi?type=1&host=atlas' title='172.16.8.126'>atlas</a> </td>
</tr>
</table>
</td>
<td align=right valign=center>
<table border=0 cellpadding=0 cellspacing=0>
<tr>
<td><a href='status.cgi?host=atlas'><img src='/nagios/images/status2.gif' border=0 alt='View Service Details For This Host' title='View Service Details For This Host'></a></td>
</tr>
</table>
</td>
</tr>
</table>
</td>
<td class='statusHOSTUP'>UP</td>
<td class='statusEven' nowrap>07-29-2014 11:55:36</td>
<td class='statusEven' nowrap> 3d 5h 44m 3s</td>
<td class='statusEven' valign='center'>PING OK - Packet loss = 0%, RTA = 92.00 ms </td>

tmcdonald · Post by **tmcdonald** » Wed Jul 30, 2014 1:58 pm

Can you post the configs for the newly-added hosts and services? There might be something in them causing the script to fail.

yaoyao · Post by **yaoyao** » Wed Jul 30, 2014 2:07 pm

I just found out it is browser issue. It works in IE, but not in chrome or firefox. Still dont know why, but I'll remember to use it in IE when needed.

tmcdonald · Post by **tmcdonald** » Thu Jul 31, 2014 10:09 am

Very odd though. Usually we see issues with IE and not the other way around. When you visit the page in FF or Chrome, are there any errors in /var/log/httpd/error_log?

yaoyao · Post by **yaoyao** » Fri Aug 01, 2014 8:48 am

Yes.

[Mon Jul 28 14:25:49 2014] [notice] Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/0.9.8x DAV/2 PHP/5.3.10 configured -- resuming normal operations
[Mon Jul 28 14:36:50 2014] [error] [client 172.16.64.49] Premature end of script headers: status.cgi, referer: http://sunmon/nagios/side.php

Nagios Support Forum

Internal server error on web interface when clicking "host"

Internal server error on web interface when clicking "host"

Re: Internal server error on web interface when clicking "ho

Re: Internal server error on web interface when clicking "ho

Re: Internal server error on web interface when clicking "ho

Re: Internal server error on web interface when clicking "ho

Re: Internal server error on web interface when clicking "ho

Re: Internal server error on web interface when clicking "ho

Re: Internal server error on web interface when clicking "ho

Re: Internal server error on web interface when clicking "ho

Re: Internal server error on web interface when clicking "ho