This is a multi-part message in MIME format.
------_=_NextPart_001_01C9A8BD.54763E94
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Hi all,=20
I'm desperately needing help with our nagios server. It's been live for
about 1.25 years but 1 week ago it started giving sporadic segmentation
faults at 3am when calling CGI files. The problem is related to the age
of the nagios configuration, because when I restore from a configuration
that's a few days younger the problem disappears for a few days before
coming back again. To address this, I updated the OS from RH 4.6 RH to
4.7 and from nagios 3.03 to 3.06. Here is the diagnostics data I have
collected so far:
[root@nagios01 nagmin]# uname -a
Linux nagios01 2.6.9-78.0.13.ELsmp #1 SMP Wed Jan 7 17:52:47 EST 2009
i686 i686 i386 GNU/Linux
From httpd/error_log:
[Thu Mar 19 03:11:41 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer:
http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:43 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x08cd34b8 ***,
referer: http://123.45.67.89/nagios/
[Thu Mar 19 03:11:43 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer: http://123.45.67.89/nagios/
[Thu Mar 19 03:11:46 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x09ac94b8 ***,
referer: http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:46 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer:
http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:51 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x09b9b4b8 ***,
referer: http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:51 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer:
http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:57 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x08aec4b8 ***
[Thu Mar 19 03:11:57 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi
[Thu Mar 19 03:12:02 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x08cac4b8 ***,
referer: http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:12:02 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer:
http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:12:07 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (fast): 0x09d06508 ***
[Thu Mar 19 03:12:07 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (fast): 0x097a44f0 ***
Next in the console I run this to see if the glibc is version related,
but it's already the most recent from redhat:
[root@nagios01 nagmin]# rpm -qa | grep glibc
glibc-headers-2.3.4-2.41.el4_7.1
glibc-kernheaders-2.4-9.1.103.EL
glibc-common-2.3.4-2.41.el4_7.1
glibc-2.3.4-2.41.el4_7.1
glibc-devel-2.3.4-2.41.el4_7.1
Then to reproduce the CGI output:
[root@nagios01 nagmin]# export REQUEST_METHOD=3D"GET"
[root@nagios01 nagmin]# /usr/local/nagios/sbin/status.cgi
When it doesn't work it outputs this single line: Segmentation fault
Now some output from valgrind:
=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D PuTTY log 2009.03.19 =
09:44:35
=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D
valgrind --tool=3Dmemcheck /usr/local/nagios/sbin/status.cgi
=3D=3D23834=3D=3D Memcheck, a memory error detector.
=3D=3D23834=3D=3D Copyright (C) 2002-2005, and GNU GPL'd, by Julian =
Seward et
al.
=3D=3D23834=3D=3D Using LibVEX rev 1575, a library for dynamic binary
translation.
=3D=3D23834=3D=3D Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks =
LLP.
=3D=3D23834=3D=3D Using valgrind-3.1.1, a dynamic binary instrumentation
framework.
=3D=3D23834=3D=3D Copyright (C) 2000-2005, and GNU GPL'd, by Julian =
Seward et
al.
=3D=3D23834=3D=3D For more details, rerun with:
...[email truncated]...
This post was automatically imported from historical nagios-devel mailing list archives
Original poster: [email protected]