glibc and valgrind memcheck errors (cgi segmentation fault)
Posted: Fri Mar 20, 2009 5:57 pm
This is a multi-part message in MIME format.
------_=_NextPart_001_01C9A98D.B7CAD33C
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Anyone have any ideas on how I could go about troubleshooting this? I
temporarily removed all of the reporting archives, nagios.log and all
other log files in the var dir, turned off process-performance-data, and
restarted nagios. Still valgrind was reporting the cgi errors. Is this a
bug?
Regards,
Justin Amburn=20
_____________________________________________
From: Justin Amburn=20
Sent: Thursday, March 19, 2009 11:06 AM
To: '[email protected]'
Subject: glibc and valgrind memcheck errors (cgi segmentation fault)
with nagios 3.06 on Redhat 4.7 x86
Hi all,=20
I'm desperately needing help with our nagios server. It's been live for
about 1.25 years but 1 week ago it started giving sporadic segmentation
faults at 3am when calling CGI files. The problem is related to the age
of the nagios configuration, because when I restore from a configuration
that's a few days younger the problem disappears for a few days before
coming back again. To address this, I updated the OS from RH 4.6 RH to
4.7 and from nagios 3.03 to 3.06. Here is the diagnostics data I have
collected so far:
[root@nagios01 nagmin]# uname -a
Linux nagios01 2.6.9-78.0.13.ELsmp #1 SMP Wed Jan 7 17:52:47 EST 2009
i686 i686 i386 GNU/Linux
From httpd/error_log:
[Thu Mar 19 03:11:41 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer:
http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:43 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x08cd34b8 ***,
referer: http://123.45.67.89/nagios/
[Thu Mar 19 03:11:43 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer: http://123.45.67.89/nagios/
[Thu Mar 19 03:11:46 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x09ac94b8 ***,
referer: http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:46 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer:
http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:51 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x09b9b4b8 ***,
referer: http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:51 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer:
http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:57 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x08aec4b8 ***
[Thu Mar 19 03:11:57 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi
[Thu Mar 19 03:12:02 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x08cac4b8 ***,
referer: http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:12:02 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer:
http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:12:07 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (fast): 0x09d06508 ***
[Thu Mar 19 03:12:07 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (fast): 0x097a44f0 ***
Next in the console I run this to see if the glibc is version related,
but it's already the most recent from redhat:
[root@nagios01 nagmin]# rpm -qa | grep glibc
glibc-headers-2.3.4-2.41.el4_7.1
glibc-kernheaders-2.4-9.1.103.EL
glibc-common-2.3.4-2.41.el4_7.1
glibc-2.3.4-2.41.el4_7.1
glibc-devel-2.3.4-2.41.el4_7.1
Then to reproduce the CGI output:
[root@nagios01 nagmin]# export REQUEST_METHOD=3D"GET"
[root@nagios01 nagmin]# /usr/local/nagios/sbin/status.cgi
When it doesn't work it outputs this single line: Segmentation fault
Now some output from valgrind:
=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D PuTTY log 2009.03.19 =
09:44:35
=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D
...[email truncated]...
This post was automatically imported from historical nagios-devel mailing list archives
Original poster: ustin Amburn=2
------_=_NextPart_001_01C9A98D.B7CAD33C
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Anyone have any ideas on how I could go about troubleshooting this? I
temporarily removed all of the reporting archives, nagios.log and all
other log files in the var dir, turned off process-performance-data, and
restarted nagios. Still valgrind was reporting the cgi errors. Is this a
bug?
Regards,
Justin Amburn=20
_____________________________________________
From: Justin Amburn=20
Sent: Thursday, March 19, 2009 11:06 AM
To: '[email protected]'
Subject: glibc and valgrind memcheck errors (cgi segmentation fault)
with nagios 3.06 on Redhat 4.7 x86
Hi all,=20
I'm desperately needing help with our nagios server. It's been live for
about 1.25 years but 1 week ago it started giving sporadic segmentation
faults at 3am when calling CGI files. The problem is related to the age
of the nagios configuration, because when I restore from a configuration
that's a few days younger the problem disappears for a few days before
coming back again. To address this, I updated the OS from RH 4.6 RH to
4.7 and from nagios 3.03 to 3.06. Here is the diagnostics data I have
collected so far:
[root@nagios01 nagmin]# uname -a
Linux nagios01 2.6.9-78.0.13.ELsmp #1 SMP Wed Jan 7 17:52:47 EST 2009
i686 i686 i386 GNU/Linux
From httpd/error_log:
[Thu Mar 19 03:11:41 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer:
http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:43 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x08cd34b8 ***,
referer: http://123.45.67.89/nagios/
[Thu Mar 19 03:11:43 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer: http://123.45.67.89/nagios/
[Thu Mar 19 03:11:46 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x09ac94b8 ***,
referer: http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:46 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer:
http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:51 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x09b9b4b8 ***,
referer: http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:51 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer:
http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:11:57 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x08aec4b8 ***
[Thu Mar 19 03:11:57 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi
[Thu Mar 19 03:12:02 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (normal): 0x08cac4b8 ***,
referer: http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:12:02 2009] [error] [client 123.45.67.89] Premature end
of script headers: status.cgi, referer:
http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:12:07 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (fast): 0x09d06508 ***
[Thu Mar 19 03:12:07 2009] [error] [client 123.45.67.89] *** glibc
detected *** free(): invalid next size (fast): 0x097a44f0 ***
Next in the console I run this to see if the glibc is version related,
but it's already the most recent from redhat:
[root@nagios01 nagmin]# rpm -qa | grep glibc
glibc-headers-2.3.4-2.41.el4_7.1
glibc-kernheaders-2.4-9.1.103.EL
glibc-common-2.3.4-2.41.el4_7.1
glibc-2.3.4-2.41.el4_7.1
glibc-devel-2.3.4-2.41.el4_7.1
Then to reproduce the CGI output:
[root@nagios01 nagmin]# export REQUEST_METHOD=3D"GET"
[root@nagios01 nagmin]# /usr/local/nagios/sbin/status.cgi
When it doesn't work it outputs this single line: Segmentation fault
Now some output from valgrind:
=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D PuTTY log 2009.03.19 =
09:44:35
=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D~=3D
...[email truncated]...
This post was automatically imported from historical nagios-devel mailing list archives
Original poster: ustin Amburn=2