Subject: [clisp-list] What causes hard exits from clisp?

From: Christophe Rhodes <[email-masked]>
Date: Tue Apr 23 01:11:06 2002

Hi,

I've been progressing in my quest to compile sbcl using clisp (2.28) as a
cross-compilation host -- it's picked up several SBCL bugs, and the
cross-compiler now runs to the extent of producing about 30 sbcl-style
fasls. However, I then get:

*** - handle_fault error2 ! address = 0xC0 not in [0x202AB000,0x20975920) !
SIGSEGV cannot be cured. Fault address = 0xC0.
Segmentation fault

and I'm wondering how to track the cause of this down. It's
reproduceable, and I'm happy to try things out, but because of the
nature of my "application" I haven't exactly got a small test case :-/

Any ideas?

Thanks,

Christophe
-- 
Jesus College, Cambridge, CB5 8BL                           +44 1223 510 299
http://www-jcsu.jesus.cam.ac.uk/~csr21/                  (defun pling-dollar 
(str schar arg) (first (last +))) (make-dispatch-macro-character #\! t)
(set-dispatch-macro-character #\! #\$ #'pling-dollar)

---

From: Sam Steingold <[email-masked]>
Date: Tue Apr 23 06:31:03 2002

> * In message <[email-masked]>
> * On the subject of "[clisp-list] What causes hard exits from clisp?"
> * Sent on Tue, 23 Apr 2002 09:10:19 +0100
> * Honorable Christophe Rhodes <[email-masked]> writes:
>

create the Makefile by giving makemake a "debug" argument, then edit
CFLAGS in Makefile to your taste, then remake.
then use gdb to debug the crash.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
<http://www.palestine-central.com/> <http://www.mideasttruth.com/>
Warning! Dates in calendar are closer than they appear!

---

From: Peter Wood <[email-masked]>
Date: Tue Apr 23 07:19:05 2002

Hi,

> nature of my "application" I haven't exactly got a small test case :-/

Can you compile your cross-compiler with debugging enabled? (either do
this by: "./makemake --with-xyz --with-etc debug > Makefile" |
alternatively, with a new cvs Clisp I think 'configure' also takes a
debug option)

Then try to compile SBCL - if Clisp segfaults again, you can run gdb
on the core file generated, and do a backtrace.  It may help.

You might want to first simply try compiling with the latest Clisp
cvs.

Regards,
Peter

---

From: Christophe Rhodes <[email-masked]>
Date: Tue Apr 23 07:24:06 2002

> then use gdb to debug the crash.

Thanks. I've attached a backtrace, in case it makes more sense to anyone
else -- it appears to be in garbage collection land, looking trying to
mark a subroutine? I'm afraid I know almost nothing of clisp internals
:-/

If people want to inspect things on the stack, do let me know -- I can
hold this gdb session; alternatively, it's fairly straightforward to
replicate the crash if you are willing to grab the sbcl source code...

Cheers,

Christophe

---

From: Sam Steingold <[email-masked]>
Date: Tue Apr 23 08:15:40 2002

> * In message <[email-masked]>
> * On the subject of "Re: [clisp-list] What causes hard exits from clisp?"
> * Sent on Tue, 23 Apr 2002 15:23:28 +0100

no need.  this is probably the usual "GC safety violation", i.e., a bug
that you will spot easily after I tell you were to look :-)

> replicate the crash if you are willing to grab the sbcl source code...

this appears to be a non-CVS CLISP (i.e., a release).
please get the CVS and try it.
while the CLISP build is raging on, please read
<http://clisp.cons.org/impnotes/gc-safety.html>

when you get the reproducible error, you will have to do the following:

walk around the stack (starting with, say, fehler_funname_source() in
your case and try `zout var` for various `object' vars, if it prints
fine, then it's okay, if it segfaults, it was corrupted - you have to
find the corrupted object, i.e., the one that was not saved on the STACK
before an un-safe call.)

thanks.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.palestine-central.com/> <http://www.mideasttruth.com/>
If a train station is a place where a train stops, what's a workstation?

---

From: Christophe Rhodes <[email-masked]>
Date: Tue Apr 23 08:54:03 2002

> that you will spot easily after I tell you were to look :-)

Erm, Ok... you have great faith in me.

> this appears to be a non-CVS CLISP (i.e., a release).

Yes.

> please get the CVS and try it.

I shall do, though I'm slightly reluctant in that I already have one or
two too many variables in my head... I might end up slightly confused
over this one -- I hope not.

> <http://clisp.cons.org/impnotes/gc-safety.html>

Right. So something wasn't pushSTACKed before calling some other
function? OK.

> find the corrupted object, i.e., the one that was not saved on the STACK

I'm slightly lost by this one -- do you mean the C stack or the clisp
STACK? An example would help. I tried zouting various things in various
frames and got nothing but segfaults, so it's possibly that I haven't
understood...

Cheers,

Christophe

---

From: Sam Steingold <[email-masked]>
Date: Tue Apr 23 10:20:08 2002

> * In message <[email-masked]>
> * Sent on Tue, 23 Apr 2002 16:52:51 +0100

yes.

> I'm slightly lost by this one -- do you mean the C stack or the clisp

walk around the C stack, checking that everything that should be on the
Lisp STACK is actually there.

> frames and got nothing but segfaults, so it's possibly that I haven't

sorry - after the memory got corrupted (which is indicated by the
segfault), you will get many random segfaults.

you have to walk (in the debugger) up to the point when CLISP crashes,
but stop before the actual segfault.
e.g., set a break in fehler_funname_source() - you probably get there
just this once (or maybe not - in that case you have to count the
fehler_funname_source hits and set a delayed break).

Note that zout is not GC-safe itself (especially when printing large
objects) - so calling it at a bad moment can get you in trouble.

No, this is not trivial, but you _can_ do it.

e.h.,

(gdb) break fehler_funname_source
(gdb) c
<stopped in fehler_funname_source, if you hit `finish', you get
the crash>
(gdb) n
<you are at the entrance to the fehler() call in fehler_funname_source() >
(gdb) zout caller
crash -- somewhere above `caller' was corrupted - restart and stop
earlier and see when `caller' gets corrupted...
otherwise,
(gdb) zout obj
crash -- somewhere above `obj' was corrupted - restart and stop
earlier and see when `obj' gets corrupted...
otherwise,
...

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.palestine-central.com/> <http://www.mideasttruth.com/>
A poet who reads his verse in public may have other nasty habits.

---

From: Christophe Rhodes <[email-masked]>
Date: Tue Apr 23 11:26:27 2002

> fehler_funname_source hits and set a delayed break).

Oh, I see. Ugggh. Um. The backtrace that I elided a couple of messages
upthread has upwards of 1000 frames. And there were quite a few
fehler_funname_source()s in there :-/ Looks like it's a long-term
project...

> objects) - so calling it at a bad moment can get you in trouble.

Ouch.

Um, ok. Thanks. I'll try to look at it.

Christophe

---

From: Sam Steingold <[email-masked]>
Date: Tue Apr 23 11:50:07 2002

> * In message <[email-masked]>
> * Sent on Tue, 23 Apr 2002 19:25:07 +0100

there is something wrong here.
fehler_funname_source cannot call fehler_funname_source.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.palestine-central.com/> <http://www.mideasttruth.com/>
Any programming language is at its best before it is implemented and used.

---

From: Christophe Rhodes <[email-masked]>
Date: Wed Apr 24 01:14:03 2002

> fehler_funname_source cannot call fehler_funname_source.

I've attached a slightly fuller backtrace from the CVS version of CLISP,
compiled with ./configure --with-debug, on the same application. It
crashes in the same place, from a user point of view, though the
small-numbered stack frames in this backtrace are slightly different in
the detail.

fehler_funname_source appears at frames 45, 80, and 115 of the attached
(hmm, 35 frames..., yes, also 150 and 185)

Cheers,

Christophe

---

From: Sam Steingold <[email-masked]>
Date: Wed Apr 24 06:24:02 2002

> * In message <[email-masked]>
> * Sent on Wed, 24 Apr 2002 09:13:26 +0100
> I've attached a slightly fuller backtrace from the CVS version of
> CLISP, compiled with ./configure --with-debug, on the same
> application. It crashes in the same place, from a user point of view,
> though the small-numbered stack frames in this backtrace are slightly
> different in the detail.
> 
> fehler_funname_source appears at frames 45, 80, and 115 of the
> attached (hmm, 35 frames..., yes, also 150 and 185)

fine.  please stop in an inner fehler_funname_source() and print all
arguments in all the 35 (hmm - please make it 70 - I want to know the
how the two adjacent periods differ) frames separating this
fehler_funname_source() and the one above (print objects with zout).

thanks.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.palestine-central.com/> <http://www.mideasttruth.com/>
There are two ways to write error-free programs; only the third one works.

---

From: Christophe Rhodes <[email-masked]>
Date: Wed May 1 10:30:05 2002

> fehler_funname_source() and the one above (print objects with zout).

Right. I've tried to look at this problem (not just the weird
fehler_funname_source thing, the whole sbcl-compile-under-clisp and
garbage-collection problem).

Firstly, let me say that the symptom discussed above seems to be an
artifact of having corrupted data; I _think_ what is going on is that
fehler_funname_source is trying to print

  *** - EVAL: <something> is not a function name

and failing because the <something> is a corrupt object; this then lands
it into confused recursive error territory. I could be wrong, of course,
but

  (gdb) frame 0
  #0  fehler_funname_source (caller=0x8247665, obj=0x8246379) at error.d:914
  914	  pushSTACK(obj); pushSTACK(caller);
  (gdb) zout caller
  ; EVAL
  $2 = (void *) 0x8247665

leads me to that diagnosis. (zouting obj, and most other things in the
backtrace at this point, reliably cause segfaults).

As for

  *** - handle_fault error2 ! address = 0xC0 not in [0x202AF000,0x20A2511C) !
  SIGSEGV cannot be cured. Fault address = 0xC0.
  Segmentation fault

well, I have tried wandering about the stack, stopping before this
segfault, but I confess that I haven't managed to find anything out.

However, sbcl's CVS is now in a state where it makes sense to give
instructions for replicating this failure, in case anyone else would
like to try their hand... it's still a little involved, so bear with me.

Firstly, to check out sbcl from CVS, do

  cvs -d:pserver:[email-masked]:/cvsroot/sbcl co sbcl

(with optional -z3 for compression, and possibly having done cvs login
first). To this, you will need to apply the attached patch (which
represent things that are needed to make some internals work under
clisp, but aren't understood sufficiently well to be committed to CVS),
with, from within the sbcl directory:

  patch -p0 < clisp-build.diff

[ feel free to look at it and explain the twisted reasoning behind the
  cmucl-derived code :-) ]

Then (and this bit assumes that you're running on x86/linux; if you're
not, this procedure is still possible, though more complex, so let me
know if you'd like the instructions):

user@host:path/to/sbcl$ sh ./make-config.sh
[ ... sets up relevant symlinks ... ]
user@host:path/to/sbcl$ SBCL_XC_HOST='path/to/clisp' sh ./make-host-1.sh
[ ... builds a cross-compiler, and dumps a header file needed to build
  the runtime ... this bit should go without errors ]
user@host:path/to/sbcl$ GNUMAKE=make sh ./make-target-1.sh
[ ... builds the runtime ... this bit should go with plenty of warnings
  but without errors ]
user@host:path/to/sbcl$ path/to/clisp
and then you get a clisp prompt
[1]> 
at which point you start pasting in forms from make-host-2.sh; normally
this would be run uninteractively, but you're going to need
interaction... so starting with the 
  (setf *print-level* 5 *print-length* 5)
  (load "src/cold/shared.lisp")
  (in-package "SB-COLD")
  ...
carry on pasting in forms one by one. After pasting in
  (load-or-cload-xcompiler #'host-load-stem)
clisp should load the cross-compiler fas files;
then, once you get down to
  (load "src/cold/compile-cold-sbcl.lisp")
the cross-compiler should start running, printing plenty of diagnostic
messages as is a verbose compiler's wont.

During compilation of src/code/pred.lisp, a continuable error is
signalled (for some reason that I haven't yet tracked down, SBCL wants
to intern "NIL-1" into the CL package; this is a bug in SBCL, but it can
be "continue"d from for now; then, about a minute later, during
compilation of src/code/array.lisp, I get this segfault the cause of
which I have not had very much success in finding :-(

I appreciate that this is the kind of bug report that is a nightmare
(it's not exactly a "small reproduceable test case", is it?), so I
understand if no-one gets round to tracking it down for a while... but
if anyone does have the time even to verify this, it would probably
help.

Many thanks,

Cheers,

Christophe
-- 
Jesus College, Cambridge, CB5 8BL                           +44 1223 510 299
http://www-jcsu.jesus.cam.ac.uk/~csr21/                  (defun pling-dollar 
(str schar arg) (first (last +))) (make-dispatch-macro-character #\! t)
(set-dispatch-macro-character #\! #\$ #'pling-dollar)
Index: src/code/defstruct.lisp
===================================================================
RCS file: /cvsroot/sbcl/sbcl/src/code/defstruct.lisp,v
retrieving revision 1.42
diff -u -r1.42 defstruct.lisp
--- src/code/defstruct.lisp	19 Apr 2002 10:51:27 -0000	1.42
+++ src/code/defstruct.lisp	1 May 2002 17:25:43 -0000
@@ -177,10 +177,18 @@
 
 ;;; Return the name of a defstruct slot as a symbol. We store it as a
 ;;; string to avoid creating lots of worthless symbols at load time.
+;;;
+;;; Also, we should be careful not to intern in the CL package, which
+;;; is undefined behaviour; in that case, we'll use SANE-PACKAGE
+;;; (relevant when we defstruct RESTART, where the accessor will be in
+;;; the CL package, but we don't want the lambda list symbols there
+;;; too...)
 (defun dsd-name (dsd)
   (intern (string (dsd-%name dsd))
 	  (if (dsd-accessor-name dsd)
-	      (symbol-package (dsd-accessor-name dsd))
+	      (if (eql (symbol-package (dsd-accessor-name dsd)) *cl-package*)
+		  (sane-package)
+		  (symbol-package (dsd-accessor-name dsd)))
 	      (sane-package))))
 
 ;;;; typed (non-class) structures
Index: src/compiler/assem.lisp
===================================================================
RCS file: /cvsroot/sbcl/sbcl/src/compiler/assem.lisp,v
retrieving revision 1.14
diff -u -r1.14 assem.lisp
--- src/compiler/assem.lisp	25 Apr 2002 19:26:55 -0000	1.14
+++ src/compiler/assem.lisp	1 May 2002 17:25:43 -0000
@@ -1294,7 +1294,11 @@
 	(i0 0))
     (flet ((frob (i0 i1)
 	     (when (< i0 i1)
-	       (funcall function (subseq buffer i0 i1)))))
+	       (let ((fp (fill-pointer buffer)))
+		 (when (< fp i1)
+		   (setf (fill-pointer buffer) i1))
+		 (funcall function (subseq buffer i0 i1))
+		 (setf (fill-pointer buffer) fp)))))
       (dolist (note (segment-annotations segment))
 	(when (filler-p note)
 	  (let ((i1 (filler-index note)))
Index: src/compiler/x86/parms.lisp
===================================================================
RCS file: /cvsroot/sbcl/sbcl/src/compiler/x86/parms.lisp,v
retrieving revision 1.27
diff -u -r1.27 parms.lisp
--- src/compiler/x86/parms.lisp	1 May 2002 13:56:51 -0000	1.27
+++ src/compiler/x86/parms.lisp	1 May 2002 17:25:44 -0000
@@ -44,8 +44,9 @@
 ;;;   bias and exponent min/max are not the same as shown in the 486 book.
 ;;;   They may be correct for how Python uses them.
 (defconstant single-float-bias 126)	; Intel says 127.
-(defconstant-eqx single-float-exponent-byte    (byte 8 23) #'equalp)
-(defconstant-eqx single-float-significand-byte (byte 23 0) #'equalp)
+(eval-when (:compile-toplevel :load-toplevel :execute)
+  (defparameter single-float-exponent-byte    (byte 8 23))
+  (defparameter single-float-significand-byte (byte 23 0)))
 ;;; comment from CMU CL:
 ;;;   The 486 book shows the exponent range -126 to +127. The Lisp
 ;;;   code that uses these values seems to want already biased numbers.
@@ -55,16 +56,18 @@
 (defconstant single-float-trapping-nan-bit (ash 1 22))
 
 (defconstant double-float-bias 1022)
-(defconstant-eqx double-float-exponent-byte    (byte 11 20) #'equalp)
-(defconstant-eqx double-float-significand-byte (byte 20 0)  #'equalp)
+(eval-when (:compile-toplevel :load-toplevel :execute)
+  (defparameter double-float-exponent-byte    (byte 11 20))
+  (defparameter double-float-significand-byte (byte 20 0)))
 (defconstant double-float-normal-exponent-min 1)
 (defconstant double-float-normal-exponent-max #x7FE)
 (defconstant double-float-hidden-bit (ash 1 20))
 (defconstant double-float-trapping-nan-bit (ash 1 19))
 
 (defconstant long-float-bias 16382)
-(defconstant-eqx long-float-exponent-byte    (byte 15 0) #'equalp)
-(defconstant-eqx long-float-significand-byte (byte 31 0) #'equalp)
+(eval-when (:compile-toplevel :load-toplevel :execute)
+  (defparameter long-float-exponent-byte    (byte 15 0))
+  (defparameter long-float-significand-byte (byte 31 0)))
 (defconstant long-float-normal-exponent-min 1)
 (defconstant long-float-normal-exponent-max #x7FFE)
 (defconstant long-float-hidden-bit (ash 1 31))		; actually not hidden
@@ -92,11 +95,12 @@
 (defconstant float-round-to-positive 2)
 (defconstant float-round-to-zero     3)
 
-(defconstant-eqx float-rounding-mode     (byte 2 10) #'equalp)
-(defconstant-eqx float-sticky-bits       (byte 6 16) #'equalp)
-(defconstant-eqx float-traps-byte        (byte 6  0) #'equalp)
-(defconstant-eqx float-exceptions-byte   (byte 6 16) #'equalp)
-(defconstant-eqx float-precision-control (byte 2  8) #'equalp)
+(eval-when (:compile-toplevel :load-toplevel :execute)
+  (defparameter float-rounding-mode     (byte 2 10))
+  (defparameter float-sticky-bits       (byte 6 16))
+  (defparameter float-traps-byte        (byte 6  0))
+  (defparameter float-exceptions-byte   (byte 6 16))
+  (defparameter float-precision-control (byte 2  8)))
 (defconstant float-fast-bit 0) ; no fast mode on x86
 
 ;;;; description of the target address space

---

From: Sam Steingold <[email-masked]>
Date: Fri May 3 11:29:51 2002

> * In message <[email-masked]>
> * Sent on Wed, 1 May 2002 18:27:57 +0100

did you try compiling CLISP with -DSAFETY=3?
this should catch many problems earlier and make debugging easier.
Please try it!
thanks.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.palestine-central.com/> <http://www.mideasttruth.com/>
cogito cogito ergo cogito sum

---

From: Christophe Rhodes <[email-masked]>
Date: Wed May 8 06:52:03 2002

On Wed, May 08, 2002 at 09:45:27AM -0400, Sam Steingold wrote:
> Bruno just fixed the bug what was the immediate cause of your crash.
> Please do "cvs up; ./configure --with-debug --build build-dir"
> and try again.

Thank you very much -- I shall give this a try (though probably not
until the weekend) and let you know what happens next...

Cheers,

Christophe

---

From: Sam Steingold <[email-masked]>
Date: Wed May 8 09:10:53 2002

thanks.

---

From: Christophe Rhodes <[email-masked]>
Date: Fri May 17 07:27:01 2002

> until the weekend) and let you know what happens next...

I've now done this... it still crashes in the same place, but now says:

*** - Lisp stack overflow. RESET

So I suspect I'm just going over some internal limit... is there a way
to rebuild with a larger stack? I assure you it's not unbounded
recursion :-)

Cheers,

Christophe

---

From: Sam Steingold <[email-masked]>
Date: Fri May 17 08:17:02 2002

> * In message <[email-masked]>
> * Sent on Fri, 17 May 2002 15:25:55 +0100

1. the previous bug was isolated to a 3 line function.
   please isolate this new bug too.

2. did you _really_ build --with-debug?
   Please do.

3. are you sure that the stack overflow is the first error message?
   please check the output _very_ carefully.

thanks.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Two wrongs don't make a right, but three rights make a left.