Subject: [clisp-list] Re: Linkkit breakage in 2.31.

From: Kaz Kylheku <[email-masked]>
Date: Fri Dec 5 16:24:01 2003

It appears that CLISP 2.31 breaks the linkkit interface.

Firstly, the ``clisp.h'' header contains:

   #include "constobj.c"

but the install scriptology does not copy this file into the linkkit
directory, and so #include "clisp.h" blows up!

Secondly, the FFI translator puts out ill-formed C, so even if you work
around the first problem you run into others.

Namely the C module that is produced from the FFI definitions makes
references to function addresses without providing extern declarations
for the functions! E.g.

  (def-call-out rmdir 
    (:language :stdc)
    (:arguments (path c-string))
    (:return-type int))

in the .lisp module turns into 

  register_foreign_function((void *)&rmdir,"rmdir",1024);

in the .c file with no prior declaration of rmdir!

A good test case for the link kit system is this: download Meta-CVS
1.0.9, unpack it, go to the src directory and type ``./install.sh
/usr/local''. :)

---

From: Sam Steingold <[email-masked]>
Date: Fri Dec 5 18:59:01 2003

> * Kaz Kylheku <[email-masked]> [2003-12-05 16:23:20 -0800]:
>

this has been fixed on 2003-09-09.

> Secondly, the FFI translator puts out ill-formed C, so even if you
> work around the first problem you run into others.

see NEWS:

* FFI does not output extern variable and function declarations unless
  you set FFI:*OUTPUT-C-VARIABLES* and/or FFI:*OUTPUT-C-FUNCTIONS* to T.
  Please use FFI:C-LINES to include the appropriate headers instead.
  See <http://clisp.cons.org/impnotes.html#ffi-extern-output> for details.

the rationale is that there is no way to reliably output extern
declaration that would be compatible with the system headers (which
should be included anyway!), so why not rely the headers instead?

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Daddy, why doesn't this magnet pick up this floppy disk?

---

From: Kaz Kylheku <[email-masked]>
Date: Fri Dec 5 23:45:01 2003

> should be included anyway!), so why not rely the headers instead?

This is a silly rationale, because all that is done with the declared
functions is to take their address and cast it to (void *). The functions
are not actually called in the generated C module, so the type signature
does not come into question.  The declarations, effectively, merely
assert the existence of these functions as entry points in the address
space.  Thus it buys you nothing that the declarations come from a more
legitimate source. It would matter in some fictitious C implementation
whose function pointers carry type information that is preserved under
pointer casts and checked at call time.

Moreover, you can include all the right headers, yet completely lie in
the DEF-CALL-OUT form to generate an incompatible call. The C compiler
won't squawk at all!

For error checking, the most useful behavior would be to have both:
enable the generation of function declarations *and* use C-LINES to
include the headers. This way the FFI-generated declarations could be
type checked against the original declarations from the external headers.
The C compiler would catch, say, the Lisp side using ``int'' for
a mode_t parameter which is typedef'd to ``long'' in the system header.

The way it is, the new behavior is just a misfeature that breaks existing
projects based on CLISP. Their users cannot independently upgrade CLISP
without obtaining fixed versions of these projects. I'm not saying that
backward compatibility is some supreme value that must be preserved at any
cost, only that it should be broken when there is no other reasonable
way to fix some actual problem.

---

From: Kaz Kylheku <[email-masked]>
Date: Sat Dec 6 00:13:01 2003

> this has been fixed on 2003-09-09.

Maybe it would be a good idea to go through a pre-release stage. CLISP
is a big and complex platform project, and you guys can't test everything!
Help is needed from the broader community of people who use CLISP.
The built-in test suite is very good, but it won't catch peripheral
things like this.

Of course anyone can get the bleeding edge from CVS at any time, but it's
helpful to have an official alpha release program which tells people
``hey, wake up! We think we have something release-worthy here, come
and try it''.

Before releasing 2.32, how about a 2.31.95 test prerelease, with binaries,
tarballs for download, and all that fanfare. If problems are found,
you can put out a 2.31.96, and so on. You have all the way up to
2.31.99, which you should never reach. :) When no more problem reports
come back from the field, you put out the latest candidate as 2.32.
This is for instance how Ulrich Drepper puts out glibc2.

Otherwise you are just relying on people who are fetching the latest
CVS, who are typically just people who are hacking on CLISP, or are
looking to have some particular problem fixed.

Test releases are important for people who have dependencies on the
project, so they can verify that nothing breaks, but who otherwise
don't want to sit there doing ``cvs up'' everyday to rebuild it.

Moreover, the test releases are clearly identified, so the third-level
users who use applications based on the platform won't bother the
application guys to make their stuff work with broken test releases!
If a user comes and says ``I can't get FooProver 3.9 to work with
CLISP 2.33.93'' the developer can just say ``That's a test release
with a known problem affecting FooProver that did not exist in 2.33 and
is already fixed in 2.33.94. Stick with 2.33 or wait for 2.34.''

---

From: Sam Steingold <[email-masked]>
Date: Sat Dec 6 07:48:01 2003

> * Kaz Kylheku <[email-masked]> [2003-12-05 23:44:23 -0800]:
>
> This is a silly rationale

well, people include the headers all the time and complain that CC
cannot compile the FFI C code.

> a mode_t parameter which is typedef'd to ``long'' in the system header.

you can enable that too.
there have been an extensive discussion on the list on this matter.

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Old Age Comes at a Bad Time.

---

From: Sam Steingold <[email-masked]>
Date: Sat Dec 6 07:50:01 2003

> * Kaz Kylheku <[email-masked]> [2003-12-06 00:12:29 -0800]:
>

the imminence of a an impending release is announced here.

> Before releasing 2.32, how about a 2.31.95 test prerelease, with
> binaries, tarballs for download, and all that fanfare. If problems are
> found, you can put out a 2.31.96, and so on. You have all the way up
> to 2.31.99, which you should never reach. :) When no more problem
> reports come back from the field, you put out the latest candidate as
> 2.32.

this is a great idea!

> This is for instance how Ulrich Drepper puts out glibc2.

do you volunteer to build all the binaries &c?

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Profanity is the one language all programmers know best.

---

From: Douglas Philips <[email-masked]>
Date: Sat Dec 6 07:55:02 2003

> do you volunteer to build all the binaries &c?

I volunteer to produce the OS X binaries if no one else wants to. Also 
the Redhat 8 and YellowDog 3 binaries. All this assuming that my 
hardware doesn't break.

BTW: As of the anon-cvs head last night, all three of those (still 
running Jaguar, haven't upgraded to Panther yet) built and make check'd 
cleanly. I haven't run my own tests against 'em yet though. ;-)

<D\'gou

---

From: Sam Steingold <[email-masked]>
Date: Sat Dec 6 09:04:01 2003

> Maybe it would be a good idea to go through a pre-release stage.

Another options is to make monthly bug-fix releases.
This August John Hinsdale volunteered to maintain a bug-fix CVS branch,
so maybe now is a good moment to take him up on his word :-)

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Booze is the answer. I can't remember the question.