Subject: [clisp-list] module support (was: DEF-LIB-CALL-OUT and dynamic FFI & library b y name)

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Thu Apr 25 08:19:40 2002

Peter,

> My understanding is that some things got placed in "USER" because some
> aspects of module support are broken. 
> Specifically, there were problems with package locking, and
> initialisation of modules.
Thanks for the excellent summary.
1. package-locking
2. package existance.

I jump in to beg for ideas:

1) Maybe package-locking can be deferred until after modules are loaded?

I consider the impossibility to add new functions in low-level module code (programmer level) because of package-locks (a user-level protection) an unacceptable show-stopper. SYS, FFI or EXT are IMHO the right places for small extensions. Predumping an image with additional packages defined, then run lisp.exe|run is too much overhead on users.

Easy to fix would be:
For LISPFUN only: package is known, unlock it as needed. Don't remember lock state, package will remain unlocked (probably useful for additional Lisp code which will define even more functions).

That doesn't help for DEF-C-CALL-OUT or DEF-CALL-IN (doesn't define LISPFUN, i.e. the module_subr_tab[], only objects/symbols, module_obect_tab[]).

It helps for typical hand-written modules with functions and objects. Since the functions are defined first, the package is unlocked, so later objects can be read/added.

That is, it would help
 o dynload.d
 o ffimext.d
but not linux.lisp
A "works for me" patch :-(

2) maybe a missing package should be created empty?
  Again, only posible for LISPFUN things.
I wonder why this was not implemented back then. It would have been trivial to do, so I suspect good reasons (despite what Napoleon said). Maybe it would break Lisp code which does:
(unless (find-package "LDAP")
  ;; packge is not known, load and init Lisp-level stuff
  (make-package "LDAP" :use #)
  (load #)
  ...)
which was quite common before DEFPACKAGE was common.

Any ideas?
	Jorg Hohle.

---

From: Peter Wood <[email-masked]>
Date: Thu Apr 25 10:32:04 2002

Hi,
On Thu, Apr 25, 2002 at 05:17:45PM +0200, Hoehle, Joerg-Cyril wrote:

> I consider the impossibility to add new functions in low-level
> module code (programmer level) because of package-locks (a
> user-level protection) an unacceptable show-stopper. SYS, FFI or EXT
> are IMHO the right places for small extensions. Predumping an image
> with additional packages defined, then run lisp.exe|run is too much
> overhead on users.

clisp-link has a variable TO_PRELOAD which works around the problem by
automatically dumping the image with the necessary packages defined
(You set TO_PRELOAD to the name of a file which defines the packages).

This is poorly/inaccurately documented in the impnotes, and I sent Sam
a patch which improves the docs slightly, but the online impnotes has
not been regenerated yet.

I "discovered" TO_PRELOAD after your post that you remembered "a
workaround" which required the user to first dump an image with the
packages defined.  I had read and reread the ffi section of impnotes
_many_ times, without realising that TO_PRELOAD was what I was looking
for.  (Mainly because it is inaccurately and misleadingly documented)

I wouldn't worry about fixing _this_ aspect of the bug properly.  It
has been in Clisp for many years, apparently without anyone the wiser.

> A "works for me" patch :-(

A "works for me" patch is fine.  Anyway, it will help anyone who
writes or wants to write modules by hand.  I hope to get around to
hand-written modules real soon now (TM) :-)

> which was quite common before DEFPACKAGE was common.

I have been wondering why missing packages aren't just created.

I have to ask: What did Napoleon say???

Regards,
Peter

---

From: Sam Steingold <[email-masked]>
Date: Thu Apr 25 10:47:04 2002

> * In message <[email-masked]>
> * On the subject of "[clisp-list] module support (was: DEF-LIB-CALL-OUT and dynamic FFI & library b y name)"
> * Sent on Thu, 25 Apr 2002 17:17:45 +0200
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>
> I consider the impossibility to add new functions in low-level module
> code (programmer level) because of package-locks (a user-level
> protection) an unacceptable show-stopper.

this is a non-existent problem.
At the LISP level you have WITHOUT-PACKAGE-LOCK.
At the C level you can lock and unlock packages as easily (see package.d
for examples.)

Just do the right thing and you should be safe.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.palestine-central.com/> <http://www.mideasttruth.com/>
As a computer, I find your faith in technology amusing.

---

From: Peter Wood <[email-masked]>
Date: Thu Apr 25 12:33:57 2002

Hi,
> At the C level you can lock and unlock packages as easily (see package.d

OK. Now I'm confused.  With a recent CVS, I was trying to test how to
use mark_pack_unlocked, and as a first step recompiled dynload.d and
ffimext.d with the original:

#define ffi  "USER" /*"CL-USER*/

changed to #define ffi "FFI"

In an attempt to recreate the problem, so I could experiment with
using mark_pack_unlocked.  But it all compiles fine, with no
complaints about locked packages!!!! And with
#'foreign-address-function, etc in "FFI".

What the fuck is going on?

> Just do the right thing and you should be safe.

Yeah! Now why couldn't _I_ have thought of that?

> As a computer, I find your faith in technology amusing.

War is a bug.  Where can I report it?  

:-/

Regards,
Peter

---

From: Sam Steingold <[email-masked]>
Date: Thu Apr 25 13:44:39 2002

> * In message <[email-masked]>
> * On the subject of "Re: [clisp-list] module support (was: DEF-LIB-CALL-OUT and dynamic FFI & library b y name)"
> * Sent on Thu, 25 Apr 2002 21:01:30 +0200
> * Honorable Peter Wood <[email-masked]> writes:
>

I am afraid I do not understand either what you are doing nor what you
are trying to accomplish.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.palestine-central.com/> <http://www.mideasttruth.com/>
Trespassers will be shot.  Survivors will be SHOT AGAIN!

---

From: Peter Wood <[email-masked]>
Date: Thu Apr 25 23:27:01 2002

> are trying to accomplish.

I am tempted to just say, forget it!  Instead, if you can be bothered
to read it, here's an executive summary, with references:

1) When building external modules with clisp-link, with a def-call-in,
Clisp segfaulted and reported bizarre things about terminal streams.

2) It turned out this was a result of (a combination)
   1) Not using TO_PRELOAD to predefine needed packages
   2) Early initialisation error

see: http://www.geocrawler.com/lists/3/SourceForge/1124/225/8282050/

3) Joerg reported a problem about packages being locked, at the same
time as he posted dynload.d to the list

see: http://www.geocrawler.com/lists/3/SourceForge/1124/200/8302221/

You can't see the attachments on geocrawler, but if you have dynload.d
or the later ffimext.d, you will see that there is a comment to the
effect that module support is broken, due to locking, and that is why
'#define ffi "USER"' was done.

*I assumed that this was correct.*

When you (Sam) wrote that this was a non-existent problem 

see: http://www.geocrawler.com/lists/3/SourceForge/1124/0/8496506/

I decided to reconstruct the problem by changing '#define ffi "USER"'
to '#define ffi "FFI"'.  I believed that when I recompiled after this
change, that I would get an error about locked packages. I didn't.
Therefore my confusion.

Joerg, is there really a problem with package locking?  If not, why is
ffi defined to "USER" in dynload.d?

Sam, please don't cc to me.  I am on the list, and get all mail sent
to the list.  It's annoying getting duplicates.

Regards,
Peter