Subject: [clisp-list] DEF-LIB-CALL-OUT and dynamic FFI & library by name (was: critique of default-foreign-language)

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Wed Apr 24 01:41:04 2002

Hi,

> > DEf-LIB-CALL-OUT (Amiga) adds (:language :STDC) arbitrarily 
> for compatibility?
> > DEF-LIB-CALL-OUT (truly dynamic FFI, if I keep this name): 
> require :language
> 
> wouldn't these be gone when all the FFI goes dynamic?

This is a design question which I've not fully sorted out yet.

The thing to remember is that CLISP will have different sorts of external functions: those coming from modules (#<ADD-ON-SYSTEM-FUNCTION COMMON-LISP-USER::FOREIGN-ADDRESS-FUNCTION>), and those coming from new shared libraries (#<FOREIGN-FUNCTION "compress" #x100018F0>).

DEF-CALL-OUT could be made to macroexpand differently depending on whether some options are present or not, e.g. introduce a (:library foo) option.
My preference is to go for DEF-LIB-CALL-OUT or similar, because
 1) it's already there for the Amiga
 2) the macroexpansion for modules and for libraries is rather different.
 3) the exact format of foo in (:library foo) is obscure (see below point II).

There's also no DEF-LIB-CALL-IN.
However, one could have a few FOREIGN-LIBRARY-VARIABLE (one or two of of them are useful on AmigaOS).

AllegroCL has only a single ff:def-foreign-call. It doesn't even mention the library! Looks nice. But I guess this is because they add all the library's symbols to their lisp.run binary (like CMUCL did at the time I looked at it), which is not portable (think about MS-Windows), or perhaps because they internally sequentially walk through all open libraries, querying them for the symbol, which I won't accept (it's not deterministic from a single library provider's point of view: if I want "compress" from zlib, I shall get exactly that one, and such a query interface is not available on AmigaOS).

There are other, related issues:
I) exact form of the internal FOREIGN-LIBRARY, FOREIGN-ADDRESS-FUNCTION (if kept), FOREIGN-LIBRARY-FUNCTION (if introduced) etc. E.g.
Ib) FOREIGN-LIBRARY or some sort: maintain a list of open libraries in *FOREIGN-LIBRARIES* (allows for REOPEN-LIBRARY and library name synonyms, see point II below).
I'm currently thinking about (name-as-string foreign-pointer os-option ({known-functions}*). The CDR would be linked from all nicknames.

II) referring to libraries, especially naming by a nickname. LispWorks calls that modules. This may be useful because on all systems, the actual library name (and path) will vary widely. So if you want to say
(DEF-LIB-CALL-OUT compress "zlib" #) ; library by name
you need something which will map "zlib" to "zlib.dll", "libz.so" or
"cygz.dll" ; Cygwin name for zlib dll
or even "/usr/lib/libz.so" or even #P"/usr/lib/libz.so" or some such.

So far, my zlib compress examples works by evaluating the second argument (the library) to DEF-LIB-CALL-OUT:
(setq *zlib* (user::foreign-library *zlib-path*) ; store FOREIGN-POINTER object
(ffi:def-lib-call-out zlib-compress-string *zlib* ...)

When you think about how this would look like with a unified DEF-CALL-OUT with :library option, I don't like the idea to have to explain that the library option will be evaluated, while all others are not.
(DEF-CALL-OUT zlib-compress-string (:name "compress")
  (:library *zlib*) ; FOREIGN-POINTER (or possibly later: a string)
  (:library "zlib") ; this needs name synonyms to work
  (:language ...))
Using (:library "zlib") needs name synonyms (nicknames), as I don't want to see here and repeated for every function all of #+WIN32 "zlib.dll"
      ;; TODO how to deal with cygwin (#+UNIX on MS-Windows)? -- it wants "cygz.dll"
      #+UNIX "libz.so"
      #+AMIGA "zlib.library"

Sometimes, I feel I'm investing too much thought in rather low-level issues few people actually care about. Worse is better may just mean: the current scheme (referring to library by its base pointer) works, so it is enough.

In other words, I recognize that the unification costs me a lot of effort and grief and very little reward. DEF-LIB-CALL-OUT just works now (although I will revisit the macroexpansion pending point I above).

Another open problem: How to recogize cygwin? It appears as #+UNIX, but wants "cygz.dll" instead of "libz.so" which I suppose works across several systems (maybe only the dlopen() ones, not shl_load()).

So providing a user settable variable
(defvar *zlib-path*
  (or #+WIN32 "zlib.dll"
      ;; TODO how to deal with cygwin (#+UNIX on MS-Windows)? -- it wants "cygz.dll"
      #+UNIX "libz.so"
      #+AMIGA "zlib.library"
  )
  "Set this variable to point to the location of the zlib library
(libz.so or zlib.dll) on your system.")
is what people are used to. Cygwin users will have to overide this. HP-UX users probably also.
There's no need for me to imagine yet another half-working (i.e. broken) "try to find name automatically" heuristic.

Suggestions are, as always, welcome (maybe the best suggestion would be one about how I could express my ideas shortly :-),
	Jorg Hohle.

---

From: Peter Wood <[email-masked]>
Date: Wed Apr 24 03:51:06 2002

Hi,

On Wed, Apr 24, 2002 at 10:39:06AM +0200, Hoehle, Joerg-Cyril wrote:

> . Worse is better may just mean: the current scheme (referring to
> library by its base pointer) works, so it is enough.

Yes!

I agree with this.  I am happy with referring to the library by its
base pointer.  Let the users do their own macro stuff.

Regards,
Peter

---

From: Reini Urban <[email-masked]>
Date: Wed Apr 24 06:22:11 2002

"Hoehle, Joerg-Cyril" schrieb:
> AllegroCL has only a single ff:def-foreign-call. It doesn't even mention the 
> library! Looks nice. But I guess this is because they add all the library's 
> symbols to their lisp.run binary (like CMUCL did at the time I looked at it), 
> which is not portable (think about MS-Windows), or perhaps because they 
> internally sequentially walk through all open libraries, querying them for the 
> symbol, which I won't accept (it's not deterministic from a single library 
> provider's point of view: if I want "compress" from zlib, I shall get exactly 
> that one, and such a query interface is not available on AmigaOS).

hmm. corman lisp reinit's all foreign variables and function pointers from all
library names 
stored in the image on (load-image). there's no other way, and it's
deterministic if no 
libs or libpaths are broken. (like current path overrides) 
corman even uses foreign win32 functions during startup.

> II) referring to libraries, especially naming by a nickname. LispWorks calls 
> that modules. This may be useful because on all systems, the actual library name 
> (and path) will vary widely. So if you want to say

same problem for the UFFI.
but I would suggest no nickname, only the common library substring.
eg. "z" for libz.so, cygz.dll, zlib.dll. or "jpeg" for /usr/lib/libjpeg.so
but one should be able to override the library name by explicitly giving the
full name.
("libz" or "/usr/lib/libz.so")
I think ACL does it this way. cygwin's dlopen() or the gcc linker for sure.

> Another open problem: How to recogize cygwin? It appears as #+UNIX, 
> but wants "cygz.dll" instead of > "libz.so" which I suppose works across several 
> systems (maybe only the dlopen() ones, not shl_load()).

So we must add a :CYGWIN feature then.

> maybe the best suggestion would be one about how I could express my ideas shortly :-)

Please not. I immensely prefer your long explanations.

-- 
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

---

From: Sam Steingold <[email-masked]>
Date: Wed Apr 24 06:40:09 2002

> * In message <[email-masked]>
> * On the subject of "[clisp-list] DEF-LIB-CALL-OUT and dynamic FFI & library by name (was: critique of default-foreign-language)"
> * Sent on Wed, 24 Apr 2002 10:39:06 +0200
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>
> The thing to remember is that CLISP will have different sorts of
> external functions: those coming from modules
> (#<ADD-ON-SYSTEM-FUNCTION COMMON-LISP-USER::FOREIGN-ADDRESS-FUNCTION>), 
> and those coming from new shared libraries
> (#<FOREIGN-FUNCTION "compress" #x100018F0>).

are these truly different objects?
are you sure both are truly needed?
I thought that the DLL FFI will make the current module facility DLL
based too, i.e., while now modules create their own lisp.run by linking
lisp.a with their *.o files, the new modules will compile to *.so
(*.dll) and load them from the lisp code.

> DEF-CALL-OUT could be made to macroexpand differently depending on
> whether some options are present or not, e.g. introduce a (:library
> foo) option.
yes!

>  3) the exact format of foo in (:library foo) is obscure (see below point II).

I abhor the "many names" approach.
Inter alia it makes it harder to write macros which expand to other
macros.
I prefer few calls taking many arguments to many calls taking few
arguments.  Note that CL has one DEFSTRUCT instead of
DEFSTRUCT-WITH-BOA, DEFSTRUCT-WITH-PRINT-FUNCTION,
DEFSTRUCT-WITH-METACLASS &c.
Similarly, I want DEF-CALL-OUT with many options, some of which may be
required, instead of DEF-C-CALL-OUT, DEF-LIB-CALL-OUT,
DEF-LIB-C-CALL-OUT, DEF-LIB-STDCALL-CALL-OUT &c.

> Sometimes, I feel I'm investing too much thought in rather low-level
> issues few people actually care about. Worse is better may just mean:
> the current scheme (referring to library by its base pointer) works,
> so it is enough.
fine.

> Another open problem: How to recogize cygwin?

I can add :CYGWIN to *FEATURES* if that is what people want.

>   this. HP-UX users probably also.

is this the only place where #+CYGWIN is useful?
then it's not worth it!

> There's no need for me to imagine yet another half-working
> (i.e. broken) "try to find name automatically" heuristic.
yes.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.palestine-central.com/> <http://www.mideasttruth.com/>
A man paints with his brains and not with his hands.

---

From: Peter Wood <[email-masked]>
Date: Wed Apr 24 07:52:04 2002

Hi,

> (*.dll) and load them from the lisp code.

You don't need this at all with the new FFI.  Why would you want to
compile a monolithic (for example) linux.lisp when you can dynamically
(and selectively) define the foreign functions which you will be
needing.  Why do you need a .so when you can define your functions in
_lisp_?  What on earth do you need to fiddle with a module .so when
you have /libc/libc.so.6?

There are currently two (IMO) quite different ways of making use of
Clisp's module facilty:

1) compiling ffi definitions to a .c file (stage 1) which gets
compiled (usually through clisp-link) to .o (stage 2) and linked to a
new lisp.run (stage 3)

2) hand coded .d modules which can be integrated into the clisp build
process by adding them to modules.h .  dynload.d and ffimext.d are
examples.

With the new FFI, the need for (1) above falls away.  It will still be
useful to be able to do (2).  And the ability to produce .c files
(stage 1) in (1) above is still useful in helping to automate (2).

You don't need to compile modules to .so files.  These files already
exist on the system, and you can dynamically create functions _in the
running lisp_ which call into them.  There will normally be no need
for the user to compile to c files at all.

Regards,
Peter

---

From: Peter Wood <[email-masked]>
Date: Wed Apr 24 08:03:11 2002

Hi,
On Wed, Apr 24, 2002 at 01:21:29PM +0000, Reini Urban wrote:

>  hmm. corman lisp reinit's all foreign variables and function
> pointers from all library names stored in the image on
> (load-image). there's no other way, and it's deterministic if no
> libs or libpaths are broken. (like current path overrides) corman
> even uses foreign win32 functions during startup.

Corman Lisp only has to deal with win32.  It gets hairier when you
have to deal with a number of different platforms, some of which may
be inconsistent in where they put libs.

I think reinit should be up to the user.  Its an issue which is far too
system and machine-instance specific to reliably automate portably.

zlib can be in *lots* of different places.  In fact it can be anywhere.

Regards,
Peter

---

From: Sam Steingold <[email-masked]>
Date: Wed Apr 24 08:13:28 2002

> * In message <[email-masked]>
> * On the subject of "Re: [clisp-list] DEF-LIB-CALL-OUT and dynamic FFI & library by name (was: critique of default-foreign-language)"
> * Sent on Wed, 24 Apr 2002 16:36:20 +0200
> * Honorable Peter Wood <[email-masked]> writes:
>

I was talking about the case when _you_ write something in C _yourself_
and want to call that from Lisp.  when you are using a pre-built DLL,
there is obviously no C from our side involved.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.palestine-central.com/> <http://www.mideasttruth.com/>
PI seconds is a nanocentury

---

From: Peter Wood <[email-masked]>
Date: Wed Apr 24 10:42:08 2002

> there is obviously no C from our side involved.

Hi

If you are already writing it in C, then you can just compile it as a
shared object library and write a lisp-side defun:

/*hello.c*/
#include <stdio.h>

int greet (char *who) {
  int n;
  n = printf("Hello, %s!\n", who);
  return (n);
}
/*end*/

compile:

bash-2.04$ gcc -fPIC -c hello.c
bash-2.04$ gcc -shared -Wl,-soname,libhello.so.l -o libhello.so.1.0.1 hello.o -lc

bash-2.04$ ./lisp.run -q -M lispinit.mem 
;; Loading file /home/prw/.clisprc ...
;; Loading of file /home/prw/.clisprc is finished.
[1]> (load #"/home/prw/lisp/Linux/libc.lisp")
;; Loading file /home/prw/lisp/Linux/libc.lisp ...
;; Loading of file /home/prw/lisp/Linux/libc.lisp is finished.
T
[2]> (setf c::*base* (user::foreign-library "/home/prw/libhello.so.1.0.1"))
#<FOREIGN-POINTER #x08266A10>
[3]> (c::deffun "greet" ((who ffi:c-string)) (ffi:int))
GREET
[4]> (greet "Sam")
Hello, Sam!
12
[5]> (greet "clisp-list")
Hello, clisp-list!
19
[6]> (quit)

Or did you mean something completely different?

Regards,
Peter

---

From: Sam Steingold <[email-masked]>
Date: Wed Apr 24 12:07:45 2002

> * In message <[email-masked]>
> * Sent on Wed, 24 Apr 2002 19:25:47 +0200

yes, that's exactly what I meant.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.palestine-central.com/> <http://www.mideasttruth.com/>
will write code that writes code that writes code for food

---

From: Marco Antoniotti <[email-masked]>
Date: Thu Apr 25 06:37:02 2002

> From: Peter Wood <[email-masked]>
> Content-Type: text/plain; charset=us-ascii
> Content-Disposition: inline
> User-Agent: Mutt/1.2.5i
> Sender: [email-masked]
> X-BeenThere: [email-masked]
> X-Mailman-Version: 2.0.9-sf.net
> Precedence: bulk
> List-Help: <mailto:[email-masked]?subject=help>
> List-Post: <mailto:[email-masked]>
> List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/clisp-list>,
> 	<mailto:[email-masked]?subject=subscribe>
> List-Id: CLISP user discussion <clisp-list.lists.sourceforge.net>
> List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/clisp-list>,
> 	<mailto:[email-masked]?subject=unsubscribe>
> List-Archive: <http://www.geocrawler.com/redir-sf.php3?list=clisp-list>
> X-Original-Date: Wed, 24 Apr 2002 19:25:47 +0200
> Date: Wed, 24 Apr 2002 19:25:47 +0200
> 
                        ^^^^

Not to be too picky, but could the FFI stuff bu put into a separate package?

Cheers

-- 
Marco Antoniotti ========================================================
NYU Courant Bioinformatics Group        tel. +1 - 212 - 998 3488
719 Broadway 12th Floor                 fax  +1 - 212 - 995 4122
New York, NY 10003, USA                 http://bioinformatics.cat.nyu.edu
                    "Hello New York! We'll do what we can!"
                           Bill Murray in `Ghostbusters'.

---

From: Peter Wood <[email-masked]>
Date: Thu Apr 25 07:17:01 2002

Hi,

> Not to be too picky, but could the FFI stuff bu put into a separate package?

My understanding is that some things got placed in "USER" because some
aspects of module support are broken.  (I don't know if it's fixed
yet)

Specifically, there were problems with package locking, and
initialisation of modules.

I believe they will go in "FFI" when things settle down a bit.

Regards,
Peter