Subject: [clisp-list] RE: Clisp FFI

From: Kick Damien-DKICK1 <[email-masked]>
Date: Mon Nov 4 15:28:02 2002

Kaz Kylheku [[email-masked]] wrote:

> On Mon, 4 Nov 2002, Kick Damien-DKICK1 wrote:
> 
> >     Compiling file /home/kick/tmp/call-f00f.lisp ...
> >     WARNING:
> >     SETQ(FFI::*FOREIGN-LANGUAGE*): #<PACKAGE FFI> is locked
> >     Ignore the lock and proceed
> >     WARNING in (DEFAULT-FOREIGN-LANGUAGE :STDC)-4 in line 4 :
> >     SETQ: assignment to the internal special symbol
FFI::*FOREIGN-LANGUAGE*
>
> That is annoying.  [...] What I do is [...]

If you wouldn't mind sending an example, I wouldn't mind receiving one
<smile>.

> >     Wrote file /home/kick/tmp/call-f00f.fas
> >     Wrote file /home/kick/tmp/call-f00f.c
> >     0 errors, 1 warning
> >     Bye.
> >     [kick@login0 ~/tmp]% $DIR/clisp-link add-module-set f00f $DIR/base
base+f00f
> >     make: Warning: File `Makefile' has modification time 44 s in the
future
> NFS?

Yes.  I'm not sure if that is the cause of the problem, though.

> >     modules.o(.data+0x30): undefined reference to
`module__f00f_code__subr_tab'

> This problem is caused by a disagreement within the link.sh script
> about what your module is called.  [...] the module name that you
> configure in your link.sh must agree with the name of your .lisp
> file.

Thank you.  This did the trick.

    [kick@login0 ~/tmp]% cat f00f/link.sh 
    file_list=''
    mod_list=''
    if test -r f00f-code.c; then
      file_list="$file_list"' f00f-code.o'
      mod_list="$mod_list"' call_f00f'
    fi
    make clisp-module CC="${CC}" CFLAGS="${CFLAGS}"
INCLUDES="$absolute_linkkitdir"
    NEW_FILES="$file_list call-f00f.o"
    NEW_LIBS="$file_list call-f00f.o"
    NEW_MODULES="$mod_list"
    TO_LOAD=''
    [kick@login0 ~/tmp]% 

> >     base+f00f/   call-f00f.fas  call-f00f.lisp  f00f-code.c
> >     call-f00f.c  call-f00f.lib  f00f/	    f00f-code.o
>
> [...] The end result should be that modules.o makes references to
> module__call_f00f__subr_tab and so forth, and these references are
> satisfied by definitions inside call-f00f.o.

I had to compile "call-f00f.c" from the shell.  For some reason, I
would've expected that this would have been done as part of
'clisp-link', yes?  Not that it is difficult to do compile
"call-f00f.c" but I'm just curious.

> Your module's name is not derived from ``f00f-code''.  That is just
> the name of a source file containing some functions to be compiled
> and linked into the Lisp executable.  CLISP doesn't care about the
> name of this at all, nor does it even require it to exist.

I'm glad you mentioned how CLISP (why is it in uppercase? is it an
acronym?  is it "(quote clisp) => CLISP"?) does not require
"f00f-code.c" to exist at all.  This exercise is really just a first
step in my attempting to produce a CLISP binding for Don Libe's Expect
library.  I've been hoping that it is somehow possible to simply
supply "libexpect.a".  Is there somewhere I can look for more
documentation than what I have found in the implementation notes
<http://clisp.sourceforge.net/impnotes.html#modules>, possibly with an
example of doing just this (creating an external module from a
library)?  Are there any gotchas one might think would be lurking for
me if I try to do this?

--
Damien Kick

---

From: Kaz Kylheku <[email-masked]>
Date: Mon Nov 4 18:38:01 2002

On Mon, 4 Nov 2002, Kick Damien-DKICK1 wrote:

> Date: Mon, 4 Nov 2002 17:27:20 -0600 
> From: Kick Damien-DKICK1 <[email-masked]>
> To: 'Kaz Kylheku' <[email-masked]>,
>      Kick Damien-DKICK1 <[email-masked]>
> Cc: [email-masked]
> Subject: RE: Clisp FFI
> 

 (defpackage :my-ffi-glue-package
   (:use :ffi :cl)
   blah blah
   (shadow :def-c-call-out))

 (in-package :my-ffi-glue-package)

 (defmacro def-c-call-out (&rest forms) 
   `(def-call-out (:language :stdc) ,@forms))

 (def-c-call-out ...)

>     TO_LOAD=''

TO_LOAD should be filled in:

    TO_LOAD='call-foof'

This will load call-foof.fas (or .lisp if it happens to not be
compiled). It's necessary, because you need that .fas in the
lispinit.mem of the generated linking set, just as much as you need
your .o files to go into lisp.run. Without loading the Lisp part of it,
you don't have the Lisp side of the glue.

Here is a gotcha I ran into. If you are using DEF-C-CALL-IN, and are
writing Lisp callbacks in your own package (rather than, say,
COMMON-LISP-USER), you have to use TO_PRELOAD as well.

Without TO_PRELOAD, the build blows up, because a lisp.run is generated
from a .o file that contains a table of symbolic references to 
the symbols in your package. But the problem is that the lisp.run is
then used in conjunction with the base lispinit.mem, which knows
nothing about the package.

With TO_PRELOAD, the image generation is split into an extra step. First,
a new lispinit.mem is generated containing your preloaded Lisp code
where you can define your packages. Then the new lisp.run is formed
with your .o modules. And lastly, this lisp.run is used with the
preloaded lispinit.mem, to take care of your TO_LOAD stuff, and
generate the final lispinit.mem.

> "call-f00f.c" but I'm just curious.

That is correct. You put it into the Makefile.

> supply "libexpect.a".

Yes it is. You just put that into the NEW_LIBS line.  If it's in the
linker path, you can just put in ``-lexpect''.

  Is there somewhere I can look for more
> library)?  Are there any gotchas one might think would be lurking for

The gotchas are that you have to replicate any struct definitions and
whatnot using the CLISP FFI language. If the library interfaces change,
then you have to make the dependent changes in your FFI definition.

It may be helpful to actually have a .c module that separates you from
such a dependency.

As an example, in Meta-CVS I have wrappers for opendir, readdir and
closedir. But the dirent structure has a binary structure that is
different on different platforms. I just care about the d_name and
d_inode members. So the approach is to define a fake dirent structure,
and implement a fake readdir, which I call impl_readdir. This function
calls the real readdir and then copies the members from the real struct
to the fake struct. 

Smoething like libexpect is less of a problem this way; a given version
of it is the same everywhere.

There may be other reasons to create an extra layer. You might want
just a simplified subset of the functionality. You can get a simplified
subset by making a lot of FFI, and then writing Lisp code on top of
that to get the simplification. But depending on the exact
circumstances, that could be more work than just writing a simplified
layer in C, and then targetting that with FFI.

---

From: Sam Steingold <[email-masked]>
Date: Mon Nov 4 20:14:05 2002

> * In message <[email-masked]>
> * On the subject of "[clisp-list] Re: Clisp FFI"
> * Sent on Mon, 4 Nov 2002 18:37:23 -0800 (PST)
> * Honorable Kaz Kylheku <[email-masked]> writes:
>

please do not do this.
use CVS or apply the appended patch.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
The program isn't debugged until the last user is dead.

Index: compiler.lisp
===================================================================
RCS file: /cvsroot/clisp/clisp/src/compiler.lisp,v
retrieving revision 1.98
retrieving revision 1.99
diff -u -w -b -u -b -w -i -B -r1.98 -r1.99
--- compiler.lisp	14 Sep 2002 22:14:24 -0000	1.98
+++ compiler.lisp	18 Sep 2002 17:43:06 -0000	1.99
@@ -3013,6 +3013,7 @@
        (HANDLER-BIND . c-HANDLER-BIND)
        (SYS::%HANDLER-BIND . c-HANDLER-BIND)
        (SYS::CONSTANT-EQL . c-CONSTANT-EQL)
+       (WITHOUT-PACKAGE-LOCK . c-WITHOUT-PACKAGE-LOCK)
        ;; Inline-compiled functions:
        (FUNCALL . c-FUNCALL)
        (SYS::%FUNCALL . c-FUNCALL)
@@ -5668,8 +5669,10 @@
     (let ((s (car l)))
       (when (and (symbolp s) (venv-search-macro s)) (return t)))))
 
+(defvar *compiler-unlocked-packages* nil)
 (defun set-check-lock (caller symbol)
-  (when (symbol-value-lock symbol)
+  (when (and (not (memq (symbol-package symbol) *compiler-unlocked-packages*))
+             (symbol-value-lock symbol))
     (c-warn (TEXT "~S: assignment to the internal special symbol ~S")
             caller symbol)))
 
@@ -7054,6 +7057,14 @@
       (c-form `(SYS::LOOSE-CONSTANT-EQL ,@form23))
       (c-form `(EQL ,@form23)))))
 
+;; compile (without-package-lock (packages) ...)
+(defun c-WITHOUT-PACKAGE-LOCK  (&optional (c #'c-form))
+  (test-list *form* 1)
+  (let ((*compiler-unlocked-packages*
+         (append (mapcar #'find-package (second *form*))
+                 *compiler-unlocked-packages*)))
+    (funcall c (macroexpand *form*))))
+
 
 ;;;;****   FIRST PASS :   INLINE   FUNCTIONS   (PRIMOPS)
 
@@ -11157,7 +11168,8 @@
                                    (symbol-suffix *toplevel-name*
                                                   (incf subform-count)))))))))
                  (return-from compile-toplevel-form))
-                ((LOCALLY EVAL-WHEN COMPILER-LET MACROLET SYMBOL-MACROLET)
+                ((LOCALLY EVAL-WHEN COMPILER-LET MACROLET SYMBOL-MACROLET
+                  WITHOUT-PACKAGE-LOCK)
                  (let ((*form* form))
                    ;; call c-LOCALLY resp. c-EVAL-WHEN resp. c-COMPILER-LET
                    ;; resp. c-MACROLET resp. c-SYMBOL-MACROLET:
@@ -11359,10 +11371,8 @@
 ;;               an Output-Stream. Default: t.
 ;; :listing      should be nil or t or a Pathname/String/Symbol or
 ;;               an Output-Stream. Default: nil.
-;; :warnings     indicates, if the Warnings should also appear on the
-;;               screen.
-;; :verbose      indicates, if the Errors also have to appear on the
-;;               screen.
+;; :warnings     indicates, if the Warnings should also appear on the screen.
+;; :verbose      indicates, if the Errors also have to appear on the screen.
 (defun compile-file (file &key (output-file 'T) listing
                                ((:warnings *compile-warnings*)
                                 *compile-warnings*)
Index: foreign1.lisp
===================================================================
RCS file: /cvsroot/clisp/clisp/src/foreign1.lisp,v
retrieving revision 1.25
retrieving revision 1.26
diff -u -w -b -u -b -w -i -B -r1.25 -r1.26
--- foreign1.lisp	14 Jun 2002 15:42:53 -0000	1.25
+++ foreign1.lisp	18 Sep 2002 17:42:35 -0000	1.26
@@ -282,7 +282,8 @@
 
 (defmacro default-foreign-language (lang)
   (language-to-flag lang)       ; error checking
-  `(eval-when (load compile eval) (setq *foreign-language* ',lang)))
+  `(eval-when (load compile eval)
+     (without-package-lock ("FFI") (setq *foreign-language* ',lang))))
 
 ;; get the even (start=0) or odd (start=1) elements of the simple vector
 (defun split-c-fun-arglist (args start)