Subject: [clisp-list] FFI troubles

From: Nikodemus Siivola <[email-masked]>
Date: Mon Jan 13 04:03:02 2003

Much confusion and the docs aren't clear enough for me. :( But if someone
helps me out, I'll promise to write a tutorial as soon as I figure this
out...

From the imp. notes:

"A foreign function description is written as a Lisp file, and when
compiled it produces a #P".c" file which is then compiled by the C compiler
and may be linked together with lisp.a."

This seem simple enough. The trouble is that the later examples show the
contents fo a c-file and a lisp-file and then hit the shell prompt with
fury:

"$ clisp-link create-module-set cfun callcfun.c
$ cc -O -c cfun.c
$ cd cfun
$ ln -s ../cfun.o cfun.o
Add cfun.o to NEW_LIBS and NEW_FILES in link.sh.
$ cd ..
$ base/lisp.run -M base/lispinit.mem -c callcfun.lisp
$ clisp-link add-module-set cfun base base+cfun
$ base+cfun/lisp.run -M base+cfun/lispinit.mem -i callcfun
> (test-c-call::call-cfun)"

My problem is twofold:

1) I'd like to get the example to work. I can't.
2) I'd like to understand what I'm doing. Unfortunately the magic mantras
   are a bit low on explanations.

Following the example (on Debian testing, for what it's worth) I hit to
first bump right at the beginning: no clisp-link. Ah! It's in
/usr/lib/clisp! Ok:

$ sudo ln -s /usr/lib/clisp/clisp-link /usr/bin/clisp-link

Then it seems theat I need linkkit-dir in current dir. Ok:

$ cp -r /usr/lib/clisp/linkkit .

Smooth sailing for a while, then I need to amend the correct directory:

$ base/lisp.run -M /usr/lib/clis/base/lispinit.mem -c callcfun.lisp

Same here:

$ clisp-link add-module-set cfun /usr/lib/clisp/base base+cfun

Then I seem to be missing lisp.run and lispinit.mem from
base+cfun. Symlink originals and try again: works. Almost: FFI complains
that the foreign function does not exist + the lisp seems to be missing
quite a few functions, EXIT & co. among them... Ctrl-Z and kill the bugger,
then write here for help...

One source of confusion is that the linking procedure is obscured by
clisp-link, which seems to be documented nowehere.

Are there better docs somewhere?

Also, am I interpreting this correctly when I think that to call a foreign
function with clisp you need to link the extension to clisp statically --
no dlopen available?

Also, does anyone have experience with using code depending on the Boehm GC
via clisp's ffi: any serious troubles to be expected?

Thanks,

  -- Nikodemus

---

From: Nikodemus Siivola <[email-masked]>
Date: Mon Jan 13 16:06:03 2003

> Much confusion and the docs aren't clear enough for me. :( But if someone

After some fast and furious head-banging I finally got it working. Somehow.
Lot's of unclear issues as of yet, but no matter.

However, since this *is* answered in the FAQ, but the entry wasn't all that
helpful for me I recorded the entire process to the best (sic!) of my
understanding into a small HOWTO, for the next bungler like me.

It's not much -- basically just a more spelled out version of what's in the
Implementation Notes.

Cheers,

  -- Nikodemus

--
CLISP-FFI-HOWTO

 Nikodemus Siivola

 Version 0.1, 2003-01-14

Foreword
--------

This is a short guide to extending CLISP on Debian sarge via it's FFI and
module facilities. Even though it may be helpful on other platforms as
well, no attempt at platform-agnosticism has been made. Please email any
comments, corrections and additions to [email-masked].

This guide is by no means meant to be authoritative: merely a
collection of the bumps the author hit along to road and how he
managed to steer clear of them.

For *real* documentation read the CLISP Implementation Notes

 /usr/share/doc/clisp/doc/impnotes.html

 or

 http://clisp.cons.org/impnotes.html

and the CLISP manual page.

Also check out the source:

 apt-get source clisp

And see clisp-2.28/modules/*

Preliminaries
-------------

Unfortunately the current version in sarge (clisp-1:2.28-1) installs
clisp-link into /usr/lib/clisp, which is not in PATH, so you
need to either:

 a) symlink it to /usr/bin

 b) add /usr/lib/clisp to your PATH

 c) alias clisp-link=/usr/lib/clisp/clisp-link

Also, you want have to set up correct environment variables:

 export CLISP_LINKKIT=/usr/lib/clisp/linkkit

Interface and Extension
-----------------------

For the sake of argument, start in a clean directory with the
following files, containg a minimal extension and an interface
for it.

Extension is in vector.c:

--
#include <stdio.h>

typedef float float_t;
typedef float_t vector_t[3];

float_t dot_product (vector_t, vector_t);

float_t
dot_product (vector_t a, vector_t b)
{
  float_t x = 0;
  int i = 0;
  for (i=0; i<3; i++)
    x += a[i] * b[i];
  return x;
}
--

Compile it without linking:

 $ cc -c vector.c

Interface is in vector-interface.lisp:

--
(ffi:def-call-out dot-product
	  (:name "dot_product")
	  (:language :stdc)
	  (:arguments
	   (a (ffi:c-ptr (ffi:c-array single-float 3)))
	   (b (ffi:c-ptr (ffi:c-array single-float 3))))
	  (:return-type single-float))
--

Compile it with clisp:

 $ clisp -c vector-interface.lisp

At this point you working directory should contain:

 vector.c
 vector.o
 vector-interface.lisp
 vector-interface.c
 vector-interface.fas
 vector-interface.lib

Creating the Extension
----------------------

Use clisp-link to create a new module-set named vector-module:

 $ clisp-link create-module-set vector-module vector-interface.c

This should create a subdirectory named vector-module that contains:

 Makefile
 links.sh
 vector-interface.c  (This one should be just a symlink.)

The links.sh needs to be amended: add "vector.o" to NEW_LIBS and
NEW_FILES and "../vector-interface.fas" to TO_LOAD. You should end up
with something like this:

--
file_list=''
mod_list=''
if test -r vector-interface.c; then
  file_list="$file_list"' vector-interface.o'
  mod_list="$mod_list"' vector_interface'
fi
make clisp-module CC="${CC}" CFLAGS="${CFLAGS}" INCLUDES="$absolute_linkkitdir"
NEW_FILES="$file_list vector.o"
NEW_LIBS="$file_list vector.o"
NEW_MODULES="$mod_list"
TO_LOAD='../vector-interface.fas'
--

Due to the deficiencies of our configuration we also need to create a
symlink to vector.o in the vector-module directory:

 $ cd vector-module
 $ ln -s ../vector.o
 $ cd ..

FIXME: ^-- This could probably be avoided. How?

Next step is to generate a new linking-set:

 clisp-link add-module-set vector-module /usr/lib/clisp/base base+vector

This creates the directory base+vector that will contain the extended
CLISP.

Let's try it out:

 $ base+vector/lisp.run -M base+vector/lispinit.mem

 ...

 [1]> (dot-product #(1.0 1.0 1.0) #(1.0 2.0 3.0))
 6.0

Seems ok!

ToDo
----

 - Dynamic linking: is it possible?
 - Writing interfaces / generating them from header-files
 - Using the extended CLISP with command-line arguments
 - More on link.sh
 - clisp-link run

---

From: Kaz Kylheku <[email-masked]>
Date: Mon Jan 13 16:40:06 2003

> Preliminaries

This stuff can be automated quite effectively. In the Meta-CVS install
script, I do some crusty shell programming to discover the
installation-specific identity of that directory which is usually 
/usr/lib/clisp:

Firstly, get the path to the clisp executable. It has to be in the
search path.

    if ! CLISP_PATH="$(which clisp)" ; then
      echo clisp command not found
      exit 1
    fi

We assume that the path always ends in PREFIX/bin/clisp, and that the
lib directory is PREFIX/lib/clisp. So we chop off the last two
components, bin/clisp and replace them with lib/clisp. 

    # Hey look, Lispy looking shell crud on the right hand side!
    CLISP_ROOT=$(dirname $(dirname "$CLISP_PATH"))
    CLISP_LIB="$CLISP_ROOT/lib/clisp"

Now we can compute CLISP_LINKKIT:

    CLISP_LINKKIT="$CLISP_LIB/linkkit"

and we can run clisp-link, which lives in $CLISP_LIB, and installs
without execute permissions:

    sh "$CLISP_LIB/clisp-link" add-module-set ... etc

> 	  (:return-type single-float))

To make your tutorial more complete, you might want to also discuss
def-call-in. There are issues there: if you want to use a Lisp function
as a callback, and that function is in a custom package, the build
has to be broken into two steps, because the generated FFI module that
is linked into your lisp executable makes references to the package.
This two-step process is selected in the link.sh script by using the
TO_PRELOAD variable, IIRC. 

The reason that the two-step process is needed is that when your lisp
executable is built with your custom module, the module stub code will
try to access the symbols in your package that are names of the Lisp
call-in functions. This means that the package has to exist, or this
will blow up. So the solution is to first compile the package defintion
to make a custom .mem image, and then use *that* .mem image with the
new lisp executable, to compile the remaining .lisp files and drop  the
final .mem that has everything.

> FIXME: ^-- This could probably be avoided. How?

By putting your stub C code into the vector-module subdirectory, and
adding its compilation to the Makefile there.

>  clisp-link add-module-set vector-module /usr/lib/clisp/base base+vector

This will only work if clisp-link is in your path. Maybe this is fixed
in recent CLISP, but clisp-link ``traditionally'' installs in
$(PREFIX)/lib/clisp, rather than the bin directory, *and* it installs
without execute permissions.

> This creates the directory base+vector that will contain the extended

Unless the directory exists already in which case it bails.  So to
force a fresh rebuild, you have to manually remove that directory.

>  - Dynamic linking: is it possible?

You bet it's possible. I have successfully wrapped the wxWindows
library, which is dynamically linked itself, and drags in a whole lot
of other dynamically linked crap. There is nothing to it. 

The linking which puts together the lisp.run executable is just
ordinary linking.

>  - Using the extended CLISP with command-line arguments

One way to package up a command line CLISP application is to do it the
way Meta-CVS does it. You create a front-end script which looks like
this:

   #!/path/to/lib/clisp/my-linking-set/lisp.run -M/path/to/lib/clisp/my-linking-set/lispinit.mem
   (my-package:my-main-function ext:*args*)

This script can be generated by another script which sticks in the
correct actual paths.

-- 
Meta-CVS: directory structure versioning; versioned symbolic links;
versioned execute permission; versioned property lists; easy branching
and merging and third party code tracking; all implemented over the
standard CVS command line client -- http://freshmeat.net/projects/mcvs

---

From: Nikodemus Siivola <[email-masked]>
Date: Mon Jan 13 16:59:09 2003

On Mon, 13 Jan 2003, Kaz Kylheku wrote:

*snip good stuff*

> adding its compilation to the Makefile there.

... what Makefile? The one clisp-link put's there (none that I can see) or
one that I (should) put there? Hmm. Have to think about integrating the FFI
to build-process.

> in recent CLISP, but clisp-link ``traditionally'' installs in

See beginning of the howto, in preliminaries. I just made an
alias for the full path, since right now I'm doing this by hand -- and
anything else would be a tad tedious.

> without execute permissions.

Oops. I remember now that I chmodded it a month back -- have to the fix doc
there. Is there a *reason* for no exec permissions?

> of other dynamically linked crap. There is nothing to it.

Hmm. Any chance of getting my grubby little hands on any code? ;)

> way Meta-CVS does it. You create a front-end script which looks like

Seems I'll have to see how MetaCVS handles things...

Thanks,

  -- Nikodemus

---

From: Sam Steingold <[email-masked]>
Date: Mon Jan 13 17:57:02 2003

> * In message <[email-masked]>
> * On the subject of "[clisp-list] CLISP-FFI-HOWTO (Was: FFI troubles)"
> * Sent on Tue, 14 Jan 2003 02:05:07 +0200 (EET)
> * Honorable Nikodemus Siivola <[email-masked]> writes:
>
> After some fast and furious head-banging I finally got it
> working. Somehow.  Lot's of unclear issues as of yet, but no matter.

if you make it into a web page, I will link to it from the CLISP pages,
but it would be better if you suggested improvements (in the form of
context diffs) to the CLISP implementation notes.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
What garlic is to food, insanity is to art.

---

From: Sam Steingold <[email-masked]>
Date: Mon Jan 13 18:08:02 2003

> * In message <[email-masked]>
> * On the subject of "[clisp-list] Re: CLISP-FFI-HOWTO (Was: FFI troubles)"
> * Sent on Mon, 13 Jan 2003 16:38:38 -0800 (PST)
> * Honorable Kaz Kylheku <[email-masked]> writes:
>

my CLISP setup would fail this:

$ which clisp
/usr/local/bin/clisp
$ ls -l /usr/local/bin/clisp
   0 lrwxrwxrwx    1 root     root           25 Mar 14  2002 /usr/local/bin/clisp -> ../src/clisp/stable/clisp*
$ ls -l /usr/local/src/clisp/stable
   0 lrwxrwxrwx    1 sds      sds            11 Dec 21 18:25 /usr/local/src/clisp/stable -> stable-2.30/

the fail-safe method is:

$ clisp -q -norc -x '*lib-directory*'
#P"/usr/local/src/clisp/stable-2.30/"
$

see <http://clisp.cons.org/clisp.html#opt-libdir>

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Save your burned out bulbs for me, I'm building my own dark room.

---

From: Kaz Kylheku <[email-masked]>
Date: Mon Jan 13 20:45:05 2003

> my CLISP setup would fail this:

[ snip ]

> #P"/usr/local/src/clisp/stable-2.30/"

Okay, I have fixed my install script to do this:

CLISP_LIB=$(clisp -q -norc -x '(progn (princ *lib-directory*) (values))')

Thanks.

---

From: Nikodemus Siivola <[email-masked]>
Date: Mon Jan 13 21:45:04 2003

> context diffs) to the CLISP implementation notes.

I'll do both. ;)

The webpage is

 http://www.niksula.cs.hut.fi/~tsiivola/clisp-ffi-howto.html

Diffs later.

Cheers,

  -- Nikodemus

---

From: Nikodemus Siivola <[email-masked]>
Date: Tue Jan 14 00:45:03 2003

As far as I have understood, FFI basically requires using the module
extension, but according to the Imp. Notes it is available for both UNIX
and Win32, but module facility is UNIX only.

Does this mean that

 a) Modules facility works on Win32 as well. (Cygwin? MinGW? M$-based?)
 b) You can use FFI without modules on any platform.
 c) You can use FFI without modules on Win32.
 d) ?

Confused once again,

  -- Nikodemus

---

From: Sam Steingold <[email-masked]>
Date: Tue Jan 14 07:46:06 2003

> * In message <[email-masked]>
> * On the subject of "[clisp-list] More FFI questions"
> * Sent on Tue, 14 Jan 2003 10:44:17 +0200 (EET)

... on which FFI is available

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Diplomacy is the art of saying "nice doggy" until you can find a rock.

---

From: Sam Steingold <[email-masked]>
Date: Tue Jan 14 08:07:04 2003

> * In message <[email-masked]>
> * On the subject of "Re: [clisp-list] CLISP-FFI-HOWTO (Was: FFI troubles)"
> * Sent on Tue, 14 Jan 2003 07:44:09 +0200 (EET)

great.

1. please spell-check it ("via it's FFI" --> "via its FFI" &c)

2. please turn URLs and into hyperlinks.

3. my I suggest that you validate your HTML with
   <http://validator.w3.org/>?

I linked to you page from the `resources' section on the CLISP site.

thanks.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Don't hit a man when he's down -- kick him; it's easier.

---

From: Nikodemus Siivola <[email-masked]>
Date: Tue Jan 14 09:44:04 2003

> 1. please spell-check it ("via it's FFI" --> "via its FFI" &c)

Done, now bad only grammar remains. ;)

> 2. please turn URLs and into hyperlinks.

Done. Lost the visible urls, though (see next).

>    <http://validator.w3.org/>?

This is unfortunately rather inconvenient, since the html is generated from
simpler, even more braindead markup: check
http://www.niksula.cs.hut.fi/~tsiivola/clisp-ffi-howto.txt for the
original.

  -- Nikodemus