Subject: Dynamic loading of shared libraries?

From: Eric Marsden <[email-masked]>
Date: Tue, 10 Nov 1998 12:31:16 +0100

Has anyone worked on allowing CLISP to load shared libraries at runtime?
Last night I was all happy at having compiled the dlfcn routines as a
module, but I realized a little belatedly that the tricky bit is argument
passing to foreign routines. Is this too difficult technically?

-- 
Eric Marsden
emarsden @ mail.dotcom.fr
It's elephants all the way down

---

From: Bruno Haible <[email-masked]>
Date: Tue, 10 Nov 1998 14:21:28 +0100 (MET)

> passing to foreign routines. Is this too difficult technically?

Passing arguments to foreign routines is best (and more portably) done
through CLISP's FFI. For example, the modules/regexp/ code is just a
wrapper around the GNU regexp routines, and the interface is made by
the following declarations:

   (def-c-type size_t uint)
   (def-c-type regex_t-ptr c-pointer)

   ; Types exported by regex.h:
   (def-c-type regoff_t int)
   (def-c-struct regmatch_t
     (rm_so regoff_t)
     (rm_eo regoff_t)
   )

   (def-c-call-out mregcomp (:arguments (ppreg (c-ptr regex_t-ptr) :out)
                                        (pattern c-string)
                                        (cflags int)
                            )
                            (:return-type int)
   )
   (def-c-call-out regexec (:arguments (preg regex_t-ptr)
                                       (string c-string)
                                       (nmatch size_t)
                                       (pmatch (c-ptr (c-array regmatch_t #.num-matches)) :out)
                                       (eflags int)
                           )
                           (:return-type int)
   )
   (def-c-call-out mregerror (:arguments (errcode int)
                                         (preg regex_t-ptr)
                             )
                             (:return-type c-string :malloc-free)
   )
   (def-c-call-out mregfree (:arguments (preg regex_t-ptr))
                            (:return-type nil)
   )

The syntax of these declarations is described in foreign.txt and impnotes.html.

Bruno

---

From: [email-masked] (Paolo Amoroso)
Date: Tue, 10 Nov 1998 15:32:30 GMT

> > Has anyone worked on allowing CLISP to load shared libraries at runtime?
[...]
> through CLISP's FFI. For example, the modules/regexp/ code is just a
[...]

This is handy but, if I get things right, C wrappers and an explicit
compile/link phase are still required with CLISP. This process is managed
by clisp-link.

Compare this to the FFI of the Scheme system STk 3.99.x
(http://kaolin.unice.fr/STk/). Since STk supports dynamic loading of shared
libraries at runtime, all that is needed to use, for example, the standard
C library function printf is:

(define-external printf ((format :string) . l)
  :return-type :int)

(printf ("This is a %s\n" "test"))

STk takes care of transparently loading the appropriate system library at
runtime. No C code or compile/link phase is required. Is this already
possible with CLISP? If not, will it be?

Paolo
-- 
Paolo Amoroso <[email-masked]>

---

From: Eric Marsden <[email-masked]>
Date: Thu, 12 Nov 1998 11:42:53 +0100

> the following declarations:

As Paolo noted, the FFI requires a compile/link phase. It would be nice if
CLISP were able to load libraries dynamically. I worked on this yesterday,
copying STk's mechanism, which implements the stack for argument passing as
an array of fake_type, where fake_type is a union of the different types
which are supported (long, int, char, pointer, float, double). There is a
bit of trickiness about alignment with doubles, but it works for me on
Linux/x86. 

Bruno is worried about portability, but I don't see this as a real problem:
firstly dynamic linking is necessarily platform-dependant, secondly STk gets
it to work on a fair range of Unixes (only by resorting to assembly for
certain architectures), and thirdly it is only an add-on; people can always
use the normal FFI on platforms which don't support dynamic loading. Does
anyone have an idea for a better mechanism for accessing dynamic libraries?

I have appended an example of the low-level interface below. What is needed
now is a macro to define an external call, which will look after loading the
library, resolving the symbol, and defining a Lisp function which pushes the
arguments and calls the correct C-level function according to the return
type. I am trying to figure out how macros work.

     (use-package "DL")
     
     (setq libc   (dl::dlopen "libc.so.5" dl::RTLD_NOW))
     (setq libm   (dl::dlopen "libm.so.5" dl::RTLD_NOW))
     (setq labs   (dl::dlsym libm "labs")) ;; long int labs(long int j)
     (setq pow    (dl::dlsym libm "pow"))  ;; double pow(double x, double y)
     (setq getpid (dl::dlsym libc "getpid"))

     (terpri)
     (dl::init_external_call)
     (format t "Current process' PID is ~a~%" (dl::return_int getpid))
     
     (dl::init_external_call)
     (dl::push_long -5)
     (format t "Absolute value of -5 is ~a~%" (dl::return_long labs))
     
     (dl::init_external_call)
     (dl::push_double 2.0d0)
     (dl::push_double 3.0d0)
     (format t "2 ^ 3 = ~a~%" (dl::return_double pow))

---

From: Eric Marsden <[email-masked]>
Date: Thu, 19 Nov 1998 11:45:09 +0100

My code implementing dynamic loading of shared libraries for CLISP using the
dlopen() mechanism and an emulated stack is available for the adventurous. I
have only tested it on Linux/x86, but with some hacking the same mechanisms
should work on most Unix platforms and architectures.

This release is mainly an invitation for comments from interested people; I
am new to Common Lisp and would appreciate any suggestions on the code.

   http://www.chez.com/emarsden/downloads/dl0.1.tar.gz

I am also working on an interface to the GD library which allows the
programmatic generation of GIF images, using the DL module.