Subject: [clisp-list] Open issues in future CLISP FFI

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Thu Apr 18 07:19:19 2002

Hi,

o Modules and deadly PACKAGE-LOCK feature interaction
 + partial solution for LISPFUN is possible -> unlock and keep unlocked (used only with own modules, not FFI).
 - but not for the module_object table (used by FFI DEF-CALL-IN and own modules)

o macrology
 - FFI:CAST, FFI:OFFSET etc. are already used in incompatible ways for SYMBOL-MACROS hiding the actual FOREIGN-VARIABLE objects.
  I feel that hiding these objects is a bad thing to do when working
with delayed references. So I let the programmer manipulate them as
the first class objects that they indeed are.

o reopen-foreign-library: update-address of all known functions/variables
 (useful when restarting from .mem file, e.g. servers)
 - need additional thoughts whether reopen or initial call can be made transparent.

o half of WITH-STACK-FOREIGN, WITH-FOREIGN-ARRAY
 - need to find good names & syntax
 - allow variable element-type instead of just size?

o allow libraries to be refered to by name instead of value

Here is working code which shows most of this (from future cl-pdf):

(def-lib-call-out zlib-compress-string *zlib*
  (:name "compress")
  (:arguments (dest ffi:c-pointer :in)
	      (destlen (ffi:c-ptr ffi:ulong) :in-out)
	      (source ffi:c-string)
	      (sourcelen ffi:ulong))
  (:return-type ffi:int)
  (:language :stdc))

(defun compress-string (source)
  "Compress the string SOURCE. Returns an array of bytes
representing the compressed data."
  (let* ((sourcelen (length source))
	 (destlen (+ 12 (ceiling (* sourcelen 1.05)))))
    (ffi::with-foreign-array (dest (c-array uint8 destlen)) ; no init
      (multiple-value-bind (status actual)
	  (zlib-compress-string (foreign-address dest) destlen source sourcelen)
	(if (zerop status)
	    ;;(subseq (ffi::foreign-value dest) 0 actual)
	    ;;ffi::%cast not usable because of different size...
	    (ffi::foreign-value (ffi::%offset dest 0 (ffi::parse-c-type `(c-array uint8 ,actual))))
	  (error "zlib error, code ~d" status))))))

o COMPLEX-MALLOC is there (i.e. must provide initial value), but no uninitialized malloc (SIMPLE-MALLOC?) and no foreign-delete.
 - Simple-malloc is dangerous. (simple-malloc (c-ptr (c-array uint8 32)) doesn't do what you may expect: it would just allocate a single pointer, not a linked structure...
A cheap trick is (complex-malloc (c-ptr (c-array-max uint8 32)) #()), e.g. provide an empty string or vector.

o Compiler optimization: Most PARSE-C-TYPE is known at compile-time and can be constant folded, instead of being recomputed at every call (for some functions).
 - BACKQUOTE expansion is unportable to analyse
 - Other non constant (QUOTE x) cases are hard to discover, e.g.
   (parse-c-type (list 'c-array 'uint8 (length source)))
   which could be optimized as well (in theory).
 - differences between compile-time and load-time environment?

o parallels between the existing (AmigaOS only) FOREIGN-LIBRARY-VARIABLE/FUNCTION and FOREIGN-ADDRESS-FUNCTION to be investigated, possibly unify interfaces.

o portable .fas files versus platform-dependent macroexpansion

o allocate-string, avoiding FFI 1:1 encoding restriction
 + would allow UTF-16
 - to be specified

I have (or decided upon):

o zlib-compress for CL-PDF (w/ preliminary dynamic library API)
o (DEF-LIB-CALL-OUT name library options...) (pending portable .fas problem)
o (resolve-foreign-name "foo" library 'FOREIGN-FUNCTION) as Sam suggested
o FOREIGN-ADDRESS-UNSIGNED and UNSIGNED-FOREIGN-ADDRESS
o FOREIGN-ADDRESS-FUNCTION
o MARK-INVALID-FOREIGN (dangerous!)
o and a few more...

Comments appreciated!

Regards,
	Jörg Höhle.