Subject: [clisp-list] food for discussion: Overcoming limitations of the CLISP FFI

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Tue Mar 12 08:08:02 2002

Hi,

I summarized some known deficiencies of the current FFI and possible exits (sketch of proposals). Please contribute!

o Arrays of variable size are not supported.
C-ARRAY-MAX helps but doesn't cover all needs.
IDL/COM's size_is() annotation would cover many situations.
In Haskell (and COM), they say something along:
gethostname([out,size_is(len)] cstring name,
            [] int len); TODO in/out??
Examples of use: COM of course, regerror's errbuf_size, regexec's nmatch, TODO others
TODO: exact specification when used with :out or :in-out or possibly different mode for the length and the array parameters
TODO: interaction with *foreign-encoding*
PROPOSAL: Add size_is ability to function declaration. Needs working out (but exists in Haskell, COM etc.). Quite some work (typical danger of "better is worse" 100% solution).

Short-time work-around: my redefine-foreign-function hack

o Kevin Rosenberg is currently releasing the universal FFI (UFFI): a single API that supports AllegroCL, Lispworks and CMUCL.
http://www.med-info.com/uffi.shtml
One may consider it a threat to CLISP, as it operates at a low level of abstraction that the CLISP FFI does not provide. Quite on the contrary, both the CLISP and CMUCL FFI try to abstract away from the low level that e.g. AllegroCL and UFFI use. CMUCL provides access to both levels. CLISP and CMUCL's provide :out etc. and are rather declarative in nature, whereas other interfaces let you feel like a C programmer in Lisp syntax, working on pointers.

This is IMHO a "worse is better" dilemma: low-level interfaces are easy to write, and programmers used to C know how to use them. OTOH, high-level interfaces must be *designed* first, and the designer is constantly at risk that his/her design is unusable and hides away required flexibility, by capturing too few usage scenarios (cf. variable sized arrays).

From my feelings, I cannot recommend to provide a low-level FFI for CLISP that UFFI could directly map to. My feeling is that the more declarative, the better. It should be the other way round: an UFFI should provide a very high level, declarative interface. This one could be expanded into a set of macros and functions for all possible implementations. Mapping UFFI or Allegro's FFI to CLISP's would require involved program analysis and even not statically visible information about the behaviour of the foreign functions. Yet maybe nobody will come up with such a high level FFI API for Lisp.

A true case of worse is better (cf. in http://www.paulgraham.com/thist.html the success of T and the failure of NIL).

o VALIDP could be made SETF'able. Maybe this should be restricted to allow invalidation only.
Currently, some code uses
  (when (validp compiled-pattern)
    (mregfree compiled-pattern)))
PROPOSAL: code could benefit from the addition of
   (setf (validp compiled-pattern) nil)
or (mark-invalid compiled-pattern)
I feel (setf (validp #) T) quite unsafe. But again, in what name am I
limiting people's freedom? The current FFI does too much of this already.

Observation:
+ The following 3 limitations come from trying to provide access to C idioms. C++, CORBA, COM, Java or other language interfaces don't have such weird requirements and use very regular interfaces. COM requires variable sized arrays though (via the size_is annotation).

o No choice for initialization of c-union types
Example: sigaction slot TODO can either be an int or a pointer to a handler.
PROPOSAL: use FOREIGN-VARIABLE objects and CAST as needed. FOREIGN-VARIABLE is IMHO nearly unusable in the current FFI. Invent functions or macros to make them usable.

o Variable argument types
e.g. ioctl or sigaction (SIG_IGN vs handler)

o No malloc or free directly available
Consider the following function
int switch_port(char **name);
On entry, name points to a malloc'ed buffer. The buffer contents must stay valid until another name is selected.
On (successfu)l return, name points to another buffer containing the previous port name. This old buffer may then be free'ed.
PROPOSAL: I may be able to provide hacks (mostly Lisp-level) that provide malloc and free. So people could experiment. Yet the current FFI that they should rarely be needed. OTOH, they may be needed more with delayed dereferencing (see below) which requires more work under programmer's control.

o FFI dereferences everything function returns.
Only C-POINTER is available to prevent this, e.g. module regexp, but then foreign structure is really opaque.
PROPOSAL: would IMHO involve FOREIGN-VARIABLE types, as they feature exactly what is needed: a pair of FOREIGN-ADDRESS and type information. Access can be made as needed.
TODO: how to declare this "mode"?

o Danger of dereferencing out parameters when function returns failure
(cf various mails from me) Same problem as above, same solution: delay

o FFI is too picky about some types. E.g. type C-POINTER only accepts passing a FOREIGN-ADDRESS, but not FOREIGN-VARIABLE or FOREIGN-FUNCTION although these are built upon FOREIGN-ADDRESS.
TODO: investigate need

o FFI copies everything on stack, or via malloc
TODO how would allocation :none work with strings?
E.g. each call to regexp-exec duplicates the string to be searched for. It can be huge (MB range)!
A more performant interface would attempt to work in place, on Lisp objects, which is not possible with the FFI. TODO Is it actually possible to work in place with Unicode strings?

(For module regex, working in place would need to use re_search or re_search_2() instead of regexec(), since the latter doesn't allow to specify a stop position and calls strlen.)

Thanks for sharing comments,

Regards,
	Jörg Höhle.