Subject: [clisp-list] FFI question: returning array of "C" structs

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Tue Aug 6 01:39:04 2002

John K. Hinsdale asked:
>I'm writing an external "C" function called from CLISP via the FFI and
>I want my "C" function to return an array of some variable number of
>structs from "C".  How do I do that?

You can do that with the standard CLISP FFI in only one case: when the array is zero-terminated. This includes a structure completely filled with 0 bytes, not just C strings of one byte characters.
Does your function match that requirement?

> how could you zero-terminate an array of structs,
No problem at all, as indicated above. :-)
When using zero'ed structs, beware of uninitialized padding bytes.

>My function fills in an array of structs (not pointers, whole structs,
Whole structs are no problem at all to the CLISP FFI.

>I've got a separate function "ncol()" that can be called to get the
>count from Lisp so it knows how many to read in.
The current FFI cannot use that, unless you start playing tricks using CAST, foreign variable objects and run-time declared FFI types.

This would become (using my FFI extensions, which are only half in  current CLISP, I don't know how much of this is in 2.29):

(let* ((entries (col-info)) (len (ncol)))
  ;; assumes col-info returns a C-POINTER type, otherwise cast
  (ffi:with-c-var (base 'ffi:c-pointer entries)
    (ffi:cast base `(ffi:c-ptr (ffi:c-array ffi:uint16 ,len)))))
    ;; run-time cast of complex thing is inefficient
or
; this form may optimize better in some possible far future
(let* ((entries (col-info)) (len (ncol)))
  ;; assumes col-info returns a C-POINTER type, otherwise cast
  (ffi:with-c-var (base `(ffi:c-ptr (ffi:c-array ffi:uint16 ,len)))
    (setf (cast base 'ffi:c-pointer) entries)
    base)) ; (deref base) is superfluous, CLISP converts to true Lisp objects

I'd appreciate comments on whether the above is a worthwile approach or obfuscation.

>The docs mention a "C-ARRAY-PTR"
>	(FFI:C-ARRAY-PTR c-type) This type is equivalent to what C
>	calls c-type (*)[]: a pointer to a zero-terminated 
>     array of items of the given c-type.
>but I'm confused as to whether this means I'm declaring a pointer to
>(1) the base of an array of "c-types" or to (2) the base of an array
>of "pointer to c-types"
>The "zero-terminated" blurb in the comments suggests (2) - how could
>you zero-terminate an array of structs, however, looking at examples
>of modules distributed with CLISP suggests (1) ...

It's exactly as the docs say (include Humpty Dumpty Lewis Carroll quote here) and don't be mislead by "suggestions", your beliefs or the typical use of zero-termination only applied to termination of char arrays. The concept is more general.

BTW, I still tend to think the following is the best approach to your needs. Yet it will probably never be implemented. The debate is whether it's worthwhile at all to provide high-level interfaces to low-level FFI stuff.
These complex prototypes are of no use to people who wish to generate declarations from existing C or IDL headers.

(def-call-out col-info
  :arguments ((len uint :alloca :out))
  :return-type (c-ptr (c-array uint16 is_size(len)))
  :guard #'(lambda (x) (not (foreign-address-null x))))
or a similar thing using an out parameter:
(def-call-out col-info
  :arguments ((entries (c-ptr (c-array uint16 is_size(return))) :alloca :out))
  :return-type uint)
or with an extra error status indication:
(def-call-out col-info
  :arguments ((len uint :alloca :out)
	      (entries (c-ptr (c-array uint16 is_size(len))) :alloca :out))
  :return-type uint
  :guard #'zerop) ; 0 is success return code

Regards,
	Jorg Hohle.

---

From: "John K. Hinsdale" <[email-masked]>
Date: Tue Aug 6 09:00:03 2002

> You can do that with the standard CLISP FFI in only one case: when the
 > array is zero-terminated. This includes a structure completely filled
 > with 0 bytes, not just C strings of one byte characters.  Does your
 > function match that requirement?

It can be made to (I'll just slap on a zeroed struct at the end).  My
structs contain info on database columns coming back from queries
(data, column name, flags, etc.).  As it turns out the data pointer is
always allocated & never NULL so I guess in my application the
zero-ised terminator struct will work.

 > When using zero'ed structs, beware of uninitialized padding bytes.

won't be a problem w/ mine
 
 > The current FFI cannot use that, unless you start playing tricks using
 > CAST, foreign variable objects and run-time declared FFI types.

I'll skip that! 

 > It's exactly as the docs say (include Humpty Dumpty Lewis Carroll
 > quote here) and don't be mislead by "suggestions", your beliefs or the
 > typical use of zero-termination only applied to termination of char
 > arrays. The concept is more general.

I guess the reason I was confused/surprised by the zero-terminate
convention is that it clearly doesn't work in some cases -- if
my C struct is, say a 3D coordinate in a graphics program, e.g.,
   struct point { int x; int y; int z; };
and if I want to return bunch of them, the zero terminator prevents me
from including (0, 0, 0) as valid data, which here is not exactly an
obscure case.  It would seem all this is moot if I use array of struct
pointers instead of array of whole structs (cost me one more malloc(),
so what...) so maybe I'll try that.  I think in theory there is even no
guarantee that a NULL ptr in "C" is really zero-bits in memory, but
that was probably for some weird HP processor of long ago.

 > BTW, I still tend to think the following is the best approach to your
 > needs. Yet it will probably never be implemented. The debate is

OK, but to be honest, what I'm trying to do is just get this to work,
hopefully w/out bugs and leaks, and then get the hell out of "C" and
into Lisp as soon as possible!

Thanks for the help....

---
John Hinsdale, Alma Mater Software, Inc., Tarrytown, NY 10591-3710 USA
[email-masked] | http://www.alma.com/staff/hin | +1 914 631 4690

---

From: "John K. Hinsdale" <[email-masked]>
Date: Tue Aug 6 23:39:05 2002

Following up on my questions of returning arrays of "C" structs to
CLISP using the FFI ...

SUMMARY: Joerg-Cyril: it's working; thanks!

The working solution I've arrived at is to have the C function return
an array of pointers to the C structures + one terminating NULL
pointer at the end.  It's a bit more hassle on the "C" side since
you have to memory-manage the pointers as well as the structs, but
that is why we are paid the big bucks :)

I would advise against returning an array of whole structs to CLISP in
any situation.

 > [Joerg-Cyril]

I have to say I strongly disagree w/ the convention of imposing a
special meaning (i.e., terminator) on "all zero byte" structures.
There would be too many applications where all-zero data was a valid
representation.  Mine isn't one of them, but I have the suspicion I'm
just lucky.  See my recent message for an example (graphics).

All this becomes moot when the array consists only of pointers, plus a
null-pointer terminator, and that is why I recommend the pointer-array
approach.  In "C", a null pointer is always well defined and and its
meaning always clear: it points to no valid data.  That's part of the
language spec.  A struct of all zeroes has no definite meaning, since
the struct members are user-defined.

---