Subject: [clisp-list] FFI guru/wizard award to be rewarded

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Wed Mar 19 08:27:12 2003

Cheers up to Aurelio Bignoli!

I wasn't aware until now that some users had developed great problem-solving ability with the CLISP FFI :-)

While his explanation - more in terms of "what steps" than "how" - was not judged entirely satisfactory by the one-person jury (me:-), the example he supplies show agility in similar situations of using the FFI, which imply understanding.

Aurelio is concerned with efficiency issues, let's have a look at his pattern:

>be determined only at runtime. If, on the contrary, data returned is a
>well known C type, it is better to avoid with-c-var/cast overhead by
>using an ad-hoc foreign function for the conversion.

>int get_int_data (void *p)
>{ return *(int *)p; }
>Not elegant, but quite fast :)

Obviously, you are concerned with efficiency, but not too much, since:
1. you're still using a byte-code virtual machine Lisp instead of a native code compiler implementation,
2. you didn't write even faster code, which completely bypasses the FFI overhead, as in:
LISPFUNN(get_int_data,1)
{ var object data=popSTACK();
  if (!faddressp(obj)) { fehler("foo of your own"); }
  /* better use int8/16/32 instead of generic compiler-dependent int */
  var sintL intdata = *(int*)(Faddress_value(obj));
  value1=L_to_I(intdata); mv_count=1; # UL_to_I on unsigned
}

So far, so good. Let's try to see if the FFI can nevertheless be fast enough for you. Please try the following:

(defun my-func-wrapper (...)
  (let* ((result (my-func ...))
         (data (my-struct-data result)))
    (unwind-protect
        (with-foreign-object (p 'c-pointer data)
          (foreign-value
            (ffi::%cast p
             (load-time-value
              (ffi::parse-c-type '(c-ptr int))))))
      (linux:free data))))
[There's still one (parse-c-type 'c-pointer) in there at run-time - we could get rid of it as well. Macroexpand-1 with-foreign-object]

I do not recommend at all that you write FFI code this way. I just want to know whether you find the speed acceptable. The compiler could/should/ought to generate code that way out of constant types, not you.

If speed becomes acceptable, I'll think about working out the details of such an optimization (read find out the complex cases where it fails).

BTW, could you show the C prototype for the real external function?
Why do you have the interface you mention, instead of
data and size as separate parameters?

I've been thinking about adding support for
(def-call-out-VARLEN my-func
  (:arguments
   (data (c-ptr (c-array uint8 (:size/count allocated-size))) :out)
   (allocated-size (c-ptr uint) :in-out)))
but not for :malloc style allocation, and your pattern is different
I fear that cannot cover your interface protocol.

>    (unwind-protect
>        (with-c-var (p 'c-pointer data)
>          (cast p `(c-ptr (c-array uint8 ,(my-struct-size result))))))
>      (linux:free data)))
Side note: A typical case (in my eyes) where FOREIGN-FREE must not invalidate the shared pointer... Oh well, the whole VALIDP issue could be revisited.

Regards,
	Jorg Hohle.

---

From: Aurelio Bignoli <[email-masked]>
Date: Fri Mar 21 09:45:04 2003

Hoehle, Joerg-Cyril writes:
 > Please try the following:
 > 

I used a "scientific" approach: I wrote three functions that convert
the foreign int object passed as argument using one of the the three
methods we discussed (ad-hoc C function, with-c-var/cast,
with-foreign-object/%cast) and then I timed a loop of 20000 calls of
the functions. I ran the tests on a PII/266 under Linux 2.4.18, and
here are the results:

ad-hoc C function:

   Real time: 0.181583 sec.
   Run time: 0.11 sec.
   Space: 0 Bytes

with-c-var/cast:

   Real time: 1.494392 sec.
   Run time: 1.17 sec.
   Space: 3440000 Bytes
   GC: 4, GC time: 0.56 sec.

with-foreign-object/%cast

   Real time: 0.638666 sec.
   Run time: 0.54 sec.
   Space: 1520000 Bytes
   GC: 2, GC time: 0.28 sec.

 > If speed becomes acceptable, I'll think about working out the
 > details of such an optimization (read find out the complex cases
 > where it fails).

well, for my needs speed was acceptable even with my first
implementation for the vector case:

(let ((result (make-array size :element-type ' (unsigned-byte 8))))
  (dotimes (i size)
    (setf (aref result i) (get-vector-element data i))))

(def-call-out get-vector-element
  (:name "get_vector_element")
  (:arguments (data c-pointer :in) (offset :in))
  (:return-type uchar))

unsigned char
get_vector_element (void *data, int offset)
{
  ((unsigned char *)data)[offset];
}

which, of course, is awfully slow! I wrote this code at the beginning
of 2002, when with-c-var and with-foreign-object weren't still
available (or, at least, I wasn't aware of them), and this was the
only solution I was able to devise.

 > data and size as separate parameters?

I'm trying to interface Berkeley DB C API
(http://www.sleepycat.com). It uses "Data Base Thangs" to read and
write data to/from DB files:

/* Key/data structure -- a Data-Base Thang. */
struct __db_dbt {
    /*
     * data/size must be fields 1 and 2 for DB 1.85 compatibility.
     */
    void     *data;      /* Key/data */
    u_int32_t size;      /* key/data length */

    u_int32_t ulen;      /* RO: length of user buffer. */
    u_int32_t dlen;      /* RO: get/put record length. */
    u_int32_t doff;      /* RO: get/put record offset. */

[some #defines omitted...]

    u_int32_t flags;
};
typedef struct __db_dbt DBT;

This is an historical interface, I think it is in use since the
earliest versions of DB under Unix.

Typical functions are

int DB->put(DB *db, DB_TXN *txnid, DBT *key, DBT *data, u_int32_t flags);
int DB->get(DB *db, DB_TXN *txnid, DBT *key, DBT *data, u_int32_t flags); 

They allow C programmers to easily read and write any C type. For Lisp
programmers things are a little more difficult :)