Subject: [clisp-list] on FFI:mark-invalid-foreign, malloc() and finalization

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Tue Mar 11 03:24:00 2003

Hi,

I wrote:
>If somebody asks, I'll explain why 
>FOREIGN-FREE doesn't do this [invoke MARK-INVALID-FOREIGN] automatically.

before seeing that Sam Steingold asked:
>why doesn't foreign-free mark the object as invalid?!

It would indeed be a safe thing to do on simple foreign memory. What I call simple is something where a single memory allocation was performed, e.g. what FFI:SIMPLE-CALLOC does.

However, DEEP-MALLOC can perform many malloc() for arbitrarily nested substructures, as required by the foreign type. Yet it returns a single address to represent all of them.

BTW, other FFI typically only have SIMPLE-CALLOC (UFFI, or perhaps with initialization, instead of all zeroes, like LW).

Even a trivial-looking (DEEP-MALLOC 'c-string "abc") performs two allocations, as I mentioned in the documentation.

FOREIGN-FREE, using the :FULL keyword is able to walk the foreign type declaration and free any complex structure.

How to free any of these chunks individually, if the programmer needs that?

Because of FOREIGN-ADDRESS sharing, DEREF, SLOT etc., when they return a foreign-variable object (or foreign place), all share the same object. This is valuable for stack-allocated foreign objects, but less so for individually malloc()'ed memory.

Suppose you'd like to free the root address only. If you mark the base foreign-address as invalid, all others will be marked likewise and become unusable.

Therefore, the programmer also needs a way to control sharing of pointer bases (and those offsets), which I have not yet fully devised. SET-FOREIGN-BASE was a candidate name.

I defined a keyword-based interface for FOREIGN-FREE, with an eye on a future :mark-invalid option. However, I wanted to get working code out, for it had been sitting in my source tree for too long already, and I believe I cannot present FOREIGN-FREE :mark-invalid without explaining or providing a model for controlling the address sharing that CLISP does. I am not ready for that.

So I agreed to disagree :), to move forward faster by providing a usable malloc() interface to CLISP users, a separate MARK-INVALID-FOREIGN function and to delay a hypothetical 100% solution.

Yet I myself would like to be able to say
(FINALIZE fv #'FREE+INVALIDATE)
using a single provided function instead of repeating the same lambda every time:
(FINALIZE fv
  #'(lambda (fv)
      (mark-invalid-foreign fv)
      (foreign-free fv :full t-or-nil-as-needed-by-the-individual-case)))

Also, an automatic (FINALIZE fv #'FREE) done by SIMPLE-CALLOC or DEEP-MALLOC would be a bad idea in CLISP. This advice is specific to CLISP, because CLISP supports the :malloc-free allocation policy (though I believe there's very little use of it). The idea is that the foreign side may do the free(), Lisp would not know about it, thus FINALIZE would call it twice, which dumps core.
OTOH, CormanLisp does automatic finalization of the dynamic memory it allocates and returns. It certainly is useful in most cases (>90%?)

Talking about comparisons, what I like best is Lispworks' with-dynamic-objects (sp?): Upon dynamic exit of its scope, all malloc() which any FLI function did are free()ed. That is a reliable way of not forgetting some of them!
I believe this is useful it most cases, as UNWIND-PROTECT is. In only a few cases would you like memory to live longer (cf. using OPEN vs. WITH-OPEN-FILE).

However, I feel this is contradictory to CLISP's :malloc-free allocation policy where it is explicitly acknowledged cases where foreign code calls free() on memory allocated by Lisp. This is not covered in Lispworks or CormanLisp.

It looks to me like two incompatible design paths. Depending on current fashion (like OO etc.) and time, one seems to cover more useful cases than the other. Typical for Java/C++/Ada style OO is more: allocation and deallocation is controlled on the same side, so there's nothing like malloc() here and free() there.

I thought about adding malloc()/free() statistics to the output of EXT:SPACE. But I cannot count the number of bytes free() frees without duplicating its internal knowledge, which is a bad idea. Furthermore, I can not catch foreign free() or malloc() calls (to generate instances+bytes output for EXT:SPACE).
I dismissed this, based on the idea that it would cost me a lot of effort to implement it in a 100% manner, with rare benefit to the programmer. It's a typical "nice/valuable to be able to turn on debug/logging/tracing/auditing mode on when needed" item.
An easy option would be SPACE output where either bytes would always be empty, or a varying format line like "permanent-instances temporary-instances total-bytes-allocated".

I thought about SETF VALID T/NIL in the past. Setf NIL would be ok. However, I decided it's a bad idea to reactivate a zombie address from the Lisp level. This shall be done by lower-level C code, which has much farther possibilities than the Lisp level, e.g. overwriting the bits of the pointer to point to a new address, preserving EQ-ness. The Lisp-level interface to controlling foreign-bases that I thought about is more limited, yet is supposed to cover enough cases. Think of how using C code, you could change the bits of a bignum object (I'd have uses for that), while at Lisp-level it's not possible. Live with that limitation in Lisp, accept MARK-INVALID-FOREIGN (which I found positive that it's more visible than setf on VALIDP) and remember C if you need more than that.

And there's this voice whispering "over-engineered" somewhere around the CLISP FFI and me working on it. Hmm, how did I end up here?

Regards,
	Jorg Hohle.

---

From: Sam Steingold <[email-masked]>
Date: Tue Mar 11 07:37:27 2003

> * In message <[email-masked]>
> * On the subject of "[clisp-list] on FFI:mark-invalid-foreign, malloc() and finalization"
> * Sent on Tue, 11 Mar 2003 11:24:40 +0100
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>

there is no MARK-INVALID-FOREIGN and I think VALIDP should be setfable instead.

> Suppose you'd like to free the root address only. If you mark the base
> foreign-address as invalid, all others will be marked likewise and
> become unusable.

whatever is free()d, must be marked invalid.
a segfault in CLISP is a bug in CLISP (except, of course, the FFI :-),
so we should always barf on an attempt to free() the same memory twice.

if you want to release only the root, you setf the lower level pointers
to NULL, then call FOREIGN-FREE, which will release everything (and
invalidate the root.)

> I thought about SETF VALID T/NIL in the past. Setf NIL would be
> ok. However, I decided it's a bad idea to reactivate a zombie address
> from the Lisp level. This shall be done by lower-level C code, which
> has much farther possibilities than the Lisp level, e.g. overwriting
> the bits of the pointer to point to a new address, preserving
> EQ-ness. The Lisp-level interface to controlling foreign-bases that I
> thought about is more limited, yet is supposed to cover enough
> cases. Think of how using C code, you could change the bits of a
> bignum object (I'd have uses for that), while at Lisp-level it's not
> possible. Live with that limitation in Lisp, accept
> MARK-INVALID-FOREIGN (which I found positive that it's more visible
> than setf on VALIDP) and remember C if you need more than that.

(defun (setf validp) (new-v obj)
  (if (eq new-v (validp obj))
      new-v
      (etypecase obj
        (foreign
          (if new-v
              (error "cannot resurrect a zombie - no voodoo in CLISP!")
              (invalidate-foreign-object obj))))))

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Live Lisp and prosper.