Subject: [clisp-list] on FFI:mark-invalid-foreign, malloc() and finalization

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Thu Mar 13 01:30:01 2003

Sam Steingold wrote:
>there is no MARK-INVALID-FOREIGN
I have one -- apparently yet another item not yet submitted to sf' patches section.
>and I think VALIDP should be setfable instead.
Actually, I don't care about that name. Make it SETF'able instead if you wish. Maybe it'll have the benefit of making the symbol VALIDP more visible.

What I do care slightly about is that
o MARK-INVALID-FOREIGN signals an error when not given a foreign entity.
What would (SETF VALIDP) do?

>whatever is free()d, must be marked invalid.
Here you're killing the messenger together with the message.
I mentioned: "I believe I cannot present FOREIGN-FREE :mark-invalid without explaining or providing a model for controlling the address sharing that CLISP does."

That's exactly what's happening here. I believe you misunderstand what's going on in the FFI w.r.t. foreign-addresses etc.
Please look at the following. In order to understand, a model of the relationship between foreign entities and VALIDP is needed. I did not yet write that.

[84]> (setq fm (deep-malloc 'uint8 123))
#<FOREIGN-VARIABLE FOREIGN-ALLOC* #x00CC1920>
[85]> (foreign-address fm)
#<FOREIGN-ADDRESS #x00CC1920>
[86]> (unsigned-foreign-address (foreign-address-unsigned fm))
#<FOREIGN-ADDRESS #x00CC1920>
[87]> (equalp * **)
NIL			;yet they print the same!

[88]> (def-call-out c-self
  (:name "ffi_identity")
  (:arguments (obj c-pointer))
  (:return-type c-pointer) (:language :stdc))
C-SELF
[89]> (mark-invalid-foreign (c-self (foreign-address fm)))
*** - MARK-INVALID-FOREIGN: must not invalidate sole FFI session pointer

It's refering to the underlying O(fp_zero) object. Almost all of FFI dies if this is destroyed.

[91]> (mark-invalid-foreign (unsigned-foreign-address (foreign-address-unsigned fm)))
*** - MARK-INVALID-FOREIGN: must not invalidate sole FFI session pointer

[93]> (mark-invalid-foreign fm)
; or  (mark-invalid-foreign (foreign-address fm))
T

[103]> fm
#<INVALID FOREIGN-VARIABLE FOREIGN-ALLOC* #x00CC1920>
[104]> (foreign-free fm)
*** - #<INVALID FOREIGN-POINTER #x00CC1920> comes from a previous Lisp session and is invalid
; the FOREIGN-POINTER object hidden beneath FOREIGN-ADDRESS and FOREIGN-VARIABLE appears here

[105]> (foreign-free (unsigned-foreign-address #x0cc1920))
NIL		; no problem at all - and that pointer must not be invalidated.

One must not attempt to invalidate foreign-address/variable objects returned via :return-type C-POINTER or UNSIGNED-FOREIGN-ADDRESS, or a future FOREIGN-ADDRESS-VARIABLE etc. FOREIGN-POINTER object sharing is happening.

It's only safe on the objects returned via FOREIGN-LIBRARY (when done with it) and SIMPLE-CALLOC/DEEP-MALLOC.

What I've been trying to do is incremental changes to the FFI. Attempting to change the above would be a major rewrite of the FFI, which is not my goal. Instead, I try to put the existing code to better use, like using the FOREIGN-POINTER sharing feature as a dependency tracking mechanism. For instance, closing a shared library immediately invalidates all dependent functions (but not objects that they may have returned, which CLISP does not know about - but my envisioned SET-FOREIGN-BASE would allow the programmer to let CLISP know).

CLISP documentation would have to be updated to explain when fresh pointers and when shared pointers are returned by FFI functions.

However, what this discussion boils down is no more that the set of defaults to FOREIGN-FREE. I don't object a default key value of :validp nil. It's even reasonable (I believe invalidating covers more cases than not invalidating).
But I strongly object to not letting the programmer control this coupling. FOREIGN-FREE must not unconditionaly invalidate the pointer. It can have a keyword argument.

Providing an initial FOREIGN-FREE which does not invalidate was an incremental approach of mine. It does not prevent programmers code from working, since MARK-INVALID-FOREIGN is (supposed to be) there as well: it does not remove freedom from the programmer. Providing a FOREIGN-FREE which inconditionaly invalidates the foreign entity object would have removed freedom and prevented useful cases from working.

Providing a FOREIGN-FREE which later on default to invalidate means an incompatible interface change which may break code. That's the price to pay for incrementality and "let people play with it so as to get feedback".

>if you want to release only the root, you setf the lower level pointers
>to NULL, then call FOREIGN-FREE, which will release everything (and
>invalidate the root.)
That's IMHO cumbersome. FOREIGN-FREE can work independently of destroying pointers in the structure it's about to free. Let that zeroing out job be the burden of people who want a conservative GC.

One could also discuss whether :FULL should default to true. It would work with SIMPLE-CALLOC on the typical cases I expect where arrays of immediates are allocated, or even opaque pointers (simple-calloc `(c-array c-pointer ,n)) or (simple-calloc 'c-pointer :count n). And it's the right thing to do with DEEP-MALLOC (again, in most cases, since the structure at the moment of deallocation may be very different from what got allocated).
There's just a little performance impact on using :full where it's not necessary for analysing the type declaration. I should try more to learn not to care about that.

Regards,
	Jorg Hohle.

---

From: Sam Steingold <[email-masked]>
Date: Thu Mar 13 07:58:16 2003

> * In message <[email-masked]>
> * On the subject of "[clisp-list] on FFI:mark-invalid-foreign, malloc() and finalization"
> * Sent on Thu, 13 Mar 2003 10:28:41 +0100
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>
> Sam Steingold wrote:
> Actually, I don't care about that name. Make it SETF'able instead if
> you wish. Maybe it'll have the benefit of making the symbol VALIDP
> more visible. 
good.

> What would (SETF VALIDP) do?

I send the pseudo-code for (SETF VALIDP) here before.
<http://article.gmane.org/gmane.lisp.clisp.general/6651>
(setf (validp "foo") t) ==> t
(setf (validp "foo") nil) ==> (error "may not invalidate non-foreign")
basically, since (VALIDP "foo") works, I see no reason to forbid
(setf (validp "foo") t).

> I mentioned: "I believe I cannot present FOREIGN-FREE :mark-invalid
> without explaining or providing a model for controlling the address
> sharing that CLISP does." 

My point is: do we want to allow users to free() the same memory twice?
Yes or No?
I am not forcing your decision here, but I want clarity.
IIUC, it is impossible to prevent the user from doing that
(due to UNSIGNED-FOREIGN-ADDRESS).
Maybe it's a good reason to withdraw UNSIGNED-FOREIGN-ADDRESS?
(OTOH, IIUC, (:RETURN-TYPE C-POINTER) does just that, so it will not
solve the problem).

> That's exactly what's happening here. I believe you misunderstand
> what's going on in the FFI w.r.t. foreign-addresses etc. 
> Please look at the following. In order to understand, a model of the
> relationship between foreign entities and VALIDP is needed. I did not
> yet write that. 
> 

please sync with the CVS head.
do you want me to provide a nightly CVS head snapshot or is
<http://cvs.sourceforge.net/cvstarballs/clisp-cvsroot.tar.gz>
enough?

BTW, please note what I write in
<https://sourceforge.net/tracker/?func=detail&atid=301355&aid=678216&group_id=1355>
---------------------------------------------------------------------
please fix exec_on_stack to use UNWIND_PROTECT, like you do here.
note also that subr_self is never lost now (see your comment there).
---------------------------------------------------------------------

thanks for your patches - at this time CVS has __EVERYTHING__ you ever
sent me.  It's a __VERY__ good time to sync!

> NIL			;yet they print the same!

this would not be hard to fix.
would you like me to?

> ; or  (mark-invalid-foreign (foreign-address fm))

huh?
shouldn't (c-self fm) be the same (under EQUAL) as fm?!

> NIL		; no problem at all - and that pointer must not be invalidated.

[106]> (foreign-free (unsigned-foreign-address #x0cc1920))

here you should see a segfault - you are releasing the same memory twice

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Marriage is the sole cause of divorce.