Subject: [clisp-list] on FFI:mark-invalid-foreign, malloc() and finalizati on

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Fri Mar 14 04:54:04 2003

Sam Steingold wrote:
>> >whatever is free()d, must be marked invalid.
>> Here you're killing the messenger together with the message.
>My point is: do we want to allow users to free() the same memory twice?
>Yes or No?
>I am not forcing your decision here, but I want clarity.
I don't want that to happen, and provide means to prevent it from happening: MARK-INVALID-FOREIGN. But I cannot prevent it.

>IIUC, it is impossible to prevent the user from doing that
>(due to UNSIGNED-FOREIGN-ADDRESS).
Not because of u-f-a. Because the C code could do it.
Memory allocation needs a protocol which specifies who allocates and who frees. CLISP explicitly supports protocols where Lisp does one part, and the foreign code does the other part.

>Maybe it's a good reason to withdraw UNSIGNED-FOREIGN-ADDRESS?
It's a tiny utility function which some people may use some day. It's like CMUCL's SAP-INT and INT-SAP.
I prefer to provide them instead of having programmers define their own version using:

(def-call-out u-f-a
  (:name "ffi_identity")
  (:arguments (obj uint32)) ;maybe uint64 is needed?
  (:return-type c-pointer) (:language :stdc))
;(u-f-a #xabadcafe) -> #<FOREIGN-ADDRESS #xABADCAFE>

BTW, I was just pleased to see that
(disassemble #'u-f-a) might work on UNIX (if gdb were installed here) because sys::code-address-of recognizes #<FOREIGN-FUNCTION>.
I.e., for foreign-functions
#+UNIX(sys::code-address-of ff) == (foreign-address-unsigned ff)

>[106]> (foreign-free (unsigned-foreign-address #x0cc1920))
>here you should see a segfault - you are releasing the same 
>memory twice
I'll not add a testcase for that:)

>> [87]> (equalp * **)
>> NIL			;yet they print the same!
>this would not be hard to fix.
>would you like me to?
No, until there's a concept of what shall be done. The two objects are different in terms of EQUALP:
#<foreign-address pointer:0 offset #x00CC1920>
#<foreign-address pointer #x00CC1920 offset 0>

So far, my opinion is that the current printed representation of #<FOREIGN-ADDRESS #x00CC1920> is just fine (suited to most user needs), and the user need not be bothered with additional level of detail.

>> [93]> (mark-invalid-foreign fm)
>> ; or  (mark-invalid-foreign (foreign-address fm))
>> T
>huh?
>shouldn't (c-self fm) be the same (under EQUAL) as fm?!
Same as above.

Here's (part of) the underlying concept (= documentation of the current state, which needs to find its way into impnotes in a polished form):

1. The validp bit is associated with a FOREIGN-POINTER object, not each FOREIGN-ADDRESS or FOREIGN-VARIABLE.

2. FOREIGN-ADDRESS, F-VARIABLE, F-FUNCTION embody such a FOREIGN-POINTER object -- and an offset from it.

3. All of (:return-type C-POINTER), U-F-A etc. return a FOREIGN-ADDRESS object which all share what I call the "FFI session pointer" - O(fp_zero). That's why this one shall not be invalidated.

4. WITH-FOREIGN-OBJECT/STRING and DEEP-MALLOC return f-a/v objects based on a new, thus distinct foreign-pointer.
Only these can be marked as invalid, which the WITH- macros do automatically (not all of them right now, as you noticed).

5. DEREF, SLOT etc. (or rather ffi::%DEREF etc.) preserve the foreign-pointer when building a new FOREIGN-VARIABLE object.
FOREIGN-ADDRESS preserves it as well (which your (compose #'u-f-a #'f-a-u) suggestion does not).
That's both useful and potentially dangerous.
(foreign-free (slot X 'part1)) shall free, but not invalidate the pointer, because all of X would become invalid.

My dynload patch creates FOREIGN-FUNCTION objects embedding the FOREIGN-POINTER to the library. That's how, by invoking MARK-INVALID-FOREIGN on the library base pointer obejct, all associated functions and variables become invalid as well.

My concept for SET-FOREIGN-BASE was to permit modification of this association between a FOREIGN-ADDRESS/VARIABLE/FUNCTION and the associated FOREIGN-POINTER.
Then FINALIZE and/or MARK-INVALID-FOREIGN become useful with arbitrary f-v/f-a objects.

(let ((x (c-self fm)))
  ;; dissociate from CLISP FFI session pointer
  (set-foreign-base x (another-foreign-entity foo))

  ;; either
  (finalize (foreign-base-pointer x) #'xx)
  ... do stuff using foo
  ;; or
  (foreign-free yy)
  (mark-invalid-foreign foo); x becomes invalid as well.
)

Function: FOREIGN-BASE == FOREIGN-POINTER, but I prefer the "base" idea.
Function: (SET-FOREIGN-BASE foreign-entity another) (you'll suggest (SETF FOREIGN-BASE), I know :)
another may also be :NEW, or :SESSION, not just a foreign-entity, which is IMHO not compatible with the SETF idea. Returns the modified entity. Maybe name it update-foreign-base?
(let ((x (set-foreign-base (c-self xx) :new)))
  ;; now x has a distinguished FOREIGN-BASE object
  (finalize (foreign-base x) #'close-windows-silently)
  ...)

Another reason for the SET-FOREIGN-BASE name instead of (SETF FOREIGN-POINTER) is that it's not only the pointer that's changed. The offset of the FOREIGN-ADDRESS object is modified suitably to preserve the FOREIGN-ADDRESS-UNSIGNED value.

But that's not all that would be needed. You may also wish to correct an address (e.g. when trying to restore a defunct shared library function from a .mem file in a new lisp session). It's not ready yet -- a basic too many open issues in brain error.

Regards,
	Jorg Hohle.

---

From: Sam Steingold <[email-masked]>
Date: Fri Mar 14 07:24:03 2003

> * In message <[email-masked]>
> * On the subject of "[clisp-list] on FFI:mark-invalid-foreign, malloc() and finalizati on"
> * Sent on Fri, 14 Mar 2003 13:52:52 +0100
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>
> Sam Steingold wrote:
> >> [87]> (equalp * **)

I don't understand the need for FOREIGN-ADDRESS to begin with.
Why isn't FOREIGN-POINTER enough?

> So far, my opinion is that the current printed representation of
> #<FOREIGN-ADDRESS #x00CC1920> is just fine (suited to most user
> needs), and the user need not be bothered with additional level of
> detail.

Well, I would suggest printing:

#<FOREIGN-ADDRESS #<FOREIGN-POINTER #x00CC1920> + 123>

and equality:

EQUAL:
 + FOREIGN-POINTERs are EQUAL when their fp_pointer's are =
 + FOREIGN-ADDRESSes are EQUAL when their fa_base are EQUAL and
   fa_offset's are =

EQUALP:
 + additionally, FOREIGN-ADDRESSes are EQUALP when their
   Faddress_value's are =

> 3. All of (:return-type C-POINTER), U-F-A etc. return a
> FOREIGN-ADDRESS object which all share what I call the "FFI session
> pointer" - O(fp_zero). That's why this one shall not be invalidated.

why???  why do they return
#<FOREIGN-ADDRESS #<FOREIGN-POINTER 0> + #x00CC1920>
and not
#<FOREIGN-ADDRESS #<FOREIGN-POINTER #x00CC1920> + 0>
to keep them bound to a session?
but #<FOREIGN-POINTER #x00CC1920> can be bound to a session as well!
or maybe you want to be able to invalidate them all in one stroke?

> My dynload patch creates FOREIGN-FUNCTION objects embedding the
> FOREIGN-POINTER to the library. That's how, by invoking
> MARK-INVALID-FOREIGN on the library base pointer obejct, all
> associated functions and variables become invalid as well.

aha!

> My concept for SET-FOREIGN-BASE was to permit modification of this
> association between a FOREIGN-ADDRESS/VARIABLE/FUNCTION and the
> associated FOREIGN-POINTER.  Then FINALIZE and/or MARK-INVALID-FOREIGN
> become useful with arbitrary f-v/f-a objects.

so you want a setfable accessor FOREIGN-POINTER, right?
That's easy,

> Function: FOREIGN-BASE == FOREIGN-POINTER, but I prefer the "base"

Occam's razor - do not multiply entities (here, symbols).

> (SETF FOREIGN-BASE), I know :)

of course! - this is CL, not scheme!

> another may also be :NEW, or :SESSION, not just a foreign-entity,
> which is IMHO not compatible with the SETF idea. Returns the modified
> entity. Maybe name it update-foreign-base?

(defun (setf foreign-pointer) (new) (f-ent)
  (if (eq f-ent :session)
    (let ((fp (allocate-fpointer)))
      O(fp_zero) = fp;
      TheFpointer(fp)->fp_pointer = new;
      return fp;)
    TheFpointer(foreign_pointer(f-ent))->fp_pointer = new))

> Another reason for the SET-FOREIGN-BASE name instead of (SETF
> FOREIGN-POINTER) is that it's not only the pointer that's changed. The
> offset of the FOREIGN-ADDRESS object is modified suitably to preserve
> the FOREIGN-ADDRESS-UNSIGNED value.

That might not be possible.
fa_offset is a uintP.

Suppose you have
fa = #<FOREIGN-ADDRESS #<FOREIGN-POINTER 0> + 1>

what should

(setf (foreign-pointer fa) 3)

do?

fa = #<FOREIGN-ADDRESS #<FOREIGN-POINTER 3> + -2>

?!!

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Whom computers would destroy, they must first drive mad.