Subject: [clisp-list] howto: Finalization in CLISP (was: Ensuring call of cleanup ("des tructor") routine for "C" object)

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Tue Jul 9 06:57:07 2002

Hi,

Kaz Kylheku wrote:
>But really, releasing resources such as open files, sockets or database
>connections at garbage collection time is probably too late! 
>Such things should be explicitly released.

>One model to look at are the WITH-* standard macros in Common Lisp.
>For example WITH-OPEN-FILE associates a new stream with a file,
>evaluates some forms and then esures that the stream is closed.
>These macros are typically based on UNWIND-PROTECT.

The following is part of a possible future article on how to write C code with CLISP (a different article from the one about how to extend CLISP with modules).

FINALIZE is in CLISP (package EXT, see impnotes.html).

FINALIZE in CLISP does not guarantee finalization. Although it is invoked by the garbage collector, it is not used when CLISP exists. This is different from the specification of the Haskell FFI. But this is not the topic of this article.

This article is about the VALID bit that CLISP provides with foreign pointers. This bit can be used to ensure that objects are deallocated/closed at most once. It does not solve the close-on-exit problem.

The CLISP FFI encapsulates objects of type FOREIGN-POINTER (and therefore FOREIGN-ADDRESS, FOREIGN-VARIABLE and FOREIGN-FUNCTION) into a box that has additional bits. One of these is the valid bit and can be accessed from C with fp_valid() and mark_fpr_[in]valid(). This bit can be queried from Lisp using EXT:VALIDP.

I recommend to protect a foreign resource twice using this bit.
 o Once by using WITH-* based unwind-protect macros
 o Once by using this valid bit, as explained here.
This may look like duplicate work. It may make you feel better.

Using this bit has the additional advantage that CLISP will generate an error when trying to access the FOREIGN-* pointer after it has become invalid (like Lisp-level streams).

When using CLISP's FFI FFI:C-POINTER :return-type specification, CLISP creates and returns an object of type FOREIGN-ADDRESS. Your low-level deallocate/close/release() C routine could look like variations on the following:

{ var object faddress = popSTACK(); # FOREIGN-ADDRESS
  TODO_check_type(faddress);
  var object fpointer = TheFaddress(faddress)->fa_base;
  if (fp_validp(TheFpointer(lib))) {
    release/deallocate(Faddress_value(faddress));
    mark_fp_invalid(TheFpointer(fpointer));
    # also? TheFpointer(fpointer)->fp_pointer = NULL;
} }

The finalization code could look as follows:
(finalize x #'(lambda (x) (when (ffi:validp x) (my-package:release x))))
or
(defun close-my-resource (x)
  (check-type x FOREIGN-ADDRESS)
  (when (ffi:validp x) ; like cl:close: don't complain if already released
    (my-package::release x)))
(finalize x #'close-my-resource)

You choose where to put the tests: at the C or Lisp level.

As a counter-example, the current regexp module doesn't use this approach. It doesn't export the release functionality. It always releases via finalization, without checking, so the programmer cannot use regexp-free() him/herself before GC or the system will crash later (attempt to free memory twice). This is a design choice.

User beware: fp_valid() etc. are to be called on the FOREIGN-POINTER object, which is a slot within the FOREIGN-ADDRESS one (pointer indirection, not substructure).

My extensions to the CLISP FFI (not sure it eevn in CVS yet) provide better access to the VALID bit from Lisp: (FFI:MARK-INVALID-FOREIGN foreign-address) and hide the FOREIGN-POINTER vs FOREIGN-ADDRESS dichotomy.
However I didn't provide MARK-FOREIGN-VALID-AGAIN because I feel this is too dangerous (this is left to C code if your 0.01% presumed usage pattern needs that).
If you're writing C code (e.g. niterface stubs for gdi/opengl/xyz), you don't need my extensions. Your code can work with a clisp from 1995.

Examples of usage: CLISP src/rexx.d, foreign.d, my previously posted dynload module and ffi hacks.

If you're interested in the close-on-exit problem, here's what I wrote in my article on extending CLISP via modules:
:Q: Why are there two module initialization, but no exit functions?
:Module exit was deemed superfluous. The OS shall free resources on program exit.
:Adding the functionality would be straightforward. Given the current lack of module :developers, I believe nobody would be bothered by the change of module structure.
:o State you need it.
:o Provide patch (foreign.lisp, spvw_*.d, TODO queens?).
[If you need a copy of that article, mail me]

Regards,
	Jorg Hohle.

---

From: Sam Steingold <[email-masked]>
Date: Tue Jul 9 12:05:35 2002

> * In message <[email-masked]>
> * On the subject of "[clisp-list] howto: Finalization in CLISP (was: Ensuring call of cleanup ("des tructor") routine for "C" object)"
> * Sent on Tue, 9 Jul 2002 15:56:24 +0200
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>
> FINALIZE in CLISP does not guarantee finalization. Although it is
> invoked by the garbage collector, it is not used when CLISP exists.

this is easy to fix - just run all finalizers on exit.  should it be?
what are the issues?  (obviously, the possible errors while running
the finalizers have to be ignored -- what else?)

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
In C you can make mistakes, while in C++ you can also inherit them!