Subject: Re: AW: AW: [clisp-list] Debugging clisp FFI calls into a newly created (c++) DLL

From: Arseny Slobodjuck <[email-masked]>
Date: Mon Jun 7 20:21:29 2004

> Do you know where I can find this thing, or what is the easiest
> way to accomplish the callback?
See the Example 30.10 in impnotes. You do not need def-call-in.
(If I understand you correctly).

-- 
Best regards,
 Arseny

---

From: "Andreas Thiele" <[email-masked]>
Date: Tue Jun 8 03:01:25 2004

>... See the Example 30.10 in impnotes. You do not need def-call-in.

Thanks for the hint.

Actually this is what I am doing. I think you are right, this is a simple
and correct way. It seems, this automatically installs begin_callback and
end_callback and passes a trompoline referrence. (Meanwhile I wondered about
http://sourceforge.net/mailarchive/message.php?msg_id=2434253 and thought I
had to use begin/end_callback)

I call my callback function from within a dll in the windows message loop,
so lisp can see the messages. This seems to work fine if the code is small.
The thing I am nervous about is, I did (print message) or (format t ...
anything ...) inside my callback function. This crashes the program from
time to time when garbage collection is executed. But not everytime! If (gc)
is include it crashed definitely after a few calls. So my question is, what
is allowed and what isn't? Why crashes (format t "~a ~a ~a ~a~%" hWnd
Message wParaem lParam) clisp totally? (after hundreds of successful calls)

I'll try to figure this out by debugging the clisp debug version. If you or
anybody has an (easy) answer, I'd be happy to know it.

Thanks
Andreas

---

From: Sam Steingold <[email-masked]>
Date: Tue Jun 8 07:31:07 2004

> * Andreas Thiele <[email-masked]> [2004-06-08 12:00:44 +0200]:
>
> I call my callback function from within a dll in the windows message
> loop, so lisp can see the messages. This seems to work fine if the
> code is small.  The thing I am nervous about is, I did (print message)
> or (format t ...  anything ...) inside my callback function. This
> crashes the program from time to time when garbage collection is
> executed. But not everytime! If (gc) is include it crashed definitely
> after a few calls. So my question is, what is allowed and what isn't?
> Why crashes (format t "~a ~a ~a ~a~%" hWnd Message wParaem lParam)
> clisp totally? (after hundreds of successful calls)

You are probably encountering a GC-safety bug in your code.
<http://clisp.cons.org/impnotes/gc-safety.html>.

Basically, you write your C code like this:

           object my_variable;
           /*assign something to my_variable*/
           pushSTACK(my_variable); /* save my_variable to the STACK */
           begin_callback(); /* save registers */
           <invoke your callback: this invalidates registers
            and, if it may trigger GC, may invalidate object variables>
           end_callback(); /* restore registers */
           my_variable = popSTACK(); /* restore my_variable from STACK */

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
I don't want to be young again, I just don't want to get any older.

---

From: "Andreas Thiele" <[email-masked]>
Date: Tue Jun 8 09:25:09 2004

<...

           object my_variable;
...>

Thanks a lot for your help. I just can't figure out how to compile this (on
win32/msvc 6.0). Until now (FFI:DEF-CALL-OUT used only) everything was easy.
Do I have to write a module? This can only be achieved using cygwin on
Win32?

Until now I did:

...

(FFI:DEF-CALL-OUT SetMsgCallBack
  (:library "ghwemu.dll")
  (:language :stdc)
  (:name "SetMsgCallBack")
  (:arguments (function-arg
    (ffi:c-function
     (:arguments (hWnd ffi:sint32) (Message ffi:uint16) (wParam ffi:sint16)
(lParam ffi:sint32))
     (:return-type ffi:sint32) (:language :stdc))))
  (:return-type sint32))

(defun WndProc (hWnd Message wParam lParam)
  (let ((m (make-msg)))
    (setf (msg-hWnd m) hWnd)
    (setf (msg-Message m) Message)
    (setf (msg-wParam m) wParam)
    (setf (msg-lParam m) lParam)
    ;following crashes occasionally
    (print m)
    ;(push m *msg*) works if used instead of (print m)
    )
  0)

(SetMsgCallBack #'WndProc)

...

c-code contains:

...

typedef int (*LispMsgFunc)(HWND, UINT, WPARAM, LPARAM);
LispMsgFunc fMsg;

long SetMsgCallBack(LispMsgFunc mcb)
{
	WaitForSingleObject(info_call, INFINITE);
	fMsg = mcb;
	ReleaseMutex(info_call);
	return (long)mcb;
}

LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM
lParam)
{
...
	WaitForSingleObject(info_call, INFINITE);
	// call lisp if avail
	if ((void*) fMsg) fMsg(hWnd, message, wParam, lParam);
	ReleaseMutex(info_call);
...

This is not GC save?

Can I use FFI:DEF-CALL-IN btw. the by compilation generated *.c file? I
don't find clisp.h which is included in the first line of the *.c file.
Neither can I make nmake create genclisph.exe. Perhaps it is much easier to
write gc save lisp code? Isn't there any tutorial? I'd be willing to write
one.

I find example 30.10 from impnotes somehow misleading. No begin_callback
etc. is included automatically? So this works only in this special situation
(calling back from within calling out)! And the code must not trigger gc?(Of
course this is already helpful, because one can use any lisp function from
c.)

Rem: I already read before <http://clisp.cons.org/impnotes/gc-safety.html>
and still find it difficult to understand.

Any help or hint is appreciated.

Thanks
Andreas

---

From: Sam Steingold <[email-masked]>
Date: Tue Jun 8 09:49:04 2004

> * Andreas Thiele <[email-masked]> [2004-06-08 18:24:43 +0200]:
>
> Do I have to write a module?
I don't think so.

> This can only be achieved using cygwin on Win32?
no, modules are fully supported using mingw also.

> This is not GC save?
I think it's OK.

> and still find it difficult to understand.

some things _are_ complicated.  this is one of them.

> Any help or hint is appreciated.

it would be very nice if you could do the following:

1. get mingw (e.g., as a part of cygwin).

2. build CLISP like this:
$ CC=g++ ./configure --with-debug build-g-gxx

3. run CLISP under GDB and try your code:
$ cd build-g-gxx
$ gdb
(gdb) run
[1]> (def-call-... ....)

you should get an abort at the first invalid access.
please send the backtrace.

Thanks a lot!

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Winners never quit; quitters never win; idiots neither win nor quit.

---

From: "Andreas Thiele" <[email-masked]>
Date: Tue Jun 8 10:01:15 2004

...

Thank you. I will do it.

(Rem: Hope you didn't get me wrong, I did't want to be impolite. I am very
happy about clisp and find it a great thing. Just fighting for days against
this callback thing only)

Andreas

---

From: Sam Steingold <[email-masked]>
Date: Tue Jun 8 10:57:08 2004

> * Andreas Thiele <[email-masked]> [2004-06-08 18:59:55 +0200]:
>
> (Rem: Hope you didn't get me wrong, I did't want to be impolite. I am
> very happy about clisp and find it a great thing. Just fighting for
> days against this callback thing only)

Yes, I understand, and all CLISP developers will greatly appreciate it
if you will help us nail this bug!

Happy hacking!

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
I'd give my right arm to be ambidextrous.

---

From: "Andreas Thiele" <[email-masked]>
Date: Fri Jun 18 11:46:01 2004

I am working on this. Until now I was not able to produce a working
cygwin/mingw w32 clisp build to run gdb on it. This seems to be a very
very difficult thing. With 

CC=g++ ./configure --with-debug --with-mingw

I was able to create the executable. But it did not pass the test suite.
Isn't there any thorough guide on how to build clisp for windows with
mingw?

Andreas

---

From: Sam Steingold <[email-masked]>
Date: Fri Jun 18 13:50:04 2004

> * Andreas Thiele <[email-masked]> [2004-06-18 20:45:19 +0200]:
>

you should also pass a directory argument

> I was able to create the executable. But it did not pass the test

which tests fail?
can you give us a gdb backtrace? (I assume abort)

> Isn't there any thorough guide on how to build clisp for windows with

it amounts to
$ ./configure --without-cygwin --with-debug --build build-g
to configure and build in the "build-g" directory

you are welcome to write a better guide :-)

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
If you try to fail, and succeed, which have you done?

---

From: "Andreas Thiele" <[email-masked]>
Date: Sun Jun 20 14:24:04 2004

>to configure and build in the "build-g" directory

I did the following

$ ./configure --without-cygwin --with-debug build-g-200

and followed the instruction at the end (-200 is an random label, no
meaning for me). Thus I typed

$ cd build-g-200
$ ./makemake --with-dynamic-ffi --verbose --win32gcc debug > Makefile
$ make config.lisp
$ make

Clisp was build but the process ends with:

g++ -mno-cygwin -I/usr/local/include -W -Wswitch -Wcomment
-Wpointer-arith -Wimp
licit -Wreturn-type  -Wno-sign-compare -falign-functions=4 -D_WIN32 -g
-DDEBUG_O
S_ERROR -DDEBUG_SPVW -DSAFETY=3 -DDEBUG_GCSAFETY -DNO_TYPECODES
-DCINTERFACE -DU
NICODE -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI -DNO_GETTEXT -I. -x none
-DUSE_CLISP_H
=0 -DCOMPILE_STANDALONE clisp-test.c -o clisp-test-lispbibl.exe
In file included from clisp-test.c:5:
lispbibl.d:7223: warning: volatile register variables don't work as you
might
   wish
clisp-test-clisp > clisp-test-clisp.out
clisp-test-lispbibl > clisp-test-lispbibl.out
cmp clisp-test-clisp.out clisp-test-lispbibl.out
clisp-test-clisp.out clisp-test-lispbibl.out differ: char 1641, line 93
make: *** [clisp.h] Error 1

Thiele@atp-ws-02 ~/clisp/build-g-200
$

I think clisp was built but did not pass a test. So what can be the
problem? My installation of cygwin? I can start this clisp even with
gdb. This clisp crashes at some other point when loading my sources. I
don't wonder because it seems to be no correct clisp.

I think my problem is setting up cygwin. In unix/INSTALL four libs are
mentioned.

   * GNU libsigsegv
   * GNU libiconv
   * GNU readline
   * GNU gettext

I only was able to build libsigsegv and could not built the other three.
This is what I mean when thinking about a guide. Which version of
Cygwin, which gcc version, how to install, how to check or verify
versions, how to built these libs etc..

Any hints?

Andreas

---

From: "Andreas Thiele" <[email-masked]>
Date: Sun Jun 20 14:42:03 2004

I forgot to mention, I tried to built 2.33.1. Might this be the problem?

---

From: Sam Steingold <[email-masked]>
Date: Mon Jun 21 06:57:00 2004

> * Andreas Thiele <[email-masked]> [2004-06-20 23:23:29 +0200]:
>

> I think clisp was built but did not pass a test.

clisp was build, but reported that it cannot build modules in C++ mode.
unfortunately, this is to be expected.
I hope Bruno will fix this.

If you wish, please run the CLISP internal testsuite with "make check".

> So what can be the problem?

no problem whatsoever!

now, start CLISP under gdb (cd build-g-200, gdb, run) and start testing
your code.  if indeed, as we expect, this is a GC-safety bug, you will
get an abort which will pinpoint the first place where an invalid
object was used.

Specifically, there is a global integer variable alloccount and the
object structure contains an integer allocstamp slot. If these two
integers are not the same, the object is invalid.  By playing with gdb,
you should be able to figure out the precise spot where a GC increments
alloccount after the object has been retrieved from a GC-visible
location.

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Incorrect time syncronization.