Subject: Win32 FFI, etc.

From: "Paul Moore" <[email-masked]>
Date: Sun, 18 Jul 1999 18:07:35 +0100

I've just started using CLisp on Win32, and one thing I want to be 
able to do is to call Win32 procedures from CLisp. The FFI as 
described in the implementation notes seems very comprehensive, 
but I can't seem to work out how to call a DLL function. (The 
documentation talks about generating and compiling C, which I 
presume is more relevant to Unix than to Win32...?)

Is there a simple example anywhere of how to (say) call the Win32 
MessageBox function (from USER32.DLL) from clisp?

Going a bit further, is there an interface from CLisp to COM, so that 
I could call COM servers (like Excel, or the ActiveX Data Objects 
interface) from CLisp?

If these things don't exist, I would certainly be interested in looking 
at adding them to CLisp - it sounds like the sort of project I could 
get my teeth into! (I know I said I've only just started with CLisp, 
but I've written similar things for other languages in the past). Can 
anybody give me any pointers on how I should do this (I've read the 
"extending CLisp" document, which is a good start, but I'd like to 
hook into the existing FFI stuff...)

Thanks,
Paul Moore.

PS Actually, what would be really good would be CLisp as an 
ActiveX Scripting engine, which would allow Lisp to be used as a 
Windows scripting language, in the Windows Scripting Host or on 
web pages... That would do some serious good to the visibility of 
Lisp on the Windows platform! (IMHO...)

---

From: Eric Marsden <[email-masked]>
Date: Mon, 19 Jul 1999 11:03:02 +0200

> presume is more relevant to Unix than to Win32...?)

yes, the CLISP FFI is designed for calling out to static objects which are
linked with CLISP. There is currently only poor support for dynamic linking:
the tricky bit is passing parameters to and from foreign code with
appropriate calling conventions. A hacky way of doing this is to implement a
call stack in C; this is the approach I used for the current dynamic loading
support for Unix CLISP: see 

   <URL:http://www.chez.com/emarsden/downloads/dl0.1.tar.gz>
   
A better approach is to use the callout code included with the CLISP FFI
(see files foreign.d and foreign.lsp in the distribution).

Improvements in these aspects would be extremely useful to Windows users.

-- 
Eric Marsden
It's elephants all the way down

---

From: "Paul Moore" <[email-masked]>
Date: Mon, 19 Jul 1999 22:43:11 +0100

On 19 July 1999 09:59, Eric Marsden said:

> linked with CLISP. There is currently only poor support for
> dynamic linking: the tricky bit is passing parameters to and from foreign
> code with appropriate calling conventions. A hacky way of doing this is to
> implement a call stack in C; this is the approach I used for the current
> dynamic loading support for Unix CLISP: see
>

Agreed. While I could (& have done) implement FFI callout code building my
own stack frame, that's basically what AVCALL does (and probably better than
I could!) Since the AVCALL stuff (just) takes a function pointer, I don't
see that it should be that hard to provide such a pointer using Windows'
LoadLibrary() and GetProcAddress() functions.

Of course, as a CLisp newcomer, I have a hard time connecting lisp-level
constructs with C-level implementation, but the idea seems very simple.

Have I missed a key point? If it really is that simple, how come it hasn't
been implemented? Lack of interest? (Surely not!)

> Improvements in these aspects would be extremely useful to Windows users.

Actually, can someone tell me how to do a STATIC FFI call in Windows? If I
set up the div() example as in the implementation notes, and then compile
(lisp -M lispinit.mem -c div.lsp) the result is a div.c file in my current
directory. OK, what now? The impnotes file discusses making "module sets"
under Unix/Linux, but not under Windows.

Hmm. The more I look at it, the more I think that my biggest hurdle is
likely to be getting to a point where I understand how things hang together,
and how to make a change - not what change to make... Is there any
documentation on how the CLisp implementation is structured, and how the
internals work, which an interested hacker could look at?

Thanks,
Paul.

---

From: Eric Marsden <[email-masked]>
Date: Tue, 20 Jul 1999 09:50:59 +0200

> been implemented? Lack of interest? (Surely not!)

None of the CLISP developers use Windows. I think it's that simple. There
have however been several requests from users.

> internals work, which an interested hacker could look at?

there are some general comments (in German :) in lispbibl.d. I don't know of
anything else, unfortunately (my german isn't very good).

---

From: Joerg Hoehle <[email-masked]>
Date: Tue, 20 Jul 1999 11:36:02 +0200 (MET DST)

> under Unix/Linux, but not under Windows.

Well, I have no knowledge about MS-Win so I'm certainly qualified to
respond :-), yet I'd say that you need either the source distribution
that you compiled yourself or a distribution of CLISP inside a library
archive that contains all of CLISP's *.o except for modules.o

You will need to recompile modules.d after adding your module definition
to modules.h (one line).  Then you link your object files together with
the newly compiled modules.o and the rest of the libclisp.a into a new
lisp.run.

>From what I understood, compiling (-c or #'compile-file) a file with FFI
definitions merely creates stubs, you may or may not have to write a
module interface file.

And you must still link statically yourself.

> likely to be getting to a point where I understand how things hang together,
You don't need to peek at all of the xMB source.

> internals work, which an interested hacker could look at?
Just ask.  It would have been nice though if the amount of explanations
that went through the lists on different topics over the years could have
found its way into some distributed documentation as well...

Regards,
	Jo"rg Ho"hle.
[email-masked]		http://fit.gmd.de/~hoehle/amiga.html

---

From: "Paul Moore" <[email-masked]>
Date: Tue, 20 Jul 1999 20:34:28 +0100

> From: [email-masked] On Behalf Of Eric Marsden

> have however been several requests from users.

Fair enough. I assumed that the fact that clisp worked so well on Windows
meant that there was an expert involved somewhere.

Time for me to be that expert? :-)

Paul.

---

From: "Paul Moore" <[email-masked]>
Date: Tue, 20 Jul 1999 20:34:22 +0100

> From: [email-masked] On Behalf Of Joerg Hoehle
>

Hmm. Thanks for the pointer - I'll look into this a bit more. If I get it
working, I'll post what I did - maybe it'd be useful for the impnotes file?

> And you must still link statically yourself.

OK, I think I see how it works. I'll try to get an example working.

> found its way into some distributed documentation as well...

Thanks for the offer. I probably will :-)

The obvious way of doing foreign calls is just as a pure "extra" module,
using the existing extension mechanism. So, we have

    (def-dll-call MessageBox (:dll user32) (:params ...) (:return ...))
    (MessageBox ...)

But the existing FFI stuff is very complete, in terms of defining parameter
and return types, etc etc. It would be nice to be able to reuse the
technology, and to do

    (def-call-out MessageBox
        (:dll user32)
        (:arguments (hwnd int) (text c-str) (title c-str) (flags int))
        (:return-type int)
        (:language :stdc-stdcall))
    (MessageBox 0 "Hello, world" "Clisp" 0)

The only differences are
    (1) the extra :dll parameter
    (2) the fact that def-call-out works immediately,
        not generating a C file at compile time

It's only point (2) that I have a problem with. Maybe I can define
def-dynamic-call-out which works at runtime instead of compile time. That's
not a problem. But if I need to define a struct parameter type, then do I
also need a def-dynamic-c-struct? And so on...

Basically, I'm sort of asking if the FFI stuff is reusable in this way at
all, or should I just define my own interface?

Thanks,
Paul.

---

From: Joerg Hoehle <[email-masked]>
Date: Thu, 22 Jul 1999 09:33:12 +0200 (MET DST)

> using the existing extension mechanism. So, we have
Yes, I wrote several such modules for the Amiga.

> all, or should I just define my own interface?

It's much better not to define yet another interface and use the existing
stuff, although I have myself defined a different FFI that I called AFFI
for AmigaOS.  Reasons are different design decisions (esp. small
footprint), and my stuff historically precedes the FFI (by little).

The extra :dll parameter is no problem I'd say, yet you could even name
that :library instead of the MS-Win specific :dll (to try to look more
generic (maybe this is a bad idea).  Then I'd suggest using strings as
names instead of symbols for the library name 'user32: you'd not get case
problems on UNIX, the day this mechanism is adapted to load UNIX .so
files.

The FFI has already been modified slightly in the past to allow for 
AmigaOS specific extensions, like registerized argument specifications
instead of stack interfaces, and also for dynamic library functions.

I think you should closely look at the #ifdef AMIGAOS in FOREIGN.D and
FOREIGN1.LSP. There is a mechanism there to define functions which are
relative to some library base that is dynamically defined and different
from run to run, instead of being static in the executable like modules.

I'd expect that something similar could be used for functions relative to
a .dll.

I invite you to see whether you can reuse part of that code, or modify it
so that it integrates both conceptions of shared run-time libraries.

> also need a def-dynamic-c-struct? And so on...

IIRC, def-c-struct doesn't generate stubs, it generates Lisp-level stuff
that is able to convert a C struct to a Lisp-level structure and
vice-versa using FFI/ffcall support functions. Only stubs for function
calls might be needed, but the FFI contains enough code to be able to
convert a C struct to a Lisp structure :in and :out.  For example, 

FOREIGN1.LSP: #+AMIGA DEF-LIB-CALL-OUT generates no C stubs.

Regards,
	Jo"rg Ho"hle.
[email-masked]		http://fit.gmd.de/~hoehle/amiga-clisp.html