Subject: [clisp-list] truly dynamic FFI in CVS

From: Sam Steingold <[email-masked]>
Date: Tue Aug 5 07:09:11 2003

DEF-CALL-OUT supports a :LIBRARY option which accepts a dll:

(def-call-out command-line
    (:name "GetCommandLineA") (:library "kernel32.dll")
    (:arguments) (:return-type ffi:c-string) (:language :stdc))
(print (command-line))
"d:/gnu/clisp/current/build-g-mingw/base/lisp.exe -B d:/gnu/clisp/current/build-g-mingw -M d:/gnu/clisp/current/build-g-mingw/base/lispinit.mem -N d:/gnu/clisp/current/build-g-mingw/locale -q -norc"

please try this on unix and win32 and report your experience.

--
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Takeoffs are optional.  Landings are mandatory.

---

From: Aurelio Bignoli <[email-masked]>
Date: Thu Aug 7 15:12:02 2003

> please try this on unix and win32 and report your experience.

I had some problem compiling "old style", static modules using CVS
HEAD. It seems that the compiler no longer generates function
prototypes in output C files for foreign fuctions defined with
DEF-CALL-OUT without the :LIBRARY keyword. For example, I had a lot of
messages like the following:

linux.c: In function `module__linux__init_function_2':
linux.c:25: `signgam' undeclared (first use in this function)
linux.c:25: (Each undeclared identifier is reported only once
linux.c:25: for each function it appears in.)
linux.c:26: `environ' undeclared (first use in this function)
linux.c:27: `opterr' undeclared (first use in this function)
linux.c:28: `optind' undeclared (first use in this function)
linux.c:29: `optopt' undeclared (first use in this function)
linux.c:30: `optarg' undeclared (first use in this function)

trying to complile Linux bindings module.

Apart from this problem, the :LIBRARY keyword seems to work very well
under Linux. It is incredible how fast and easy it has become to
interact with external libraries!

There is a problem using the :LIBRARY keyword with DEF-C-VAR: it does
not accept it:

[16]> (def-c-var external-variable
    (:name "external_variable")
  (:library "libshared.so")
  (:type c-string))

*** - Invalid option in 
(DEF-C-VAR EXTERNAL-VARIABLE (:NAME "external_variable")
 (:LIBRARY "libshared.so") (:TYPE C-STRING)): (:LIBRARY "libshared.so")
1. Break [17]> 

You simply forgot to add :library to the option list passed to
PARSE-OPTIONS by DEF-C-VAR:

(defmacro DEF-C-VAR (&whole whole name &rest options)
  (check-symbol (first whole) name)
  (let* ((alist (parse-options options '(:name :type :read-only :alloc) whole))
   ...

---

From: Sam Steingold <[email-masked]>
Date: Thu Aug 7 15:18:02 2003

> * In message <[email-masked]>
> * On the subject of "[clisp-list] truly dynamic FFI in CVS"
> * Sent on Fri, 8 Aug 2003 00:01:55 +0200
> * Honorable Aurelio Bignoli <[email-masked]> writes:
>

John Hinsdale fixed this, I think.
please try again.

> PARSE-OPTIONS by DEF-C-VAR:

fixed, thanks

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
You think Oedipus had a problem -- Adam was Eve's mother.

---

From: "John K. Hinsdale" <[email-masked]>
Date: Thu Aug 7 16:36:19 2003

The module/bindings/glibc (formerly linuxlibc6) is compiling now on my
system at least.

It's taking the C function declarations from the system headers, which
will make it more susceptible to incompatibilities of the different
Linuxes, but also more quick to catch errors brought about by changes
in the system functions.

---
John Hinsdale, Alma Mater Software, Inc., Tarrytown, NY 10591-3710 USA
[email-masked] | http://www.alma.com/staff/hin | +1 914 631 4690

---

From: Arseny Slobodjuck <[email-masked]>
Date: Fri Aug 8 02:44:02 2003

Hi,

> "d:/gnu/clisp/current/build-g-mingw/base/lisp.exe -B d:/gnu/clisp/current/build-g-mingw
> -M d:/gnu/clisp/current/build-g-mingw/base/lispinit.mem -N d:/gnu/clisp/current/build-g-mingw/locale -q -norc"

> please try this on unix and win32 and report your experience.

I tried several system functions with cygwin/gcc and msvc6 built clisp.
Very simple! Great! It's a pity that def-call-in hardly can be made
dynamic while it seems crucial for typical event-driven windows app.

-- 
Best regards,
 Arseny

---

From: Sam Steingold <[email-masked]>
Date: Fri Aug 8 06:07:01 2003

> * In message <[email-masked]>
> * On the subject of "Re: truly dynamic FFI in CVS"
> * Sent on Fri, 8 Aug 2003 19:15:31 +1000
> * Honorable Arseny Slobodjuck <[email-masked]> writes:
>
> I tried several system functions with cygwin/gcc and msvc6 built
> clisp.  Very simple! Great!

please add to modules/bindings/win32/win32.lisp whatever useful
functions you managed to call.

> It's a pity that def-call-in hardly can be made dynamic while it seems
> crucial for typical event-driven windows app.

well, this is caused by the non-dynamic nature of C/C++, right?

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Illiterate?  Write today, for free help!

---

From: Arseny Slobodjuck <[email-masked]>
Date: Fri Aug 8 16:20:03 2003

Hi,

> well, this is caused by the non-dynamic nature of C/C++, right?
But you have already done dynamic parameter/result translation! How
about to made a fixed number of entry points/slots being defined by
def-call-in ? (and clisp being compiled as a shared library)

-- 
Best regards,
 Arseny

---

From: Sam Steingold <[email-masked]>
Date: Sat Aug 9 18:30:04 2003

Hi Arseny,

> * In message <[email-masked]>
> * On the subject of "Re: Re: truly dynamic FFI in CVS"
> * Sent on Sat, 9 Aug 2003 10:27:25 +1000

I am afraid I do not understand what you mean.
(I have never called lisp from C)

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
I'm out of my mind, but feel free to leave a message...

---

From: Arseny Slobodjuck <[email-masked]>
Date: Sun Aug 10 02:38:02 2003

Hello Sam,

> (I have never called lisp from C)
Well, I'm not sure I do. I haven't called lisp from C either, so don't
take it too seriously. As far as I can see, since there is already
dynamic stack handling (translation of lisp call frame to C call
frame - I don't really understand how it works) - a fixed number of
exported functions (slots) can be made. These functions will translate C stack
to lisp stack and call corresponding lisp functions (how to translate and
which lisp functions for each 'slot' should be called is programmed by
def-call-in). So there will no need to recompile clisp as application
is changed. But it requires clisp to be dll.

-- 
Best regards,
 Arseny

---

From: Aurelio Bignoli <[email-masked]>
Date: Mon Aug 11 14:05:10 2003

> The module/bindings/glibc (formerly linuxlibc6) is compiling now on my

on my system I got the usual "undeclared" error for scalb and scalf. I
can't understand the reason, they are declared in bits/mathcall.h,
which is included by math.h.

So I added the two prototypes to linux.c and compiled it without
problems.

---

From: Sam Steingold <[email-masked]>
Date: Mon Aug 11 14:14:07 2003

> * In message <[email-masked]>
> * On the subject of "Re: [clisp-list] truly dynamic FFI in CVS"
> * Sent on Sun, 10 Aug 2003 19:11:02 +0200

they are guarded with

#ifdef __USE_ISOC99

I suggest adding "-std=c99" to CFLAGS in Makefile.

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Programming is like sex: one mistake and you have to support it for a lifetime.

---

From: "John K. Hinsdale" <[email-masked]>
Date: Mon Aug 11 14:25:07 2003

> which is included by math.h.

Looking at bits/matchcalls.h, the declarations for scalb() are only
included-in by the preprocessor if either of these macros are set;

   # if defined __USE_MISC || defined __USE_XOPEN_EXTENDED

These get set only by the compiler, and only if you #define
_GNU_SOURCE before the first system file is included (see my post to
the devel. list below).

It's an open issue among the developers how to deal with this.

 > So I added the two prototypes to linux.c and compiled it without

That is pretty much all you can do at the moment.

----  Begin forwarded message -------

>From hin Fri Aug  8 16:17:24 -0400 2003
From: "John K. Hinsdale" <[email-masked]>
To: [email-masked]
Subject: Re: Problems in modules/glibc/linux.lisp

 > Can you replace
 >         ; You can uncomment this if your compiler sets __USE_GNU
 > with
 >         (c-lines "#define __USE_GNU~%")

It turns out you cannot set __USE_GNU directly; it is internal to the
compiler and treated very specially.  It can be inspeced with #ifdef,
but it can only be turned on by the compiler (#define __USE_GNU does
nothing), and even then only by #define'ing

	_GNU_SOURCE

prior to the VERY FIRST #include of any system header file.  (Much
experimentation has led me to these facts).  This also triggers other
defines that bring in BSD and POSIX calls.  This is all for my Linux
2.4.20 environment, with gcc 2.95.4.  This is a pretty common
environment as many are still clinging tightly to GCC 2.95.

Now, it does not work to just do

	(c-lines "#define _GNU_SOURCE~%")

at the start of linux.lisp, because in the generated file "C"
file, the FFI sticks
	#include "clisp.h"
before anything generated via C-LINES.  And "clisp.h" in turn does a
	#include <stdlib.h>
thereby ensuring that __USE_GNU will not be set for the build of the
module.

Pretty messy, eh?

So we either need to:

1 - live without the function calls that are exported
    only when __USE_GNU is set.  Not desirable?

2 - Set _GNU_SOURCE buildwide in "clisp.h".  I don't know
    whether this is a good idea.  Your call.
    
3 - Give the FFI away to let the module stick lines in
    the output that will come before the 
	#include "clisp.h"

Please let me know which option you prefer.

I should at least fix the comments in the Lisp file reading "if your
compiler sets __USE_GNU" are nonsensical: the end user sets __USE_GNU
indirectly, not the compiler.

---

---

From: "John K. Hinsdale" <[email-masked]>
Date: Tue Aug 12 06:28:04 2003

>> [Aurelio Bignoli]

 > [Sam Steingold]

FYI on my GCC 2.95.4 -std=c99 is not a valid option.  There seems to
be yet another #define of _ISOC99_SOURCE required to turn things on in
<feature.h>, however it also seems to have the odd behavior that it
doesn't work unless its set before the very first system include.
Which means adding -D_ISOC99SOURCE to the Makefile won't work.

More info:
http://sources.redhat.com/ml/libc-alpha/2000-07/msg00249.html
http://gcc.gnu.org/c99status.html

---

From: Sam Steingold <[email-masked]>
Date: Tue Aug 12 06:45:07 2003

> * In message <[email-masked]>
> * Sent on Tue, 12 Aug 2003 08:57:50 -0400
> * Honorable "John K. Hinsdale" <[email-masked]> writes:
>
>    >> [Aurelio Bignoli]

I don't understand.
I think that -D_ISOC99SOURCE has the effect of defining this CPP macro
"before the very first system include".

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
If it has syntax, it isn't user friendly.

---

From: "John K. Hinsdale" <[email-masked]>
Date: Tue Aug 12 07:11:01 2003

> "before the very first system include".

Yes, I hadn't had my coffee this morning :) and I was about to offer
an embarrassed retraction, then for grins I went and compiled a test
program (again w/ gcc 2.95.4), and I've observed that using
-D_ISOC99_SOURCE on the command line does *not* trigger the
__USE_ISOC99 guarded stuff, whereas adding

	#define _ISOC99_SOURCE
	#include <stdlib.h>

does define __USE_ISOC99, but again only if the #define appears before
any includes like <stdlib.h>.  This is just what I observe on my
ancient stock Debian "woody" GCC 2.95.4 ... weird.

Reading threads on Usenet and the GCC list it appears that all this
__USE_THIS and __USE_THAT is very compiler version dependant.  In
modules/glibc would we be better off just leaving the "nonstandard"
sys calls just commented out and let interested users enable them?

My own opinion is that the defaul build of the module should be such
that it is very likely to succeed on the users's platform even at the
expense of leaving out some calls he might need find absent and need
to enable manually.

Also, may I ask what version of GCC Linux CLISP users are using?  I've
been living in fear of GCC 3 for some time now having had a lot of
problems w/ hte early versions.

---

---

From: Sam Steingold <[email-masked]>
Date: Tue Aug 12 07:47:03 2003

> * In message <[email-masked]>
> * Sent on Tue, 12 Aug 2003 09:30:19 -0400
> then for grins I went and compiled a test program (again w/ gcc
> 2.95.4), and I've observed that using -D_ISOC99_SOURCE on the command
> line does *not* trigger the __USE_ISOC99 guarded stuff, whereas adding
> 
> 	#define _ISOC99_SOURCE
> 	#include <stdlib.h>
> 

indeed.  are you sure this is indeed the only way?
can you ask the GCC people?
there might be a gcc command line option to enable this stuff.

> sys calls just commented out and let interested users enable them?

ok.

> expense of leaving out some calls he might need find absent and need

I guess we could make autoconf handle this...

> problems w/ hte early versions.

I have been a happy user of gcc3 since it became standard on RH.
I now use gcc 3.3 with RH and cygwin/mingw.

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
"Syntactic sugar causes cancer of the semicolon."	-Alan Perlis

---

From: Aurelio Bignoli <[email-masked]>
Date: Thu Aug 14 09:53:13 2003

> problems w/ hte early versions.

I'm using GCC 3.2.2.