Subject: [clisp-list] CLISP FFI and (lack of) unicode support (was: dynamic loading of shared libraries/objects)

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Wed Apr 10 09:10:26 2002

Hi,

Reini Urban,
> > # don't -DUNICODE sysdll.c via CFLAGS or you'll get 
> LoadLibraryW(), not -A
> 
> but then you cannot load foreign libraries on win32 (namely 
> japanese, chinese,

Thanks for raising this topic.

CLISP's FFI doesn't correctly support UNICODE anyway. That's another one on my growing FFI-limits list.

> (setf custom:*foreign-encoding* (ext:make-encoding :charset 'charset:utf-8))
*** - SYSTEM::SET-FOREIGN-ENCODING: #<ENCODING CHARSET:UTF-8 :UNIX> is not a 1:1 encoding

The problem is C-STRING etc. It's currently limited to "1:1" translations.
Changing that is not easy at all. Otherwise it would have been done since day one.

I *highly* welcome proposals for how an API which works with unicode would look like.

I've been thinking about things like
(with-c-string (varname "foobar" &optional charset-or?-encoding)
  &body ...)
-> FOREIGN-VARIABLE of type (C-ARRAY FFI:uint8-or-character-or-what? <len>
(FFI::FOREIGN-SIZE *) -> length in 8bit bytes
or something closer to make-array with &key :length :initial-contents ...

That would do to stack allocation of foreign strings what ext:convert-string-to-bytes does in Lisp.

It doesn't tell what DEF-x-CALL-OUT for such a function would look like.

Is using vectors of (unsigned-byte 8) resp. (c-array uint8 n) all
that's needed by people?
It doesn't look nice a priori. And it sounds strange (at least
to me) that an implementation which supports dozens of encodings
uses uint8 for the FFI. Another case of worse is better?

1:2 encoding (UTF-16) could be easy to add as well, but variable
size (UTF-8) is no-no so far. Think about all the functions
where you pass in a char* buffer and the buffer size... (my old
praise of the IDL is_size declaration, cf my FFI design article
and others).

> > I now use sysdll-1.7.c and sysdll-1.3.h from XEmacs CVS
> The current version is quite unstable IMHO, considering the 
> unicode and mule [...]

That's XEmacs stability, not necessarily affecting supporting libraries. E.g. unstability of CLISP doesn't imply ffcall would be unstable. In any case, if sysdll-1.7.c suits the need, why not just take this version? At some time in future, somebody may look if they changed it. E.g., nobody so far felt a need for upgrading modules/regexp/regex.c in three years.

Regards,
	Jorg Hohle.