Subject: [clisp-list] help about [ clisp-Bugs-550603 ] UTF-16 min/max_bytes_per_char mu st be 2

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Thu May 2 01:28:09 2002

Hi,

I've got a FFI:WITH-FOREIGN-STRING ready, except for one single, yet major issue/bug, without which it cannot be released. I'd like to poll suggestion on it.

The macro allows to circumvent the 1:1 restriction on string conversions
It's parameters are very similar to LispWorks'.

The problem left is about byte-count for zero-terminated-strings.

(defmacro with-foreign-string
  ((foreign-variable element-count byte-count
   &key (encoding 'custom:*foreign-encoding*) (null-terminated-p 'T))
   string &body body)
LispWorks says:
The macro with-foreign-string is used to dynamically convert
a Lisp string to a foreign string and execute a list of
forms using the foreign string. The macro first converts
string, a Lisp string, into a foreign string. The symbol pointer
is bound to a pointer to the start of the string, the symbol
element-count is set equal to the number of elements in the
string, and the symbol byte-count is set equal to the number of
bytes the string occupies. Then the list of forms specified by
body is executed. Finally, the memory allocated for the foreign
string and pointer is de-allocated. [...]
The null-terminated-p keyword specifies whether the foreign
string is terminated with a null character. It defaults to t. If
the string terminates in a null character, it is included in the
element-count.

The core of the problem is: how to derive the byte-count from the encoding, so that zero-termination makes sense?

A single zero *byte* is not correct for all encodings, since foreign functions receiving a UTF-16/UCS-2/UNICODE-32 string can do 16-bit or 32-bit wide access to such a string.
Of course, I could always add 4 zero bytes at the end of the buffer, but LispWorks' interface yields the byte-count, so I must produce it. It's not reasonable IMHO to drop the byte-count, since the correct buffer size may be important for foreign-functions, so the programmer ought to know it and pass it on.

(with-foreign-string (fv e b :encoding charset:utf-16) "abc"
   (list e b))
Ought to yield (4 10), for #(254 255 0 97 0 98 0 99 0 0)
Currently it returns (4 9).
Using charset:unicode-32 is currently not broken:
(4 8) for #(0 0 0 97 0 0 0 98 0 0 0 99 0 0 0 0)

You can see from the example that a single 0 byte cannot terminate
UTF-16 or UCS-2 strings: 0+junk is not distinguishible from 0+97, i.e. #\a.

I thought I could use the min_bytes_per_char slot of
CLISP's encoding structure:
  # Minimum number of bytes needed to represent a character.
  uintL min_bytes_per_char;

It works perfectly for many encodings (e.g. UCS-2, UNICODE-32,
ISO-8859-1 ...)
But it is broken for UTF-16, which is important for MS-Windows.

The problem may be with libiconv.

Sam Steingold says:
>are you sure you need these numbers to be correct to get the
>correct translations?

Is there any other idea beside using this slot?

Here's the reference to the CLISP bug report with additional information:
> Bugs item #550603, was opened at 2002-04-30 08:41
> You can respond by visiting: 
> http://sourceforge.net/tracker/?func=detail&atid=101355&aid=55
> 0603&group_id=1355

> Category: libiconv
> Summary: UTF-16 min/max_bytes_per_char must be 2

> Initial Comment:

> impnotes says: "UTF-16, the 16-bit UNICODE character 
> set. Every character is represented as two bytes."
 
> However, TheEncoding->min/max_bytes_per_char is 
> currently 1, resp.8.
[...]

Regards,
	Jorg Hohle.

---

From: Sam Steingold <[email-masked]>
Date: Thu May 2 06:15:20 2002

> * In message <[email-masked]>
> * On the subject of "[clisp-list] help about [ clisp-Bugs-550603 ] UTF-16 min/max_bytes_per_char mu st be 2"
> * Sent on Thu, 2 May 2002 10:27:16 +0200
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>
> I've got a FFI:WITH-FOREIGN-STRING ready, except for one single, yet
> major issue/bug, without which it cannot be released. I'd like to poll
> suggestion on it.
> 
> The core of the problem is: how to derive the byte-count from the
> encoding, so that zero-termination makes sense?

use CONVERT-STRING-TO-BYTES to get the appropriate byte array, then
NULL-terminate it and turn into a foreign string.

> A single zero *byte* is not correct for all encodings, since foreign
> functions receiving a UTF-16/UCS-2/UNICODE-32 string can do 16-bit or
> 32-bit wide access to such a string.

> Of course, I could always add 4 zero bytes at the end of the buffer,
> but LispWorks' interface yields the byte-count, so I must produce
> it. It's not reasonable IMHO to drop the byte-count, since the correct
> buffer size may be important for foreign-functions, so the programmer
> ought to know it and pass it on.

> Is there any other idea beside using this slot?

(length (convert-string-to-bytes string encoding))

> _______________________________________________________________
> 
> Have big pipes? SourceForge.net is looking for download mirrors. We supply
> the hardware. You get the recognition. Email Us: [email-masked]
> _______________________________________________
> clisp-list mailing list
> [email-masked]
> https://lists.sourceforge.net/lists/listinfo/clisp-list

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.palestine-central.com/> <http://www.mideasttruth.com/>
Small languages require big programs, large languages enable small programs.