Subject: [clisp-list] RFE: convert-string-to-bytes and encoding's :line-terminator

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Wed May 7 01:24:08 2003

Hi,

I'd appreciate a mode where conversion from a Lisp string to bytes would respect the encoding's :line-terminator. I.e., I could choose among CR, LF or CRLF being written to byte arrays.

Currently, it is ignored (witness ext:convert-string-to-bytes and impnotes section2 29.4.3 and 13.1.8). Therefore, a single code 10 is written for #\newline and/or #\return, which only suits UNIX systems, but neither networking nor other systems. It's is only when writing to actual files, sockets or pipes that an encoding's :line-terminator is considered.

I find this inconsistent. It also makes some stuff harder at Lisp-level than necessary, or than in other implementations.

Example 1: a tcp wrapper, i.e. a HTTP proxy which I could use to look at what a web browser sends to a server. I believe people involved in that area have silently moved away from CLISP to other implementations (I myself used Python).

Example 2: Let's suppose I were to add SSL bindings to CLISP. SSL needs the following:
1. do a conversion from characters to bytes
2. encrypt these bytes
3. send them over a socket

For each it is required that the network protocols be obeyed. This typically requires use of CRLF as line-terminator in headers (SMTP, HTTP ...).

Currently, an application in CLISP would use the encoding's :line-terminator on the stream to have the correct CR and/or LF ending written to the socket. With SSL, this is not possible anymore.

So what must the application do with current CLISP? Invoke (POSITION #\newlin output) to split the output line by line, invoke ext:convert-string-to-bytes individually on each, then add #x0D/0A after each.

Great, isn't it?

Wouldn't
(defmethod write-string ((stream ssl-stream) ...)
   (ssl-cipher-output
    (the-underlying-socket ssl-stream)
    (ext:convert-string-to-bytes
     string
     ;; how about an alias :network instead of :dos?
     (stream-external-format stream))))))
be most of what is needed if :line-terminator were used?

For FFI purposes, C-STRING still needs an 1:1 encoding. It could be one where :line-terminator is either :unix (LF) or :mac (CR), or remain ignored. WITH-FOREIGN-STRING is not limited to 1:1 encodings. It should follow the redesigned ext:convert-string-to-bytes.

BTW, CLISP doesn't export a reader function for the line-terminator slot. (EXT:encoding-charset is a usable reader for the other slot.)

Well, maybe a function separate from ext:convert-string-to-bytes would be fine with me. That might not internally use the with_string(), cstombs() and other associated C macros, since I presume iconv() does not deal with line-termination (otherwise it would be there in CLISP already).
Yet convert-string-to-bytes seems the right place for such a change.

"How end of lines are exactly encoded is not really pertinent to the charset," says the "recode" manual. Yet recode deals with them all.

BTW, I'm not talking about input here. I have no problems in 95% of the situations with CR and/or LF all mapped to #\newline. It is mostly useful behaviour.

Regards,
	Jörg Höhle.

---

From: Sam Steingold <[email-masked]>
Date: Thu May 8 10:00:11 2003

> * In message <[email-masked]>
> * On the subject of "[clisp-list] RFE: convert-string-to-bytes and encoding's :line-terminator"
> * Sent on Wed, 7 May 2003 10:18:40 +0200
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>
> I'd appreciate a mode where conversion from a Lisp string to bytes
> would respect the encoding's :line-terminator. I.e., I could choose
> among CR, LF or CRLF being written to byte arrays.

I think the idea was that CONVERT-STRING-TO-BYTES should be used on
strings returned by READ-LINE and thus #\Newline-free.

Please feel free to fix this bug.

Note that wr_ch_array_unbuffered_dos and friends already do something
similar, so you may be able to extract a common subset.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Whether pronounced "leenooks" or "line-uks", it's better than Windows.