Subject: [clisp-list] read-line (clisp 2.30/woody)

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Tue Jan 21 01:26:06 2003

Michael,

I came to exactly the same conclusions!

>   A code calling
>  socket-status and read-line is broken by design,
Yes, since read-line will block until the line completes.

>  I think I need two low level lisp functions :
>        (unix-read array &optional stream) => count; errno
>        (unix-write array &optional stream) => count; errno

We had a thread here about READ-SOME-BYTES some time ago. I didn't implement it back then because it's a nightmare, as the code would run across half a dozen files (stream.d for top-level, socket.d, unixaux.d, winaux.d etc.).

That doesn't mean it's not useful.

Thinking more on this as time passes by, I came to the conclusion that READ-SOME-SEQUENCE or READ-SOME-CHARACTERS (as suggested back then) are broken by design. There's nothing like partial reads when given multibyte characters (or CRLF markers or 2- or 4-byte integers).
Such a partial result only makes sense for the smallest entity: (unsigned-byte 8). From there, one could obtain characters using EXT:convert-from-bytes - which needs to be extended to return how much of the input could be converted.
My feeling is that CLISP ought to implement simple streams natively. Simple streams seem to solve such problems and has this nice layering of low-level byte streams and upper-level character or integer streams. Simple streams at least has shown evidence that it works in Allegro and CMUCL, so it cannot be that broken. But rewriting stream.d (again) is a major undertaking.

I'll look into providing a READ-SOME-BYTES hack that would only work on sockets (possibly pipes), and not bother at all about integration with the rest of stream.d. At least the function would be there, and possibly integrated at a later time, should simple streams eventually get there.

Some further comments:

>  Its a question, if count
>  should become NIL or stay -1 on error conditions.
Iik, please stay away from broken C "magic values" when doing Lisp! We have multiple values, closures, signaling and error handling.

>  block. But an array size of one should work - i thought ;-(
Of course not. CLISP will fill it up, like a longer one.

>    Line: (#(g)) Flag:(1)
>    Status (NIL INPUT)
>    Line: (#()) Flag:(1)                    < perhaps CR - 
>where is LF ?
>    Status (NIL NIL)
It's not that good an idea to write out a string using (format t "~A" #) when you expect to look at non-printable characters.

>  so i cant distinct between <CR> and <LF> as both are mapped #()
>  and i still dont have a way to catch the :eof condition.

READ-LINE will not let you distinguish them. That's its design. In Lisp, as in C, the line is conceptually terminated by #\newline (resp. \n in C). Whether OS conventions want something else(Mac)/more(DOS) should not matter to the programmer. That is the abstraction.

From all the problems people reported with that in CLISP, I came to the conclusion that this does not work for interactive stuff. At least not in a world where programmers throw out LF-only terminated lines when all specifications (e.g. SMTP, HTTP) clearly specify CRLF and your code tries to "be liberal in what you accept".

READ-SEQUENCE, hmm? Not sure it may help.

Did you consider (ext:make-encoding :charset ... :line-terminator :dos/:unix:mac)? I'm not sure it would make a difference. Probably not in conjunction with :element-type 'character, which will also convert CR/LF int o a single #\newline.
Impnotes clearly states: "The line terminator mode is relevant only for output (writing to a file/pipe/socket). During input, all three kinds of line
terminators are recognized."

Regards,
	Jorg Hohle.

---

From: Michael Koehne <[email-masked]>
Date: Tue Jan 21 11:29:01 2003

Moin Joerg-Cyril Hoehle, 

> I came to exactly the same conclusions!
  *cheer*
> >  A code calling socket-status and read-line is broken by design,
  a TCP packet could also contain several lines, most pipe packets are
  4069 bytes long and likely contain several lines,  ..., and lots of
  other race conditions.
  
  Rewriting read-line to work with socket-status, would have to code
  dozens of race conditions. Implementing IAC telnet options, within
  this read-line/socket-status combo, would add an other can of worms. 

> Such a partial result only makes sense for the smallest entity:

  !NO! - think about 1000 users ;-( I want to read complete blocks,
  as they are, as they come - low level. So only !ONE! read() call
  per socket that has input. So i just need access to the low level
  read and write. To implement the logic in LISP.

  This logic would not follow the stream design pattern, of a process
  waiting for a line. It would be a scheduler sending line events to
  objects. So there wont be a read-line method of the stream but a
  process-line method of the object containing the socket.

> My feeling is that CLISP ought to implement simple streams natively.

  a simple stream means that there is 1 process waiting for read-line ?
  this already works - with one process and one stream at a time. 

> But rewriting stream.d (again) is a major undertaking.

  would this mean, that there wont be a socket-status but some ?

  (loop (create-process-with-socket (socket-accept server)
    (do-something-with-read-line)))

  does clisp has its internal multithreading ? or internal processes ?
  do they already work with streams in a way like above ?

> I'll look into providing a READ-SOME-BYTES hack that would only work
> on sockets (possibly pipes), and not bother at all about integration
> with the rest of stream.d.

  exactly: furthermore, sockets and pipes, who are used in socket-status
  should be tainted. This mean only socket-accept on server sockets and
  system-read-buffer/system-write-buffer could be used without warning.

> It's not that good an idea to write out a string using (format t "~A" #)
  when you expect to look at non-printable characters.

  *hm* whats the best way doing this ?

> READ-LINE will not let you distinguish them. That's its design. In Lisp,
> as in C, the line is conceptually terminated by #\newline (resp. \n in
> C). Whether OS conventions want something else(Mac)/more(DOS) should
> not matter to the programmer. That is the abstraction.

  now: rfc854 states, that #\newline is either "\r\n" or "\r\000" !

  In MUD context read-record return its buffer on "\r\n", "\r\000",
  "\377\371" or "\377\357". A corrosponding write-record would need
  a flag what kind of record termination is needed.

Bye Michael
-- 
  mailto:[email-masked]             UNA:+.? 'CED+2+:::Linux:2.4.6'UNZ+1'
  http://www.xml-edifact.org/           CETERUM CENSEO WINDOWS ESSE DELENDAM