Subject: AW: [clisp-list] FFI - Creating c-places in LISP? -> READ-SOME-SE QUENCE

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Wed Oct 2 09:51:01 2002

Dave Richards wrote:
>> I don't understand why you need C for this.
>It's quite simple. [...]  I already have a reasonably 
>high-performance,
>single-threaded, multi-user implementation. [...]
>I have my code running in CMUCL now.
>I've already implemented a CLISP FFI
>library which gives me accesses to all the interfaces I need.

I see - you already have a large code base and want to integrate with that.
Then I suggest the following: write a little C extension that does the following (what READ-/WRITE-SOME-SEQUENCE would do for socket streams):

A) locate .d code to get to the pointer to the contents of a Lisp array (the unpack_...() stuff), even with displaced arrays, :start and :end keywords etc.. Copy&Paste that code.
B) couple that with a single call to recv() or send(). Add in the usual error reporting of CLISP.

Maybe it would even become trivial if READ-SOME-SEQUENCE would be implemented as a set of functions specialized for files, sockets etc. You could then call (the hopefully exported) read_some_bytes_socket(object array, SOCKET fd, ...) from the future STREAM.D.

Sam Steingold wrote:
>Note that we already have READ-BYTE-SEQUENCE and READ-CHAR-SEQUENCE (in
>addition to READ-SEQUENCE).  In addition to READ-SEQUENCE-ONCE, you
>will need also READ-CHAR-SEQUENCE-ONCE and READ-BYTE-SEQUENCE-ONCE
>(and make them generic functions too - see gray.lisp)

Argh... Too much overhead IMHO. :-(
Where's the KISS principle gone?
No wonder nobody ever added this functionality in the past.

I fear this is going into nightmares -- consider the multi-byte character / partial read interaction (e.g. that other current thread about Chinese): what if you read only half of a 2byte character?

BTW, I forgot why READ-BYTE/CHAR-SEQUENCE are needed in addition to the CLHS READ-SEQUENCE. Bruno Haible explained that once IIRC.

Dave wrote:
>I seem to keep running into CLISP walls.
Yeah. I think CLISP lost a few users in the past because of this. Err, I digress.

One wall - where I admitedly recognize myself - is about trying to find a general solution for partial reads when people would be satisfied with (unsigned-byte 8) only and don't care about multi-byte characters and encodings. "Keep the ball running" is in danger in all these MIT vs. Jersey situations.

Regards,
	Jörg Höhle.

---

From: Sam Steingold <[email-masked]>
Date: Wed Oct 2 10:49:39 2002

> * In message <[email-masked]>
> * On the subject of "AW: [clisp-list] FFI - Creating c-places in LISP? -> READ-SOME-SE QUENCE"
> * Sent on Wed, 2 Oct 2002 18:49:48 +0200
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>
> Sam Steingold wrote:

no overhead at all.  STREAM-READ-SEQUENCE-ONCE is generic, but it
is used only for gray streams.  The built-in streams, like socket
steams, use the built-in READ-SEQUENCE-ONCE directly.

> BTW, I forgot why READ-BYTE/CHAR-SEQUENCE are needed in addition to
> the CLHS READ-SEQUENCE. Bruno Haible explained that once IIRC.

I hope Bruno will speak up here.

defs2.lisp says:

;; READ-SEQUENCE and WRITE-SEQUENCE are badly specified because they assume
;; that the stream has a unique element type, either subtype of CHARACTER or
;; subtype of INTEGER. But some streams (esp. generic-streams) have a type
;; of (OR CHARACTER INTEGER).

So, Joerg, are you gonna do the READ/WRITE-SEQUENCE-ONCE patch?

--
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
In the race between idiot-proof software and idiots, the idiots are winning.

---

From: Bruno Haible <[email-masked]>
Date: Wed Oct 2 17:00:15 2002

Sam writes:
> >  In addition to READ-SEQUENCE-ONCE, you

If you rename them to READ-CHAR-SEQUENCE-NONBLOCKING and
READ-BYTE-SEQUENCE-NONBLOCKING it's easier to understand what they do.

> > the CLHS READ-SEQUENCE. Bruno Haible explained that once IIRC.

They are needed because some streams support both characters and
integers, and when you read into a sequence that can hold both (like a
LIST or a (VECTOR T)) the implementation couldn't know which side of
the stream to use.

Bruno

---

From: Sam Steingold <[email-masked]>
Date: Wed Oct 2 17:02:15 2002

> * In message <[email-masked]>
> * On the subject of "Re: READ-SEQUENCE-ONCE"
> * Sent on Thu, 3 Oct 2002 01:54:55 +0200 (CEST)
> * Honorable Bruno Haible <[email-masked]> writes:
>
> Sam writes:

But they are _not_ non-blocking!
they call read/write just once, and if that blocks, they do too.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
The difference between theory and practice is that in theory there isn't any.

---

From: Bruno Haible <[email-masked]>
Date: Wed Oct 2 17:24:28 2002

> they call read/write just once, and if that blocks, they do too.

This is not a well-defined description of some semantics, because a
single read() on a BSD system is not the same thing as a single read()
on a SysV system. On SysV, read() can return with a "short read"
spuriously, i.e. it can fill into the buffer fewer bytes than are
actually available - for no visible reason. Whereas on BSD systems,
your READ-BYTE-SEQUENCE-ONCE function will behave just like
READ-BYTE-SEQUENCE.

It's useless to introduce such variants if you cannot describe
in a portable way how they behave.

Bruno

---

From: "Dave Richards" <[email-masked]>
Date: Wed Oct 2 19:26:10 2002

> I see - you already have a large code base and want to integrate

Not really.  I have abstracted the low-level access routines to a class
called network-descriptor (ndesc).  This class handles all socket I/O, so I
am not struggling with a legacy issue here.  I am trying to gain the same
level of efficiency/performance I had in C and can get in CMUCL.

> Then I suggest the following: write a little C extension that
> does the following (what READ-/WRITE-SOME-SEQUENCE would do for
> socket streams):

That may be the solution.  Because the FFI already provided 99% of what I
needed, I'd hoped that being able to create a c-pointer from an (unsigned
byte 8) array would finish it off and I'd be done.  The FFI does everything
else I need.

	Dave