Subject: [clisp-list] FFI - Creating c-places in LISP? -> READ-SOME-SEQUEN CE

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Wed Oct 2 01:08:17 2002

Dave Richards wrote:

>; data is allocated in initialize-instance as:
>; (make-array (nq-size nq) :element-type '(unsigned-byte 8))
>
>My desire is to be able to call send() and recv() on the data part of
>(nq-data nq), i.e. I want a c-pointer to (aref (nq-data data) index).

I don't understand why you need C for this.

Suppose you had READ-SOME-BYTES
 (or READ-SOME-SEQUENCE
  or READ-PARTIAL-SEQUENCE
  or READ-SEQUENCE-PARTIAL), you'd call

(READ-SOME-SEQUENCE sequence socket) ;? &optional start end

and stay in Lisp. no C.

READ-SOME-SEQUENCE is defined as a function which does a single OS read() resp. recv() into the supplied sequence and returns the number of bytes/characters/elements read.

I didn't decide yet whether it would be allowed to return 0 except at EOF. I.e., this could well be blocking. The reason is that information whether it would return immediately or not - if that is of interest to the application - comes from another source: select(). Therefore a LOOP of just READ-SOME-SEQUENCE until filled up is guaranteed both not to poll and to eat data as soon as possible.

Similarly for WRITE-SOME-SEQUENCE. Same open issue for returning 0. The pragmatics is: get me what I can get in one sweep, but don't poll.

The pair is trivial to write using the existing CLISP code base: don't loop until the array is filled up. Just nobody did/does it for years, I wonder why.

Regards,
	Jörg Höhle.

---

From: Sam Steingold <[email-masked]>
Date: Wed Oct 2 07:17:40 2002

> * In message <[email-masked]>
> * On the subject of "[clisp-list] FFI - Creating c-places in LISP? -> READ-SOME-SEQUEN CE"
> * Sent on Wed, 2 Oct 2002 10:06:45 +0200
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>
> Dave Richards wrote:
> 
> READ-SOME-SEQUENCE is defined as a function which does a single OS
> read() resp. recv() into the supplied sequence and returns the number
> of bytes/characters/elements read.

Note that we already have READ-BYTE-SEQUENCE and READ-CHAR-SEQUENCE (in
addition to READ-SEQUENCE).  In addition to READ-SEQUENCE-ONCE, you
will need also READ-CHAR-SEQUENCE-ONCE and READ-BYTE-SEQUENCE-ONCE
(and make them generic functions too - see gray.lisp)

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
He who laughs last thinks slowest.

---

From: "Dave Richards" <[email-masked]>
Date: Wed Oct 2 07:26:19 2002

> I don't understand why you need C for this.

It's quite simple.  I don't want/need the incomvenience, assumptions and
baggage of socket streams.  I already have a reasonably high-performance,
single-threaded, multi-user implementation.  Each socket must be
non-blocking.  No loops in read or write can exist.  I need to manipulate
socket options both at the SOL_SOCKET and IPPROTO_TCP level.  This
application knows a lot more about how it wants to do I/O than CLISP's
stream abstraction.  It does intellifent buffering, both on input and
output.

I have my code running in CMUCL now.  I prefer CLISP to CMUCL, but at least
I have an existence proof.  I seem to keep running into CLISP walls.

If I can't support 50 simultaneous users (which my current server can)
there's no point in making the transition.  I've already implemented a CLISP
FFI library which gives me accesses to all the interfaces I need.  I just
can't move data conveniently between my nqueues and the kernel's socket
buffers.

	Dave

---

From: "Dave Richards" <[email-masked]>
Date: Wed Oct 2 07:32:11 2002

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Sam
> Steingold
> Sent: Wednesday, October 02, 2002 7:15 AM
> To: Hoehle, Joerg-Cyril
> Cc: [email-masked]
> Subject: Re: [clisp-list] FFI - Creating c-places in LISP? ->
> READ-SOME-SEQUEN CE
>
>
> > * In message
> <[email-masked]>
> > * Honorable "Hoehle, Joerg-Cyril"
> <[email-masked]> writes:
> >
> > Dave Richards wrote:
> >

Yes, but the semantics of this interface is: do the requested I/O and either
(a) block until the specifies I/O completes, or (b) raise a condition
indicating it didn't.  That's great for almost all applications.  What I am
trying to create is an interface that layers right on top of the OS.  In
this interface the semantics are: do the requested I/O without blocking;
then tell me how much you actually accomplished.  This latter style is
exactly what my application needs and is much lighter weight.

Realizing the difference between what I wanted and how CLISP streams are
perceived to work, I decided to drop below the stream level and use the FFI.
(This is not dissimilar from dropped below the C FILE * (stdio) abstraction
and using the open(), close(), read() and write() abstraction.

	Dave

---

From: Sam Steingold <[email-masked]>
Date: Wed Oct 2 07:43:09 2002

> * In message <[email-masked]>
> * On the subject of "RE: [clisp-list] FFI - Creating c-places in LISP? -> READ-SOME-SEQUEN CE"
> * Sent on Wed, 2 Oct 2002 07:29:45 -0700
> * Honorable "Dave Richards" <[email-masked]> writes:
>
> Yes, but the semantics of this interface is: do the requested I/O and
> either (a) block until the specifies I/O completes, or (b) raise a
> condition indicating it didn't.  That's great for almost all
> applications.  What I am trying to create is an interface that layers
> right on top of the OS.  In this interface the semantics are: do the
> requested I/O without blocking; then tell me how much you actually
> accomplished.  This latter style is exactly what my application needs
> and is much lighter weight.

The contract of the READ-*-ONCE (WRITE-*-ONCE) functions is to make a
single read() (write()) OS call.  They will return the number of
objects (bytes or characters) read (written) - just like READ-SEQUENCE
(WRITE-SEQUENCE).
If this number is smaller than the size of the SEQUENCE argument, the
next i/o call will either block or return EOF (i.e., no more data is
immediately available).
We can even return a second value: NIL, :EOF, or :BLOCK.

I think this is what you want.  Am I right?

> Realizing the difference between what I wanted and how CLISP streams
> are perceived to work, I decided to drop below the stream level and
> use the FFI.  (This is not dissimilar from dropped below the C FILE *
> (stdio) abstraction and using the open(), close(), read() and write()
> abstraction.

I think you are making a mistake here.
Please reconsider.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
The software said it requires Windows 3.1 or better, so I installed Linux.

---

From: Sam Steingold <[email-masked]>
Date: Wed Oct 2 08:00:47 2002

> * In message <[email-masked]>
> * On the subject of "RE: [clisp-list] FFI - Creating c-places in LISP? -> READ-SOME-SEQUENCE"
> * Sent on Wed, 2 Oct 2002 07:19:29 -0700
> Each socket must be non-blocking.

I am not sure you really need this.
Please do not get upset - I recognize the obvious, that you know TCP/IP
much better than I do, and you certainly have a better idea of what you
are trying to accomplish.
OTOH, I might have a slightly better understanding of CLISP's
(potential) capabilities.

> No loops in read or write can exist.  

I think that READ-*-ONCE and WRITE-*-ONCE would do what you need.

> I need to manipulate socket options both at the SOL_SOCKET and

I think you have it now with SOCKET-OPTIONS, right?
what is missing?

> IPPROTO_TCP level.

what is this?

> This application knows a lot more about how it wants to do I/O than
> CLISP's stream abstraction.

Yes, so you might be right that you do need to go low level.
OTOH, you might be wrong. :-)

> It does intellifent buffering, both on input and output.

CLISP does i/o buffering too.

> I've already implemented a CLISP FFI library which gives me accesses
> to all the interfaces I need.

would you like to make it available to other CLISP users?
IIUC, it is running on top of CLISP bindings/linuxlibc6 module.
maybe we can make it a part of CLISP distribution?

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Fighting for peace is like screwing for virginity.

---

From: "Dave Richards" <[email-masked]>
Date: Wed Oct 2 08:02:56 2002

> I think this is what you want.  Am I right?

In the previous discussion about TCP-based servers, I raised the issue than
the sockets needed to be marked non-blocking.  I thought we'd concluded that
doing so could potentially break existing stream code.  Did I misunderstand?
Non-blocking is required to ensure that each I/O only performs what is
possible with no delay and that the server is then available to perform
other work.  This is what I meant by partial read()s/write()s.

	Dave

---

From: Sam Steingold <[email-masked]>
Date: Wed Oct 2 08:20:45 2002

> * In message <[email-masked]>
> * Sent on Wed, 2 Oct 2002 08:01:08 -0700
> In the previous discussion about TCP-based servers, I raised the issue
> than the sockets needed to be marked non-blocking.  I thought we'd
> concluded that doing so could potentially break existing stream code.
> Did I misunderstand?  Non-blocking is required to ensure that each I/O
> only performs what is possible with no delay and that the server is
> then available to perform other work.  This is what I meant by partial
> read()s/write()s.

CMIIAW: if you have a regular blocking socket and you call read/write
on it and it has _something_, it will not block.
the only way it might block is if it has _nothing_.
So you can use the regular CLISP buffered socket streams with the
proposed WRITE-SEQUENCE-ONCE and READ-SEQUENCE-ONCE and use the second
value to decide whether the next i/o will block or not.
Alternatively you can use SOCKET-STATUS before i/o: this should be
cheap enough because you will be doing block i/o, not byte i/o.

WAIM?

---

From: "Dave Richards" <[email-masked]>
Date: Wed Oct 2 08:27:12 2002

> cheap enough because you will be doing block i/o, not byte i/o.

We can try it.  The reason it was unsafe to trust this was because there
were implementation of select() (Linux used to be one of them) that would
falsely select() true when read() would block.  I had to work around this
case in my C server.  If that happened and the socket wasn't non-blocking,
the server would intermittently hang.  I am willing to try again.  I dislike
coding around bugs.  But this used to be a problem.

	Dave