Subject: [clisp-list] socket-streams vs. two-way streams

From: Todd Sabin <[email-masked]>
Date: Wed Jan 2 20:23:08 2002

Is there a reason why socket-streams are not real two-way streams?
The implementation notes refer to them as two-way streams as opposed
to just IO streams, but two-way-stream-input-stream, e.g. fails on
them.

However, if I just do this:

 # check whether the stream S is a two-way-stream
 #define stream_twoway_p(s)                                              \
-  (builtin_stream_p(s) && (TheStream(s)->strmtype == strmtype_twoway))
+  (builtin_stream_p(s) && ((TheStream(s)->strmtype == strmtype_twoway)
+                           || (TheStream(s)->strmtype == strmtype_twoway_socket)))
 
Then buffered sockets, at least, work as you would expect.  At least,
in my two minutes of testing :).  With a little work, the unbuffered
case could be handled similarly.  Would you accept a patch which did
this?

Todd

---

From: Sam Steingold <[email-masked]>
Date: Thu Jan 3 07:55:07 2002

> * In message <[email-masked]>
> * On the subject of "[clisp-list] socket-streams vs. two-way streams"
> * Sent on 02 Jan 2002 23:22:35 -0500
> * Honorable Todd Sabin <[email-masked]> writes:
>

I hope Bruno will answer this question.

> +                           || (TheStream(s)->strmtype == strmtype_twoway_socket)))
this patch will crash TWO-WAY-STREAM-INPUT-STREAM.

> case could be handled similarly.  Would you accept a patch which did

why do you want this?

-- 
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
It's not just a language, it's an adventure.  Common Lisp.

---

From: Todd Sabin <[email-masked]>
Date: Thu Jan 3 08:53:04 2002

> this patch will crash TWO-WAY-STREAM-INPUT-STREAM.

No, it doesn't.  Because there's already:

  # define strm_twoway_socket_input  strm_twoway_input  # input side, a socket stream
  #define strm_twoway_socket_output  strm_twoway_output # output side, a socket stream

Anyway, I wasn't suggesting that that patch would be sufficient, or
the right way to do it, just that it doesn't seem like it'd be hard to
do it completely.

> > case could be handled similarly.  Would you accept a patch which did

It would allow you to set different element-types and encodings on the
input and output sides of the socket.

It would provide a nice interface to shutdown(2) a socket.  You could
call (close (two-way-stream-output-stream *my-sock*)), and have it
implemented with shutdown(2) in C, instead of close(2).

It just seems right to me.

Of course, there may be implications that I don't see.  Turning the
question around, why wouldn't you want this?

Todd

---

From: Sam Steingold <[email-masked]>
Date: Sat Jan 5 12:05:43 2002

> * In message <[email-masked]>
> * On the subject of "Re: [clisp-list] socket-streams vs. two-way streams"
> * Sent on 03 Jan 2002 11:52:18 -0500

I don't understand this.
why is shutdown better than close?
what does this have to do with two-way-stream-output-stream being
applicable to a socket?

> question around, why wouldn't you want this?

because I don't know all the possible pitfalls.
are you prepared to submit tested (at least on two platforms) patches
and take responsibility (as in "fix bugs") for the code you write?
if yes, your patches (against CVS) will be accepted.
(we should probably move this discussion to <clisp-devel>.)

-- 
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
Isn't "Microsoft Works" an advertisement lie?

---

From: Todd Sabin <[email-masked]>
Date: Sun Jan 6 14:10:03 2002

> why is shutdown better than close?

shutdown(2) allow you to say, I'm done sending on this socket, but I
still want to be able to receive, or vice-versa.  close(2) closes both
directions.  In TCP terms, doing a shutdown on output will cause your
machine to send a FIN and close that half of the channel.  The other
side will in effect get an EOF on its input, and still be able to send
output back to you.

> what does this have to do with two-way-stream-output-stream being

I'm just trying to map the socket/TCP concepts onto common lisp stream
semantics.  IMO, the obvious way of doing it in this case is to have
sockets be two-way streams, and specifying the semantics above.

Generally speaking, I want to be able to do everything with sockets
that I can do in C, but in lisp.  Maybe this isn't the way to do it, I
don't know.

> because I don't know all the possible pitfalls.

That's fair enough.

> (we should probably move this discussion to <clisp-devel>.)

This particular feature isn't something that I'm dying for.  I'm just
trying to understand the psychology behind why you would or wouldn't
want it.  If I'm going to be writing code, I'd rather fix other things
first.  Currently, my biggest gripe is that buffered sockets are more
or less unusable for me, because they only work on 4K blocks, so for
things like HTTP you can't ever read all the waiting data unless the
other end has closed (or shutdown) their end of the connection.  I'd
rather that buffered sockets let you deal with the data you've got
waiting.  (Actually, I don't care so much about the buffering on the
input side, but you can't do buffering on output without doing it on
input, too.)

I haven't even gotten to UDP sockets, or unix domain sockets, yet,
which don't seem to be supported, currently.  Are they features you'd
want?  Do you know of good lispy interfaces for them?  Does clisp have
something analogous to unix's select(2) that lets you wait on multiple
streams until there's data/connections available?  (cmucl seems to
have serve-event for this.)

Todd

---

From: Sam Steingold <[email-masked]>
Date: Sun Jan 6 17:41:02 2002

> * In message <[email-masked]>
> * Sent on 06 Jan 2002 17:09:22 -0500

I am not sure this is a valid desire.
sockets are very hairy and lisp favor clarity.

> Currently, my biggest gripe is that buffered sockets are more or less
> unusable for me, because they only work on 4K blocks, so for things
> like HTTP you can't ever read all the waiting data unless the other
> end has closed (or shutdown) their end of the connection.  I'd rather
> that buffered sockets let you deal with the data you've got waiting.
> (Actually, I don't care so much about the buffering on the input side,
> but you can't do buffering on output without doing it on input, too.)

you are on your own here.  sorry.

> want?  Do you know of good lispy interfaces for them?

see "How to open a UNIX socket with clisp?" thread in this list (October
2001).  Since a unix socket is just a special file in the filesystem,
all we need to do is to make open_file aware of such an option (CLOSE
already is, I think). we even have the necessary code - see
connect_to_x_server() in socket.d.
The issue is, basically, that many systems lack the S_ISSOCK() macro.
It is a lot of tedious work to make sure that all systems that don't
have it will work correctly with our S_ISSOCK() macro (otherwise we will
have to support socket opening only on systems which do have such a
macro...)

> Does clisp have something analogous to unix's select(2) that lets you
> wait on multiple streams until there's data/connections available?

see SOCKET-STATUS
<http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/clisp/clisp/doc/impnotes.html?rev=HEAD&content-type=text/html#so-status>

-- 
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
Beauty is only a light switch away.

---

From: Todd Sabin <[email-masked]>
Date: Mon Jan 7 00:24:03 2002

> sockets are very hairy and lisp favor clarity.

I hardly know what to say to this, but I know I'm not alone in this
desire.  Quoting from the db-sockets page at
<URL:http://ww.telent.net/lisp/sockets.html>, "I wish to have the full
flexibility of the C interface available from a sensible language."
That's not "valid"?  I guess what your telling me is to go use cmucl
or sbcl.

> you are on your own here.  sorry.

On my own as in, I've got to figure it out, but you'd (maybe) accept
a patch if I do it?

or on my own as in, that's the way buffered sockets are supposed to
work and you're not changing it?

>[...]

Unix domain sockets are not just special files in the filesystem and
trying to treat them that way is the wrong approach, IMO.  Even if you
overcome the S_ISSOCK() issue, how are you going to know whether it's
a STREAM or DGRAM socket?  I guess you could just try both, but that's
rather kludgy.  And what about acting as a server with unix domain
sockets?  How would you accept connections from a file?

> <http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/clisp/clisp/doc/impnotes.html?rev=HEAD&content-type=text/html#so-status>

I saw that, but found the documentation confusing.  Particularly,

"Note that this function never waits for input or output to arrive,
only for information on input or output presense ... to become
available."

which makes it sound like the function will return once it can figure
out that there is no input or output data present anywhere.  I want to
block until data is available (or timeout).  Am I misunderstanding the
docs?

Also, is there a reason that it should only work with socket-streams?
Wouldn't it be a good thing for it to work on pipe-io-streams, too?
(Perhaps it already does, I haven't checked.)

Todd

---

From: Sam Steingold <[email-masked]>
Date: Mon Jan 7 10:48:09 2002

> * In message <[email-masked]>
> * Sent on 07 Jan 2002 03:23:51 -0500

not at all.
what I am saying is that _I_ know too little about sockets (as you can
probably guess from my messages already) to support this.

If you are willing to take responsibility for CLISP socket code, you
will be more than welcome.

> On my own as in, I've got to figure it out, but you'd (maybe) accept

yep.  "maybe" depends on how clean the patch is, and, most of all, on
your commitment to supporting it (for the rest of your earthly existence
and beyond it :-)

> sockets?  How would you accept connections from a file?

how would you like to do this?
maybe we should unify the internet sockets with the unix sockets.
like, SOCKET-CONNECT, when PORT is a string, will interpret it as a
unix socket path...

> block until data is available (or timeout).  Am I misunderstanding the

note the &OPTIONAL [seconds [microseconds]] args.
if SECONDS is NIL, it will block indefinitely, otherwise, it will wait
for this much time.

> (Perhaps it already does, I haven't checked.)

please do check.  i believe it should work.  if it doesn't - send a
patch.

I am sorry if I sound like "do it yourself of f..k off" dork.
I _am_ overworked, and have a lot on my plate now (and I have a family,
and I am not paid for CLISP).
I also gladly accept patches.

-- 
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
"Syntactic sugar causes cancer of the semicolon."	-Alan Perlis

---

From: Todd Sabin <[email-masked]>
Date: Thu Jan 10 20:21:08 2002

> your commitment to supporting it (for the rest of your earthly existence

Heh, well, let me see if I can figure it out first.

> like, SOCKET-CONNECT, when PORT is a string, will interpret it as a

I haven't looked at db-sockets much yet, but from what little I
understand of it, I think it's got the right idea.  I.e., just expose
the socket api calls to lisp, and then build on top of those in lisp.
In clisp, that would mean adding a new gc'd lisp object, Socket, and
functions socket, bind, setsockopt, accept, ioctl(!), connect, etc.
and some stuff for resolving names.  Then you have to add something
like cmucl's make-fd-stream which gives you a stream from a
SOCK_STREAM socket.

Once that's done, I think most of the existing socket.d could be
redone in lisp.  You can build friendly functions on top of it, e.g.,
socket-connect could be done something like

(defun socket-connect (port host ...)
  (let ((sock (socket :PF_INET :STREAM))
        (addr (make-addr (resolv host) port)))
    (connect sock addr)
    (make-socket-stream sock)))

and people who want to (like me) can get grungy with the low level
stuff.

socket-status would be modified to support socket-streams and plain
old sockets.

Anyway, that's my Blue Sky thinking at the moment.  From the sounds of
the clocc list, db-sockets might just be portable to clisp.  (I'm
still trying to figure out how all those pieces fit together.)  I
imagine that socket-status wouldn't work in that scenario without some
surgery, though.

I guess I need to take a good look at db-sockets...

> > "Note that this function never waits for input or output to arrive,
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> if SECONDS is NIL, it will block indefinitely, otherwise, it will wait

I think the docs could be clearer, particulary the part I marked
above.  Perhaps just saying it exposes select(2) would be sufficient.

> please do check.  i believe it should work.  if it doesn't - send a

It doesn't work.  It complains that it's not a socket-stream.  No
patch at the moment, maybe later.

> I also gladly accept patches.

No problem.  I don't mind and completely understand that you don't
want to implement things yourself just because I'm asking for them.  I
was just getting the impression that you didn't want the stuff, even
if _I_ implemented it, which was puzzling.

Todd

---

From: Sam Steingold <[email-masked]>
Date: Fri Jan 11 07:11:11 2002

> * In message <[email-masked]>
> * Sent on 10 Jan 2002 23:19:09 -0500

This is _not_ how this is done in CLISP.
CLISP is "high-level" and "cross-platform".
The functions you mention are "low-level" and "unix-specific".

If you really want to build upon all these calls, you _already can_ do
it in CLISP on Linux as it is distributed - just build
        ./configure --with-module=bindings/linux6.
and use the full image.
(this exposes _all_ libc calls to lisp, in package LINUX)
what?  you mean you want these on non-Linux platforms?
Okay, add a bindings module for your platform, or, better yet, create
a cross-platform LIBC module which will work uniformly on all flavors
of UNIX.  You know better than I do how all these low-level things
differ in, say, BSD, Solaris, and Linux.  Good luck there.

Note that it really _is_ easier to add a module than to put what you
want into the CLISP core functionality.

> people who want to (like me) can get grungy with the low level stuff.

try "clisp -K full -x '(apropos "" "LINUX")'"

> I was just getting the impression that you didn't want the stuff, even

There is some design philosophy behind CLISP.
There are things I do not want there, no matter what (e.g., crashes :-).
Other things are better off in modules, not in the base (e.g., bind(2)).

CLISP is _not_ for everyone (sadly).
If you think you want to tinker in Assembly, CLISP is not for you.
(But I will strive to convince you that you actually do _not_ want to do
Assembly :-)
If you want high performance floating point computations, CMUCL is a
better choice, there is nothing I can do here.
OTOH, if you want to write in a high-level language, opening sockets as
easily as files and handling them in a uniform manner, CLISP is your
friend.

-- 
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
A clear conscience is usually the sign of a bad memory.

---

From: Bruno Haible <[email-masked]>
Date: Fri Jan 11 08:58:05 2002

> > Is there a reason why socket-streams are not real two-way streams?

Because the input part and the output part of the socket-stream are
not independent: After closing one of them, the other one would be
nonfunctional (because its C 'SOCKET' has been closed).

> > in my two minutes of testing :).

What is wrong with buffered socket streams as they are implemented
now?

Bruno

---

From: Sam Steingold <[email-masked]>
Date: Fri Jan 11 09:55:05 2002

> patch at the moment, maybe later.

please note that select() is a socket-specific syscall on win32, so you
cannot pass just any handle to it.
if you can make it work - great!

for now, listen works with pipes as expected.

-- 
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
You can have it good, soon or cheap.  Pick two...

---

From: Sam Steingold <[email-masked]>
Date: Fri Jan 11 13:48:05 2002

> * In message <[email-masked]>
> * Sent on Fri, 11 Jan 2002 17:54:11 +0100 (CET)
> * Honorable Bruno Haible <[email-masked]> writes:
>

(setq s (socket-connect 80 "www.gnu.org" :external-format :dos :buffered t))
(format s "GET / HTTP/1.0~2%")
(finish-output s)
(socket-status s)
 ==> :output

i.e., there is nothing to be read from the buffered socket S.
this is why (I think) Todd wants to separate the input and output
streams of the sockets: he thinks that after he closes the output
stream, the input will become available.
(I don't think this is the case since the buffering is done by CLISP,
not the TCP/IP stack or OS).
 
-- 
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
An elephant is a mouse with an operating system.

---

From: Todd Sabin <[email-masked]>
Date: Fri Jan 11 17:06:11 2002

> not the TCP/IP stack or OS).

I hadn't noticed this particular behavior, but it also seems wrong,
doesn't it?

The problem I have with buffered sockets is on the server side, not
the client.  There's no way a server can use buffered sockets, because
it's totally impossible to read a client's requests unless they happen
to terminate on 4K boundaries (or the client closes his end of the
socket with shutdown(2) before receiving a reply).  And, as I said
before, I don't really care much about buffering on input, but you
can't buffer on output unless you also buffer on input, and I do care
about buffering on output.

I really don't care that much about whether sockets are two-way
streams, or not.  It just seemed like a natural way to expose more
functionality, without getting too "low level".

Todd

---

From: Todd Sabin <[email-masked]>
Date: Fri Jan 11 17:08:17 2002

> if you can make it work - great!

I don't think this is a (clean) way to make it work on win32.  Would
you take a patch which makes socket-status work on pipe streams on
linux/unix only?

Todd

---

From: Todd Sabin <[email-masked]>
Date: Fri Jan 11 17:38:01 2002

> (this exposes _all_ libc calls to lisp, in package LINUX)

Aha!  Well, the socket stuff isn't currently defined in there, but I
guess you'd suggest adding it?

> what?  you mean you want these on non-Linux platforms?

No, not really.  Linux only would be fine with me.  :)

Todd

---

From: Todd Sabin <[email-masked]>
Date: Fri Jan 11 17:49:02 2002

> I don't think this is a (clean) way to make it work on win32.  Would

Err, meant, "I don't think _there_ is a ..."

Todd

---

From: Sam Steingold <[email-masked]>
Date: Sat Jan 12 07:10:03 2002

> * In message <[email-masked]>
> * Sent on 11 Jan 2002 20:37:28 -0500

sure.  add whatever you want to bindings/linuxlib6, test it, make it
work well, &c.  I will take it.

-- 
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
Linux - find out what you've been missing while you've been rebooting Windows

---

From: Sam Steingold <[email-masked]>
Date: Mon Jan 14 11:06:03 2002

> * In message <[email-masked]>
> * Sent on 11 Jan 2002 20:07:28 -0500
>  [reagrding SOCKET-STATUS & pipe streams]

sure. note that the following trivial patch is no good:

Index: src/stream.d
===================================================================
RCS file: /cvsroot/clisp/clisp/src/stream.d,v
retrieving revision 1.249
diff -u -w -b -r1.249 stream.d
--- src/stream.d        2002/01/11 18:00:42     1.249
+++ src/stream.d        2002/01/14 18:51:05
@@ -14831,7 +14831,26 @@
   if (socket_server_p(obj)) {
     if (check_open) test_socket_server(obj,true);
     return TheSocket(TheSocketServer(obj)->socket_handle);
-  } else {
+  }
+ #if defined(PIPES) && defined(UNIX)
+  #ifdef PIPES2
+  else if ((TheStream(obj)->strmtype == strmtype_twoway) &&
+           (TheStream(TheStream(obj)->strm_twoway_input)->strmtype
+            == strmtype_pipe_in) &&
+           (TheStream(TheStream(obj)->strm_twoway_output)->strmtype
+            == strmtype_pipe_out)) { # bidirectional pipe
+    # assume output is always possible, check only input
+    obj = TheStream(obj)->strm_twoway_input;
+    goto pipe;
+  }
+  #endif
+  else if (TheStream(obj)->strmtype == strmtype_pipe_in)
+  pipe:
+    return (SOCKET)TheHandle(TheStream(obj)->strm_ichannel);
+  else if (TheStream(obj)->strmtype == strmtype_pipe_out)
+    return (SOCKET)TheHandle(TheStream(obj)->strm_ochannel);
+ #endif
+  else {
     obj = test_socket_stream(obj,check_open);
     return SocketChannel(obj);
   }

first, both input _and_ output must be checked
second, for some reason it doesn't work :-)

I would _really_ prefer a cross-platform patch.

OTOH, if you want to go "UNIX-only", note that it should work with any
FD-based stream then, including terminal streams, arbitrary two-way
streams (as long as their components are acceptable to SOCKET-STATUS),
broadcast streams &c. 

-- 
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
Of course, I haven't tried it. But it will work.    - Isaak Asimov

---

From: Sam Steingold <[email-masked]>
Date: Fri Jan 18 10:04:08 2002

> * In message <[email-masked]>
> * Sent on 11 Jan 2002 20:03:30 -0500

actually, the output _is_ available, i.e., even though SOCKET-STATUS
returns :OUTPUT (i.e., no :INPUT is possible), READ-LINE _will_ return
right away as expected.
Note that READ-CHAR-WILL-HANG-P _hangs_ on a buffered socket when it
cannot return NIL:

(setq s (socket-connect 80 "www.gnu.org" :buffered t :external-format :dos))
#<IO BUFFERED SOCKET-STREAM CHARACTER www.gnu.org:80>
(format s "GET / HTTP/1.0~2%")
(READ-CHAR-WILL-HANG-P s)
__HANG__

(setq s (socket-connect 80 "www.gnu.org" :buffered t :external-format :dos))
#<IO BUFFERED SOCKET-STREAM CHARACTER www.gnu.org:80>
(format s "GET / HTTP/1.0~2%")
(finish-output s)
(READ-CHAR-WILL-HANG-P s)
NIL

> about buffering on output.

confirmed.

-- 
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
The difference between theory and practice is that in theory there isn't any.

---

From: Todd Sabin <[email-masked]>
Date: Fri Jan 18 11:02:09 2002

> Note that READ-CHAR-WILL-HANG-P _hangs_ on a buffered socket when it

Yes, I was looking into it last night, and it's a real can of worms.
The problem is in handle_isset (quoted here for reference)

# check the appropriate fd_sets for the socket,
# either a socket-server, a socket-stream or a (socket . direction)
# see socket_status() for details
local object handle_isset (object socket, fd_set *readfds, fd_set *writefds,
                           fd_set *errorfds) {
  object sock = (consp(socket) ? Car(socket) : socket);
  direction_t dir = (consp(socket)?check_direction(Cdr(socket)):DIRECTION_IO);
  SOCKET handle = socket_handle(sock,true);
  if (FD_ISSET(handle,errorfds)) return S(Kerror);
  else {
    if (socket_server_p(sock)) {
      return FD_ISSET(handle,readfds) ? T : NIL;
    } else {
      bool wr = WRITE_P(dir) && FD_ISSET(handle,writefds),
        rd = (READ_P(dir) && FD_ISSET(handle,readfds)
              && (ls_avail_p(stream_char_p(sock))
                  ? listen_char(sock)
                  : listen_byte(sock)));
      if      ( rd && !wr) return S(Kinput);
      else if (!rd &&  wr) return S(Koutput);
      else if ( rd &&  wr) return S(Kio);
      else                 return NIL;
    }
  }
  return NIL;
}

There are multiple problems here.  The most obvious is that the
listen_char/byte stuff is in the wrong place.  It belongs inside the
ls_avail_p call, not outside it:

              && (ls_avail_p(stream_char_p(sock)
                             ? listen_char(sock)
                             : listen_byte(sock))));

Ok.  But that's just the first problem.  Next is the fact that this
stream_char_p(sock) call is checking the stream's element type, but it
doesn't do any of the gymnastics to handle the cases of buffered
sockets, or two-way streams, etc.  (I'm pretty sure this is why your
attempt at adding the pipe stuff failed).  So it just looks at the
eltype or whatever member of the Stream structure, (which is
uninitialized in these cases), compares it to Character, finds that
it's different, and then tries to listen_byte on a character stream,
which naturally fails.  So it thinks there's no input available.

The kicker is that, at least according to the comments, both
listen_char and listen_byte can trigger GC, which means that there is
at least one GC safety bug (I think) in socket_status, the variable
'list':

    for(; !nullp(list); list = Cdr(list), index++) {
      object tmp = handle_isset(Car(list),&readfds,&writefds,&errorfds);
      pushSTACK(tmp);
    }

I don't know if it matters or not, but I just noticed this occurs
inside of a begin_system_call/end_system_call pair, too.

Semi-related is the fact that socket-status :input on a buffered
stream should probably check whether there's data in the buffer before
calling select.  If there's data available, set the timeout to zero,
remove the stream from the list to check, use select (with the zero
timeout) to check the other streams, and then cons up the answers.
Otherwise, socket-status will still give the wrong answer in some
cases.

> (READ-CHAR-WILL-HANG-P s)

Dunno about that one.

Todd

---

From: Sam Steingold <[email-masked]>
Date: Fri Jan 18 12:42:02 2002

> * In message <[email-masked]>
> * Sent on 18 Jan 2002 13:58:25 -0500

the version you quote is fairly old - how long ago did you do a `cvs up`?

> The most obvious is that the listen_char/byte stuff is in the wrong
> place.
this have been fixed for some time.

> which naturally fails.  So it thinks there's no input available.

I don't think this is the problem.
my pipe attempt did not fail.
my patch does work - it's just incomplete.

> at least one GC safety bug (I think) in socket_status, the variable

thanks!  fixed now!

> Otherwise, socket-status will still give the wrong answer in some

why remove it?

--
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
When you are arguing with an idiot, your opponent is doing the same.

---

From: Todd Sabin <[email-masked]>
Date: Fri Jan 18 13:34:01 2002

> the version you quote is fairly old - how long ago did you do a `cvs up`?

About a week ago, apparently just before you fixed it.

> I don't think this is the problem.

Why not?  What's inaccurate in what I said?

>second, for some reason it doesn't work :-)

Maybe you have a more recent version?

> > Otherwise, socket-status will still give the wrong answer in some

Because select will probably give you the wrong answer.  I don't mean
remove as in not to give an answer to the caller, just that
socket-status can't rely on select to provide it in that case.

Todd

---

From: Sam Steingold <[email-masked]>
Date: Fri Jan 18 14:18:02 2002

> * In message <[email-masked]>
> * Sent on 18 Jan 2002 16:33:17 -0500

you are actually correct!!
I debugged the patch and what you said was actually the root of the
problem.
I will not have unix access over the weekend though - please try to
work on this yourself.

> socket-status can't rely on select to provide it in that case.

using (buffer || select) should work

--
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
OK, so you're a Ph.D.  Just don't touch anything.

---

From: Sam Steingold <[email-masked]>
Date: Mon Jan 21 09:21:02 2002

> Maybe you have a more recent version?

SOCKET-STATUS now works with any handle-based stream provided your OS's
select call can accept the handle (i.e., pipes work on UNIX but not win32)

--
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
Never underestimate the power of stupid people in large groups.

---

From: Todd Sabin <[email-masked]>
Date: Mon Jan 21 10:25:01 2002

> select call can accept the handle (i.e., pipes work on UNIX but not win32)

I haven't tested it, but I think your patch gets the case of buffered
io pipes wrong.  I.e.

(setq p (make-pipe-io-stream "bash" :buffered t))
(format p "ls~%")
(force-output p)
(socket-status p)

will yield :output, not :io.  I made the same mistake when I tried it
this weekend.  For buffered pipes, you need to use
ChannelStream_buffered(stream) to get the handle.  Probably makes
sense to define a PipeChannel similar to SocketChannel.

I also did some work on getting buffered streams to work with non-4096
blocks of data, and to not hang on read-{byte,char}-will-hang-p.  It's
not complete at this point, but I'll send it separately as food for
thought...

Todd

---

From: Todd Sabin <[email-masked]>
Date: Mon Jan 21 10:57:02 2002

> > about buffering on output.

Here's some progress on this.  (Ignore the socket-status stuff, since
you've already done that.)  There are two sets of changes, both still
works in progress.

The first makes it possible to use buffered streams when you don't
have 4096 blocks of data to work with.  What I did is change the
eofindex member of the stream to an 'endvalid' member, and added an
explicit flag which indicates that the eof is at endvalid.  This
actually simplified a number of things, though it's not complete, yet.
I'm sure the eof_hit flag isn't set correctly in all cases.  The other
piece needed for this is to change full_read in unixaux.d to return
less than 4096 bytes of data if possible.  (Other _read functions
would need to be similarly modified.)  With those changes buffered
sockets are quite usable, though the hang problem still exists.

Fixing the hang problem was worse.  I modified buffered_nextbyte to
accept a bool* which means that the caller wants to know if
buffered_nextbyte _might_ block.  Most existing callers of
buffered_nextbyte just pass NULL and block if necessary, as before.
listen_char and listen_byte, however, are interested, and if
buffered_nextbyte might hang, they call low_listen_handle.
low_listen_handle is what used to be low_listen_unbuffered_handle,
minus the unbuffered stuff.  In the case where low_listen_handle has
to resort to a read, it makes a callback with the read data so that it
can be handled appropriately in the buffered and unbuffered cases.
This last bit is quite ugly (and not totally correct at this point,
either), and there's probably a better way to handle it.

Anyway, food for thought.  With these changes, my web proxy gets the
same throughput for large downloads as I get with the C proxy I have
been using, ~190K/s.  I.e. the limiting factor is my bandwidth.  When
I have to use unbuffered sockets, the best it could do was ~30K/s.  So
fixing this is a big win.

Comments appreciated.

Todd

Index: stream.d
===================================================================
RCS file: /cvsroot/clisp/clisp/src/stream.d,v
retrieving revision 1.253
diff -u -r1.253 stream.d
--- stream.d	2002/01/18 22:07:51	1.253
+++ stream.d	2002/01/21 18:40:29
@@ -4604,18 +4604,17 @@
     return b;
   }
 }
-
-local signean low_listen_unbuffered_handle (object stream) {
-  if (UnbufferedStream_status(stream) < 0) # already EOF?
-    return ls_eof;
-  if (UnbufferedStream_status(stream) > 0) # bytebuf contains valid bytes?
-    return ls_avail;
+  
+local signean low_listen_handle (object stream, uintL read_amt,
+                                 void (*handle_read)(object stream, uintB *buff, uintL amt)) {
   # Method 1: select, see SELECT(2)
   # Method 2: ioctl FIONREAD, see FILIO(4)
   # Method 3: switch temporarily to non-blocking I/O and try read(),
   #           see READ(2V), FILIO(4), or
   #           see READ(2V), FCNTL(2V), FCNTL(5)
-  var Handle handle = TheHandle(TheStream(stream)->strm_ichannel);
+  var bool buffered = ChannelStream_buffered(stream); 
+  var Handle handle = buffered ? TheHandle(TheStream(stream)->strm_buffered_channel)
+                               : TheHandle(TheStream(stream)->strm_ichannel);
   #if defined(EMUNIX)
   {
     var struct termio oldtermio;
@@ -4722,7 +4721,7 @@
   #if !(defined(HAVE_SELECT) && !defined(UNIX_BEOS))
   if (!nullp(TheStream(stream)->strm_isatty)) { # Terminal
     # switch to non-blocking mode, then try read():
-    var uintB b;
+    var uintB b[4096];
     var int result;
   restart_read_tty:
     #ifdef FIONBIO # non-blocking I/O a la BSD 4.2
@@ -4732,7 +4731,7 @@
       if (!( ioctl(handle,FIONBIO,&non_blocking_io) ==0)) {
         OS_error();
       }
-      result = read(handle,&b,1);
+      result = read(handle,b,min(sizeof(b),read_amt));
       non_blocking_io = 0;
       if (!( ioctl(handle,FIONBIO,&non_blocking_io) ==0)) {
         OS_error();
@@ -4753,7 +4752,7 @@
         OS_error();
       }
       #endif
-      result = read(handle,&b,1);
+      result = read(handle,b,min(sizeof(b),read_amt));
       if ( fcntl(handle,F_SETFL,fcntl_flags) <0) {
         OS_error();
       }
@@ -4780,7 +4779,7 @@
       return ls_wait;
     } else {
       # Stuff the read byte into the buffer, for next low_read call.
-      UnbufferedStreamLow_push_byte(stream,b);
+      handle_read (stream, b, result);
       return ls_avail;
     }
     # If this doesn't work, should use a timer 0.1 sec ??
@@ -4790,8 +4789,9 @@
     {
       # try to read a byte:
     restart_read_other:
-      var uintB b;
-      var int result = read(handle,&b,1);
+      var uintB b[4096];
+#define min(a,b) (((a)<(b))?(a):(b))
+      var int result = read(handle,b,min(sizeof(b),read_amt));
       if (result<0) {
         if (errno==EINTR)
           goto restart_read_other;
@@ -4799,10 +4799,13 @@
       }
       end_system_call();
       if (result==0) {
-        UnbufferedStream_status(stream) = -1; return ls_eof;
+        if (!buffered)
+          UnbufferedStream_status(stream) = -1;
+        return ls_eof;
       } else {
         # Stuff the read byte into the buffer, for next low_read call.
-        UnbufferedStreamLow_push_byte(stream,b);
+        handle_read (stream, b, result);
+        # UnbufferedStreamLow_push_byte(stream,b);
         return ls_avail;
       }
     }
@@ -4934,6 +4937,18 @@
   #endif
 }
 
+local void handle_unbuffered_read (object stream, uintB *buff, uintL amt) {
+  UnbufferedStreamLow_push_byte(stream,*buff);
+}
+
+local signean low_listen_unbuffered_handle (object stream) {
+  if (UnbufferedStream_status(stream) < 0) # already EOF?
+    return ls_eof;
+  if (UnbufferedStream_status(stream) > 0) # bytebuf contains valid bytes?
+    return ls_avail;
+  return low_listen_handle (stream, 1, handle_unbuffered_read);
+}
+
 local bool low_clear_input_unbuffered_handle (object stream) {
   if (nullp(TheStream(stream)->strm_isatty))
     return false; # it's a file -> nothing to do
@@ -5872,10 +5887,10 @@
   uintL (* low_fill)  (object stream);
   void  (* low_flush) (object stream, uintL bufflen);
   uintL buffstart;              # start position of buffer
-  sintL eofindex;               # index up to which the data is valid
-                                # (for recognizing EOF)
-  #define eofindex_all_invalid  (-1)
-  #define eofindex_all_valid    (-2)
+  sintL endvalid;               # index up to which the data is valid
+  bool eof_hit : 8;             # have we hit the EOF?
+  # define eofindex_all_invalid  (-1)
+  # define eofindex_all_valid    (-2)
   uintL index;                  # index into buffer (>=0, <=strm_buffered_bufflen)
   bool modified : 8;         # true if the buffer contains modified data, else false
   bool regular : 8;          # whether the handle refers to a regular file
@@ -5928,8 +5943,12 @@
   ((strm_buffered_extrafields_t*)&TheStream(stream)->strm_channel_extrafields)->buffstart
 #define BufferedStream_eofindex(stream)  \
   ((strm_buffered_extrafields_t*)&TheStream(stream)->strm_channel_extrafields)->eofindex
+#define BufferedStream_endvalid(stream)  \
+  ((strm_buffered_extrafields_t*)&TheStream(stream)->strm_channel_extrafields)->endvalid
 #define BufferedStream_index(stream)  \
   ((strm_buffered_extrafields_t*)&TheStream(stream)->strm_channel_extrafields)->index
+#define BufferedStream_eof_hit(stream)  \
+  ((strm_buffered_extrafields_t*)&TheStream(stream)->strm_channel_extrafields)->eof_hit
 #define BufferedStream_modified(stream)  \
   ((strm_buffered_extrafields_t*)&TheStream(stream)->strm_channel_extrafields)->modified
 #define BufferedStream_regular(stream)  \
@@ -6108,8 +6127,8 @@
                  BufferedStream_buffstart(stream),SEEK_SET,); # positioning back
     end_system_call();
   }
-  # write eofindex Bytes:
-  BufferedStreamLow_flush(stream)(stream,BufferedStream_eofindex(stream));
+  # write endvalid Bytes:
+  BufferedStreamLow_flush(stream)(stream,BufferedStream_endvalid(stream));
 }
 
 # UP: Writes the modified Buffer back.
@@ -6118,7 +6137,7 @@
 # < modified_flag of stream : deleted
 # changed in stream: index
 local void buffered_flush (object stream) {
-  if (BufferedStream_eofindex(stream) == eofindex_all_valid) # Buffer entirely valid?
+  if (BufferedStream_endvalid(stream) == strm_buffered_bufflen) # Buffer entirely valid?
     buffered_full_flush(stream);
   else
     buffered_half_flush(stream);
@@ -6131,50 +6150,43 @@
 # < result : NULL if EOF (and then index=eofindex),
 #                 else: Pointer to the next Byte
 # changed in stream: index, eofindex, buffstart
-local uintB* buffered_nextbyte (object stream) {
-  var sintL eofindex = BufferedStream_eofindex(stream);
+local uintB* buffered_nextbyte (object stream, bool* non_block) {
+  var sintL endvalid = BufferedStream_endvalid(stream);
   var uintL index = BufferedStream_index(stream);
-  if (!(eofindex == eofindex_all_valid)) { # Bufferdata only half valid
-    if (eofindex == eofindex_all_invalid) # Bufferdata entirely invalid
-      goto reread;
-    else # EOF occurs in this Sector
-      goto eofsector;
-  }
-  # Bufferdata entirely valid
-  if (index != strm_buffered_bufflen) { # index = bufflen ?
-    # no, so 0 <= index < strm_buffered_bufflen -> OK
-    return BufferedStream_buffer_address(stream,index);
-  }
-  # Buffer must be newly filled.
-  if (BufferedStream_modified(stream))
-    # Beforehand the Buffer must be flushed out:
-    buffered_full_flush(stream);
-  BufferedStream_buffstart(stream) += strm_buffered_bufflen;
- reread: { # From here, read the Buffer newly:
+
+  if ((endvalid == index) && !BufferedStream_eof_hit (stream)) {
+    if (non_block) {
+      *non_block = true;
+      return (uintB*) NULL;
+    }
+    if (BufferedStream_modified(stream))
+      # Beforehand the Buffer must be flushed out:
+      buffered_full_flush(stream);
+    BufferedStream_buffstart(stream) += endvalid;
+
     var sintL result;
     if (BufferedStream_blockpositioning(stream)
-        || !((TheStream(stream)->strmflags & strmflags_rd_B) == 0)) {
+        || (TheStream(stream)->strmflags & strmflags_rd_B)) {
       result = BufferedStreamLow_fill(stream)(stream);
-      if (result==strm_buffered_bufflen) { # the entire Buffer was filled
-        BufferedStream_index(stream) = 0; # Index := 0
-        BufferedStream_modified(stream) = false; # Buffer unmodified
-        BufferedStream_eofindex(stream) = eofindex_all_valid; # eofindex := all_valid
-        return BufferedStream_buffer_address(stream,0);
-      }
     } else {
       result = 0;
     }
-    # result (< strm_buffered_bufflen) Bytes were read.
-    # Not the entire Buffer was filled -> EOF is reached.
-    BufferedStream_index(stream) = index = 0; # Index := 0
-    BufferedStream_modified(stream) = false; # Buffer unmodified
-    BufferedStream_eofindex(stream) = eofindex = result; # eofindex := result
-  }
- eofsector: # eofindex is a Fixnum, i.e. EOF occurs in this Sector.
-  if (index == eofindex)
-    return (uintB*)NULL; # EOF reached
-  else
+    BufferedStream_index(stream) = index = 0;
+    BufferedStream_modified(stream) = false;
+    BufferedStream_endvalid(stream) = endvalid = result;
+    if (result == 0) {
+      BufferedStream_eof_hit (stream) = true;
+    }
+  }
+  if (index < endvalid) {
+    # no, so 0 <= index < strm_buffered_bufflen -> OK
+    if (non_block) *non_block = false;
     return BufferedStream_buffer_address(stream,index);
+  } elif (BufferedStream_eof_hit (stream)) {
+    if (non_block) *non_block = false;
+    return (uintB*) NULL;
+  } else
+    ASSERT (false);
 }
 
 # UP: Prepares the writing of a Byte at EOF.
@@ -6185,19 +6197,19 @@
 # changed in stream: index, eofindex, buffstart
 local uintB* buffered_eofbyte (object stream) {
   # EOF.  eofindex=index.
-  if (BufferedStream_eofindex(stream) == strm_buffered_bufflen) {
+  if (BufferedStream_endvalid(stream) == strm_buffered_bufflen) {
     # eofindex = strm_buffered_bufflen
     # Buffer must be filled newly. Because after that EOF will occur anyway,
     # it is sufficient, to flush the Buffer out:
     if (BufferedStream_modified(stream))
       buffered_half_flush(stream);
     BufferedStream_buffstart(stream) += strm_buffered_bufflen;
-    BufferedStream_eofindex(stream) = 0; # eofindex := 0
+    BufferedStream_endvalid(stream) = 0; # eofindex := 0
     BufferedStream_index(stream) = 0; # index := 0
     BufferedStream_modified(stream) = false; # unmodified
   }
   # increase eofindex:
-  BufferedStream_eofindex(stream) += 1;
+  BufferedStream_endvalid(stream) += 1;
   return BufferedStream_buffer_address(stream,BufferedStream_index(stream));
 }
 
@@ -6207,7 +6219,7 @@
 # > b : Byte to be written
 # changed in stream: index, eofindex, buffstart
 local void buffered_writebyte (object stream, uintB b) {
-  var uintB* ptr = buffered_nextbyte(stream);
+  var uintB* ptr = buffered_nextbyte(stream,0);
   if (!(ptr == (uintB*)NULL)) {
     if (*ptr == b) # no real Modification?
       goto no_modification;
@@ -6241,12 +6253,9 @@
 local void position_file_buffered (object stream, uintL position) {
   # Is the new Position in the same Sector?
   {
-    var sintL eofindex = BufferedStream_eofindex(stream);
+    var sintL endvalid = BufferedStream_endvalid(stream);
     var uintL newindex = position - BufferedStream_buffstart(stream);
-    if (newindex
-        <= ((eofindex == eofindex_all_valid) ? strm_buffered_bufflen :
-            (!(eofindex == eofindex_all_invalid)) ? eofindex :
-            0)) { # yes -> only index has to be changed:
+    if (newindex <= endvalid) {
       BufferedStream_index(stream) = newindex;
       return;
     }
@@ -6260,7 +6269,7 @@
     handle_lseek(stream,BufferedStream_channel(stream),position,SEEK_SET,);
     end_system_call();
     BufferedStream_buffstart(stream) = position;
-    BufferedStream_eofindex(stream) = eofindex_all_invalid; # eofindex := all_invalid
+    BufferedStream_endvalid(stream) = 0; # eofindex := all_invalid
     BufferedStream_index(stream) = 0; # index := 0
     BufferedStream_modified(stream) = false; # unmodified
   } else {
@@ -6275,17 +6284,17 @@
       BufferedStream_buffstart(stream) = newposition;
     }
     # read Sector:
-    BufferedStream_eofindex(stream) = eofindex_all_invalid; # eofindex := all_invalid
+    BufferedStream_endvalid(stream) = 0; # eofindex := all_invalid
     BufferedStream_index(stream) = 0; # index := 0
     BufferedStream_modified(stream) = false; # unmodified
     var uintL newindex = position % strm_buffered_bufflen; # desired Index in the Sector
     if (!(newindex==0)) { # Position between Sectors -> nothing needs to be read
-      buffered_nextbyte(stream);
+      buffered_nextbyte(stream,0);
       # Now index=0.
       # set index to (position mod bufflen) , but check beforehand:
-      var sintL eofindex = BufferedStream_eofindex(stream);
+      var sintL endvalid = BufferedStream_endvalid(stream);
       # Either eofindex=all_valid or 0<=newindex<=eofindex must be true:
-      if (!((eofindex == eofindex_all_valid) || (newindex <= eofindex))) {
+      if (newindex > endvalid) {
         # Error. But first position back to the old Position:
         check_SP();
         position_file_buffered(stream,oldposition); # position back
@@ -6308,13 +6317,11 @@
 local uintB* read_byte_array_buffered (object stream, uintB* byteptr,
                                        uintL len) {
   do {
-    var uintB* ptr = buffered_nextbyte(stream);
+    var uintB* ptr = buffered_nextbyte(stream,0);
     if (ptr == (uintB*)NULL)
       break;
-    var sintL eofindex = BufferedStream_eofindex(stream);
-    var uintL available =
-      (eofindex == eofindex_all_valid ? strm_buffered_bufflen : eofindex)
-      - BufferedStream_index(stream);
+    var sintL endvalid = BufferedStream_endvalid(stream);
+    var uintL available = endvalid - BufferedStream_index(stream);
     if (available > len)
       available = len;
     # copy all available bytes:
@@ -6341,13 +6348,12 @@
   var uintL remaining = len;
   var uintB* ptr;
   do { # still remaining>0 Bytes to be filed.
-    ptr = buffered_nextbyte(stream);
+    ptr = buffered_nextbyte(stream,0);
     if (ptr == (uintB*)NULL)
       goto eof_reached;
-    var sintL eofindex = BufferedStream_eofindex(stream);
+    var sintL endvalid = BufferedStream_endvalid(stream);
     var uintL next = # as many as still fit in the Buffer or until EOF
-      ((eofindex==eofindex_all_valid) ? strm_buffered_bufflen : eofindex)
-      - BufferedStream_index(stream); # > 0 !
+      endvalid - BufferedStream_index(stream); # > 0 !
     if (next > remaining)
       next = remaining;
     { # copy next Bytes in the Buffer:
@@ -6374,8 +6380,8 @@
         # so it is sufficient to flush the buffer:
         if (BufferedStream_modified(stream))
           buffered_half_flush(stream);
-        BufferedStream_buffstart(stream) += strm_buffered_bufflen;
-        BufferedStream_eofindex(stream) = 0; # eofindex := 0
+        BufferedStream_buffstart(stream) += BufferedStream_endvalid(stream);
+        BufferedStream_endvalid(stream) = 0; # eofindex := 0
         BufferedStream_index(stream) = 0; # index := 0
         BufferedStream_modified(stream) = false; # unmodified
         # Then try again:
@@ -6392,7 +6398,7 @@
       remaining = remaining - next;
       # increment index and eofindex
       BufferedStream_index(stream) += next;
-      BufferedStream_eofindex(stream) += next;
+      BufferedStream_endvalid(stream) += next;
     } while (remaining != 0);
   }
   return byteptr;
@@ -6407,7 +6413,7 @@
 # READ-CHAR - Pseudo-Function for File-Streams of Characters
 local object rd_ch_buffered (const object* stream_) {
   var object stream = *stream_;
-  var uintB* bufferptr = buffered_nextbyte(stream);
+  var uintB* bufferptr = buffered_nextbyte(stream,0);
   if (bufferptr == (uintB*)NULL) # EOF ?
     return eof_value;
   # fetch next character:
@@ -6416,10 +6422,8 @@
   var object encoding = TheStream(stream)->strm_encoding;
   # Does the buffer contain a complete character?
   {
-    var sintL eofindex = BufferedStream_eofindex(stream);
-    var uintL available =
-      (eofindex == eofindex_all_valid ? strm_buffered_bufflen : eofindex)
-      - BufferedStream_index(stream);
+    var sintL endvalid = BufferedStream_endvalid(stream);
+    var uintL available = endvalid - BufferedStream_index(stream);
     var const uintB* bptr = bufferptr;
     var chart* cptr = &c;
     Encoding_mbstowcs(encoding)
@@ -6461,7 +6465,7 @@
           }
           break;
         }
-        bufferptr = buffered_nextbyte(stream);
+        bufferptr = buffered_nextbyte(stream,0);
         if (bufferptr == (uintB*)NULL)
           return eof_value;
       }
@@ -6477,7 +6481,7 @@
     ChannelStream_lineno(stream) += 1;
   } else if (chareq(c,ascii(CR))) {
     # check next character for LF
-    bufferptr = buffered_nextbyte(stream);
+    bufferptr = buffered_nextbyte(stream,0);
     if ((bufferptr != NULL) && chareq(as_chart(*bufferptr),ascii(LF))) {
       # increment index and position
       BufferedStream_index(stream) += 1;
@@ -6489,6 +6493,22 @@
   return code_char(c);
 }
 
+
+local void handle_buffered_read (object stream, uintB *buff, uintL amt) {
+  # FIXME: if modified, really need to flush, and then seek back to the
+  # right place.  ugh.
+  # hrif (BufferedStream_modified(stream))
+      # Beforehand the Buffer must be flushed out:
+  #    buffered_full_flush(stream);
+  BufferedStream_buffstart(stream) += BufferedStream_endvalid(stream);
+
+  BufferedStream_index(stream) = 0;
+  BufferedStream_modified(stream) = false;
+  BufferedStream_endvalid(stream) = amt;
+  memcpy(BufferedStream_buffer_address(stream,0),
+         buff, amt);
+}
+
 # Determines, if a character is available on a File-Stream.
 # listen_char_buffered(stream)
 # > stream: File-Stream of Characters
@@ -6496,13 +6516,19 @@
 #             ls_eof   if EOF is reached,
 #             ls_wait  if no character is available, but not because of EOF
 local signean listen_char_buffered (object stream) {
-  if (buffered_nextbyte(stream) == (uintB*)NULL)
-    return ls_eof; # EOF
+  var bool might_block = false;
+  var uintB* ptr = buffered_nextbyte(stream,&might_block);
   # In case of UNICODE, the presence of a byte does not guarantee the
   # presence of a multi-byte character. Returning ls_avail here is
   # therefore not correct. But this doesn't matter since programs seeing
   # ls_avail will call read-char, and this will do the right thing anyway.
-  return ls_avail;
+  if (ptr)
+    return ls_avail;
+  elif (!might_block)
+    return ls_eof; # EOF
+  else
+    return low_listen_handle (stream, strm_buffered_bufflen,
+                              handle_buffered_read);
 }
 
 # READ-CHAR-ARRAY - Pseudo-Function for File-Streams of Characters:
@@ -6517,15 +6543,13 @@
   var chart* endptr = startptr+len;
   loop {
     var chart* startptr = charptr;
-    var uintB* bufferptr = buffered_nextbyte(stream);
+    var uintB* bufferptr = buffered_nextbyte(stream,0);
     if (bufferptr == (uintB*)NULL) # EOF -> finished
       break;
     # Read as many complete characters from the buffer as possible.
     {
-      var sintL eofindex = BufferedStream_eofindex(stream);
-      var uintL available =
-        (eofindex == eofindex_all_valid ? strm_buffered_bufflen : eofindex)
-        - BufferedStream_index(stream);
+      var sintL endvalid = BufferedStream_endvalid(stream);
+      var uintL available = endvalid - BufferedStream_index(stream);
       var const uintB* bptr = bufferptr;
       var chart* cptr = charptr;
       Encoding_mbstowcs(encoding)
@@ -6565,7 +6589,7 @@
             }
             break;
           }
-          bufferptr = buffered_nextbyte(stream);
+          bufferptr = buffered_nextbyte(stream,0);
           if (bufferptr == (uintB*)NULL) # EOF -> finished
             break;
         }
@@ -6586,7 +6610,7 @@
         } else if (chareq(c,ascii(CR))) {
           # check next character for LF
           if (ptr1 == charptr) {
-            uintB* bufferptr = buffered_nextbyte(stream);
+            uintB* bufferptr = buffered_nextbyte(stream,0);
             if ((bufferptr != NULL)
                 && chareq(as_chart(*bufferptr),ascii(LF))) {
               # increment index and position
@@ -6610,7 +6634,7 @@
   return charptr - startptr;
   #else
   do {
-    var uintB* ptr = buffered_nextbyte(stream);
+    var uintB* ptr = buffered_nextbyte(stream,0);
     if (ptr == (uintB*)NULL) # EOF -> finished
       break;
     var chart ch = as_chart(*ptr);
@@ -6621,7 +6645,7 @@
       ChannelStream_lineno(stream) += 1;
     } else if (chareq(ch,ascii(CR))) {
       # check next character for LF
-      ptr = buffered_nextbyte(stream);
+      ptr = buffered_nextbyte(stream,0);
       if (!(ptr == (uintB*)NULL) && chareq(as_chart(*ptr),ascii(LF))) {
         # increment index and position
         BufferedStream_index(stream) += 1;
@@ -6937,7 +6961,7 @@
     return;
   if (# Is the addressed position situated in the first byte after EOF ?
       ((!((position_bits%8)==0))
-       && (buffered_nextbyte(stream) == (uintB*)NULL))
+       && (buffered_nextbyte(stream,0) == (uintB*)NULL))
        # Is the addressed position situated in the last byte too far?
       || ((bitsize < 8)
           && (position > BufferedStream_eofposition(stream)))) {
@@ -6967,7 +6991,7 @@
   #if 0 # equivalent, but slower
   var uintL count;
   dotimespL(count,bytesize, {
-    var uintB* ptr = buffered_nextbyte(stream);
+    var uintB* ptr = buffered_nextbyte(stream,0);
     if (ptr == (uintB*)NULL)
       goto eof;
     # fetch next Byte:
@@ -7004,7 +7028,7 @@
     var uintL bitindex = BufferedStream_bitindex(stream);
     var uintL count = bitindex + bitsize;
     var uint8 bit_akku;
-    var uintB* ptr = buffered_nextbyte(stream);
+    var uintB* ptr = buffered_nextbyte(stream,0);
     if (ptr == (uintB*)NULL)
       goto eof;
     # Get first partial byte:
@@ -7015,7 +7039,7 @@
       # increment index, because *ptr is processed:
       BufferedStream_index(stream) += 1;
       count -= 8; # still count (>0) Bits to fetch.
-      var uintB* ptr = buffered_nextbyte(stream);
+      var uintB* ptr = buffered_nextbyte(stream,0);
       if (ptr == (uintB*)NULL)
         goto eof1;
       # fetch next Byte:
@@ -7051,7 +7075,7 @@
     &TheSbvector(TheStream(stream)->strm_bitbuffer)->data[0];
   var uintL count = bitsize;
   var uintL bitshift = BufferedStream_bitindex(stream);
-  var uintB* ptr = buffered_nextbyte(stream);
+  var uintB* ptr = buffered_nextbyte(stream,0);
   if (ptr != (uintB*)NULL) {
     # start getting bytes:
     var uint16 bit_akku = (*ptr)>>bitshift;
@@ -7062,7 +7086,7 @@
       # bit_akku: bits (bitshift-1)..0 are valid.
       # have to get count (>0) bits.
       {
-        var uintB* ptr = buffered_nextbyte(stream);
+        var uintB* ptr = buffered_nextbyte(stream,0);
         if (ptr == (uintB*)NULL)
           goto eof;
         # get next byte:
@@ -7076,7 +7100,7 @@
       count -= 8;
     }
     # count > 0 -- the number of bits to get
-    ptr = buffered_nextbyte(stream);
+    ptr = buffered_nextbyte(stream,0);
     if (ptr == (uintB*)NULL) # EOF ?
       bit_akku = *buffered_eofbyte(stream);
     *bitbufferptr = (uint8)(bit_akku & (uint8)(bit(count)-1));
@@ -7121,7 +7145,7 @@
 
 # READ-BYTE - Pseudo-Function for File-Streams of Integers, Type au, bitsize = 8 :
 local object rd_by_iau8_buffered (object stream) {
-  var uintB* ptr = buffered_nextbyte(stream);
+  var uintB* ptr = buffered_nextbyte(stream,0);
   if (!(ptr == (uintB*)NULL)) {
     var object obj = fixnum(*ptr);
     # increment index and position
@@ -7152,9 +7176,15 @@
 #             ls_eof   if EOF is reached,
 #             ls_wait  if no byte is available, but not because of EOF
 local signean listen_byte_ia8_buffered (object stream) {
-  if (buffered_nextbyte(stream) == (uintB*)NULL)
-    return ls_eof; # EOF
-  return ls_avail;
+  var bool might_block = false;
+  var uintB* ptr = buffered_nextbyte(stream,&might_block);
+  if (ptr)
+    return ls_avail;
+  elif (!might_block)
+    return ls_eof;
+  else
+    return low_listen_handle (stream, strm_buffered_bufflen,
+                              handle_buffered_read);
 }
 
 # Output side
@@ -7179,7 +7209,7 @@
 # write last byte (count bits):
 #define WRITE_LAST_BYTE                                                 \
  if (!(count==0)) {                                                     \
-   ptr = buffered_nextbyte(stream);                                     \
+   ptr = buffered_nextbyte(stream,0);                                     \
    if (ptr == (uintB*)NULL) { /* EOF */                                 \
      ptr = buffered_eofbyte(stream); /* 1 Byte */                       \
      *ptr = (uint8)bit_akku; /* write byte */                           \
@@ -7199,7 +7229,7 @@
   var uintL bitshift = BufferedStream_bitindex(stream);
   var uint16 bit_akku = (uint16)(TheSbvector(TheStream(stream)->strm_bitbuffer)->data[0])<<bitshift;
   var uintL count = bitsize;
-  var uintB* ptr = buffered_nextbyte(stream);
+  var uintB* ptr = buffered_nextbyte(stream,0);
   # start getting bytes:
   if (!(ptr == (uintB*)NULL))
     bit_akku |= (*ptr)&(bit(bitshift)-1);
@@ -7225,7 +7255,7 @@
   var uintL bitshift = BufferedStream_bitindex(stream);
   var uintL count = bitsize;
   var uint16 bit_akku;
-  var uintB* ptr = buffered_nextbyte(stream);
+  var uintB* ptr = buffered_nextbyte(stream,0);
   # start getting bytes:
   bit_akku = (ptr==(uintB*)NULL ? 0 : (*ptr)&(bit(bitshift)-1) );
   count += bitshift;
@@ -7378,9 +7408,10 @@
   if (!BufferedStream_blockpositioning(stream)) {
     # Now position at the End:
     BufferedStream_buffstart(stream) = eofbytes;
-    BufferedStream_eofindex(stream) = eofindex_all_invalid; # eofindex := all_invalid
+    BufferedStream_endvalid(stream) = 0; # eofindex := all_invalid
     BufferedStream_index(stream) = 0; # index := 0
     BufferedStream_modified(stream) = false; # unmodified
+    BufferedStream_eof_hit(stream) = true; # unmodified
   } else { # position to the start of the last Sector:
     {
       var uintL buffstart;
@@ -7392,17 +7423,17 @@
       BufferedStream_buffstart(stream) = buffstart;
     }
     # read Sector:
-    BufferedStream_eofindex(stream) = eofindex_all_invalid; # eofindex := all_invalid
+    BufferedStream_endvalid(stream) = 0; # eofindex := all_invalid
     BufferedStream_index(stream) = 0; # index := 0
     BufferedStream_modified(stream) = false; # unmodified
     var uintL eofindex = eofbytes % strm_buffered_bufflen;
     if (!((eofindex==0) && (eofbits==0))) { # EOF at end of Sector -> nothing to read
-      buffered_nextbyte(stream);
+      buffered_nextbyte(stream,0);
       # Now index=0. set index and eofindex:
       BufferedStream_index(stream) = eofindex;
       if (!(eofbits==0))
         eofindex += 1;
-      BufferedStream_eofindex(stream) = eofindex;
+      BufferedStream_endvalid(stream) = eofindex;
     }
   }
   if (!((bitsize % 8) == 0)) { # Integer-Stream of type b,c
@@ -7534,7 +7565,8 @@
         stream = popSTACK();
         BufferedStream_buffer(stream) = buffer;
       }
-      BufferedStream_eofindex(stream) = eofindex_all_invalid; # eofindex := all_invalid
+      BufferedStream_endvalid(stream) = 0; # eofindex := all_invalid
+      BufferedStream_eof_hit(stream) = false;
       BufferedStream_index(stream) = 0; # index := 0
       BufferedStream_modified(stream) = false; # Buffer unmodified
       BufferedStream_position(stream) = 0; # position := 0
@@ -7688,7 +7720,7 @@
       var uintL eofposition = 0;
       var uintC count;
       for (count=0; count < 8*sizeof(uintL); count += 8 ) {
-        var uintB* ptr = buffered_nextbyte(stream);
+        var uintB* ptr = buffered_nextbyte(stream,0);
         if (ptr == (uintB*)NULL)
           goto too_short;
         eofposition |= ((*ptr) << count);
@@ -7866,7 +7898,7 @@
   # and reposition:
   var uintL position = BufferedStream_buffstart(stream) + BufferedStream_index(stream);
   BufferedStream_index(stream) = 0; # index := 0
-  BufferedStream_eofindex(stream) = eofindex_all_invalid; # eofindex := all_invalid
+  BufferedStream_endvalid(stream) = 0; # eofindex := all_invalid
   if (!BufferedStream_blockpositioning(stream)) {
     BufferedStream_buffstart(stream) = position;
   } else {
@@ -7892,7 +7924,7 @@
   BufferedStream_channel(stream) = NIL; # Handle becomes invalid
   BufferedStream_buffer(stream) = NIL; # free Buffer
   BufferedStream_buffstart(stream) = 0; # delete buffstart (unnecessary)
-  BufferedStream_eofindex(stream) = eofindex_all_invalid; # delete eofindex (unnecessary)
+  BufferedStream_endvalid(stream) = 0; # delete eofindex (unnecessary)
   BufferedStream_index(stream) = 0; # delete index (unnecessary)
   BufferedStream_modified(stream) = false; # delete modified_flag (unnecessary)
   BufferedStream_position(stream) = 0; # delete position (unnecessary)
@@ -14866,6 +14898,56 @@
   }
 }
 
+local SOCKET stream_handle (object stream) {
+  if (builtin_stream_p(stream)) {
+    switch (TheStream(stream)->strmtype) {
+      case strmtype_socket:
+        return SocketChannel(test_socket_stream(stream,true));
+      case strmtype_pipe_in:
+        return (SOCKET)TheHandle(ChannelStream_buffered(stream)
+                                 ? BufferedStream_channel (stream)
+                                 : ChannelStream_ichannel (stream));
+      case strmtype_pipe_out:
+        return (SOCKET)TheHandle(ChannelStream_buffered(stream)
+                                 ? BufferedStream_channel (stream)
+                                 : ChannelStream_ochannel (stream));
+      default:
+        break;
+    }
+  }
+  pushSTACK(stream);       # TYPE-ERROR slot DATUM
+  pushSTACK(S(stream)); # TYPE-ERROR slot EXPECTED-TYPE
+  pushSTACK(stream);
+  pushSTACK(TheSubr(subr_self)->name);
+  fehler(type_error,GETTEXT("~: argument ~ is not a SOCKET-STREAM"));
+}
+
+local object reduce_stream (object stream, direction_t dir) {
+  if (builtin_stream_p(stream)) {
+    switch (TheStream(stream)->strmtype) {
+      case strmtype_twoway_socket:
+        return dir==DIRECTION_INPUT
+               ? TheStream(stream)->strm_twoway_socket_input
+               : TheStream(stream)->strm_twoway_socket_output;
+      case strmtype_pipe_in:
+      case strmtype_pipe_out:
+      case strmtype_socket:
+        return stream;
+      case strmtype_twoway:
+        return dir==DIRECTION_INPUT
+               ? reduce_stream(TheStream(stream)->strm_twoway_input,dir)
+               : reduce_stream(TheStream(stream)->strm_twoway_output,dir);
+      default:
+        break;
+    }
+  }
+  pushSTACK(stream);       # TYPE-ERROR slot DATUM
+  pushSTACK(S(stream)); # TYPE-ERROR slot EXPECTED-TYPE
+  pushSTACK(stream);
+  pushSTACK(TheSubr(subr_self)->name);
+  fehler(type_error,GETTEXT("~: argument ~ is not a SOCKET-STREAM"));
+}
+
 # set the appropriate fd_sets for the socket,
 # either a socket-server, a socket-stream or a (socket . direction)
 # see socket_status() for details
@@ -14873,13 +14955,23 @@
                        fd_set *errorfds) {
   object sock = (consp(socket) ? Car(socket) : socket);
   direction_t dir = (consp(socket)?check_direction(Cdr(socket)):DIRECTION_IO);
-  SOCKET handle = socket_handle(sock,true);
-  FD_SET(handle,errorfds);
   if (socket_server_p(sock)) {
-    if (READ_P(dir)) FD_SET(handle,readfds);
-  } else { # sock is a socket stream
-    if (READ_P(dir)  && input_stream_p(sock))  FD_SET(handle,readfds);
-    if (WRITE_P(dir) && output_stream_p(sock)) FD_SET(handle,writefds);
+    if (READ_P(dir)) {
+      SOCKET handle = socket_handle(sock,true);
+      FD_SET(handle,readfds);
+      FD_SET(handle,errorfds);
+    }
+  } else { # sock should be a stream
+    if (READ_P(dir)  && input_stream_p(sock)) {
+      SOCKET handle = stream_handle(reduce_stream(sock,DIRECTION_INPUT));
+      FD_SET(handle,readfds);
+      FD_SET(handle,errorfds);
+    }
+    if (WRITE_P(dir) && output_stream_p(sock)) {
+      SOCKET handle = stream_handle(reduce_stream(sock,DIRECTION_OUTPUT));
+      FD_SET(handle,writefds);
+      FD_SET(handle,errorfds);
+    }
   }
 }
 
@@ -14891,22 +14983,34 @@
                            fd_set *errorfds) {
   object sock = (consp(socket) ? Car(socket) : socket);
   direction_t dir = (consp(socket)?check_direction(Cdr(socket)):DIRECTION_IO);
-  SOCKET handle = socket_handle(sock,true);
-  if (FD_ISSET(handle,errorfds)) return S(Kerror);
-  else {
-    if (socket_server_p(sock)) {
-      return FD_ISSET(handle,readfds) ? T : NIL;
-    } else {
-      bool wr = WRITE_P(dir) && FD_ISSET(handle,writefds),
-        rd = (READ_P(dir) && FD_ISSET(handle,readfds)
-              && (ls_avail_p(stream_char_p(sock)
-                             ? listen_char(sock)
-                             : listen_byte(sock))));
-      if      ( rd && !wr) return S(Kinput);
-      else if (!rd &&  wr) return S(Koutput);
-      else if ( rd &&  wr) return S(Kio);
-      else                 return NIL;
-    }
+  if (socket_server_p(sock)) {
+    SOCKET handle = socket_handle(sock,true);
+    if (FD_ISSET(handle,errorfds)) return S(Kerror);
+    return FD_ISSET(handle,readfds) ? T : NIL;
+  } else {
+    bool wr = 0, rd = 0, in_err = 0, out_err = 0;
+    if (WRITE_P(dir)) {
+      SOCKET handle = stream_handle(reduce_stream(sock,DIRECTION_OUTPUT));
+      wr = FD_ISSET(handle,writefds);
+      out_err = FD_ISSET(handle,errorfds);
+    }
+    if (READ_P(dir)) {
+      object str = reduce_stream(sock,DIRECTION_INPUT);
+      SOCKET handle = stream_handle(str);
+      in_err = FD_ISSET(handle, errorfds);
+      if (FD_ISSET(handle,readfds)
+#if 0
+          && (ls_avail_p(stream_char_p(str)
+                         ? listen_char(str)
+                         : listen_byte(str)))
+#endif
+          )
+        rd = 1;
+    }
+    if      ( rd && !wr) return S(Kinput);
+    else if (!rd &&  wr) return S(Koutput);
+    else if ( rd &&  wr) return S(Kio);
+    else                 return NIL;
   }
   return NIL;
 }
Index: unixaux.d
===================================================================
RCS file: /cvsroot/clisp/clisp/src/unixaux.d,v
retrieving revision 1.20
diff -u -r1.20 unixaux.d
--- unixaux.d	2001/02/05 19:29:45	1.20
+++ unixaux.d	2002/01/21 18:40:30
@@ -279,20 +279,12 @@
       # does not specify anything about possible side effects.
       handle_fault_range(PROT_READ_WRITE,(aint)buf,(aint)buf+nbyte);
       #endif
-      until (nbyte==0) {
-        retval = read(fd,buf,nbyte);
-        if (retval == 0)
-          break;
-        elif (retval < 0) {
-          #ifdef EINTR
-          if (!(errno == EINTR))
-          #endif
-            return retval;
-        } else {
-          buf += retval; done += (RW_SIZE_T)retval; nbyte -= (RW_SIZE_T)retval;
-        }
-      }
-      return done;
+    again:
+      retval = read(fd,buf,nbyte);
+      if (retval < 0 && errno == EINTR)
+        goto again;
+      else
+        return retval;
     }
 
 # Ein Wrapper um die write-Funktion.

---

From: Sam Steingold <[email-masked]>
Date: Mon Jan 21 12:21:02 2002

[let's try to move this to <clisp-devel> - Reply-To is set]

> * In message <[email-masked]>
> * Sent on 21 Jan 2002 13:52:53 -0500
> (Ignore the socket-status stuff, since you've already done that.)

I incorporated some of your suggestions too - check it out

BTW, what does this mean:
struct bar {
  bool foo : 8;
  /*       ^^^  what is this? */
}
it appears my C is no good anymore...

> sockets are quite usable, though the hang problem still exists.

cool!  please keep going - I will not delve into this but wait for your
results.

BTW, what is this 4096 world constant?
why are you using it explicitly in the code - why not define a macro?

> fixing this is a big win.
great!

> Comments appreciated.

1. use the latest CVS version - specifically, macros ChannelStream_[io]handle

2. why does your unixaux patch drop the "#ifdef EINTR" check?
   (I am not sure this is the only problem with that patch...)

3. shouldn't `end_valid' be uintL, not sintL?  eofindex had to be
   signed because it incorporated other - non-numeric - information too

you might want to split your patch into several smaller ones, e.g., a
separate patch replacing `eofindex' with `end_valid' and `eof_hit_p'
this would make reading the patches easier...

-- 
Sam Steingold (http://www.podval.org/~sds)
Keep Jerusalem united! <http://www.onejerusalem.org/Petition.asp>
Read, think and remember! <http://www.iris.org.il> <http://www.memri.org/>
Who is General Failure and why is he reading my hard disk?