Subject: [clisp-list] TCP-based Network Server in CLISP

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Fri Sep 27 06:43:03 2002

Hi,

I find the current discussion about low-water-marks and using lots of special socket flags likely to run into trouble with all the many TCP/IP stacks around.
I also believe it's complete overkill for Dave Richards' initial problem.

>Have other people written network servers in CLISP?
Most I know have either done something at a trivial (unsafe, not robust) level or given up on using CLISP.

May I suggest something simple, like what I believe many other programmers in other Lisps or languages do and which is portable for sockets: in non-blocking mode, you can read _as much as is available_ using a single read(). I think you can also write() a large number, and the OS will tell you that it couldn't write all and tell you how much it actually wrote. It's then up to you/the application to deal with sending the rest at a later time.

So this E-Mail is a feature request for Lisp-level access to a single OS-level read() and write. READ-SOME-BYTE/SEQUENCE et al.

> By using a read-byte-no-hang I can read data
>until the entire request has been read
Using READ-CHAR-NO-HANG for i/o instead of sequence/array functions is going to keep your program a _toy prototype_ with extremely poor performance. Stated otherwise: if it works not too badly for you, thank the OS (scheduler, dispatcher, TCP stack implementations, system load etc.), not your application.

Sadly, CLISP has no better way yet. I think I once suggested READ-SOME-BYTES and WRITE-SOME-BYTES, which would do a single OS-dependent read() or write(). There was a thread about "buffering and partial read/writes" in February.

>Another approach
>would be to implement "atomic" I/O functions, i.e. a version of
>read-sequence or write-sequence which returned the requested amount or
>actually transferred amount of data.
Exactly my proposal. Please proceed in this direction.

I think it's important to realize, stress and document that the result of a single call to write() is higly OS and file/pipe/socket dependent. E.g. I once debugged a REXX application where the programmer assumed the READ function would return a full read. The story is, the string buffer was not filled up when working on pipes, but would always be when working with disk files...
Thankfully CLISP does full READ on all kinds of files.

Regards,
	Jörg Höhle.

---

From: "Dave Richards" <[email-masked]>
Date: Fri Sep 27 07:25:05 2002

> I find the current discussion about low-water-marks and using
> lots of special socket flags likely to run into trouble with all
> the many TCP/IP stacks around.

This is true.  The low-water mark approach was an attempt at addressing the
non-partial write() problem.  Since neither OS I was testing with supports
it, it's fair to suggest that there are likely to be problems.

The :so-snd/rcvbuf really only address server performance.  :so-keepalive
and :so-oobinline, however, address correctness issues.

> May I suggest something simple, like what I believe many other
> programmers in other Lisps or languages do and which is portable
> for sockets: in non-blocking mode, you can read _as much as is
> available_ using a single read(). I think you can also write() a
> large number, and the OS will tell you that it couldn't write all
> and tell you how much it actually wrote. It's then up to you/the
> application to deal with sending the rest at a later time.

Yes, the final solution is going to require:

	- an interface to select()
	- non-blocking sockets
	- interfaces to partial read()/write()

I'd also like an interface to the real interbal time, which is currently
unavailable because the interruptprocessing logic is using it.

I am quickly coming to the conclusion that CMUCL is a better platform for
this application as these interfaces already exist.

	Dave

---

From: Sam Steingold <[email-masked]>
Date: Fri Sep 27 08:12:03 2002

> * In message <[email-masked]>
> * On the subject of "RE: [clisp-list] TCP-based Network Server in CLISP"
> * Sent on Fri, 27 Sep 2002 07:24:21 -0700
> * Honorable "Dave Richards" <[email-masked]> writes:
>

"final solution" is a nasty word combination.

> 	- an interface to select()

SOCKET-STATUS

> 	- non-blocking sockets

I can add FIONREAD, FIONWRITE & FIONBIO to SOCKET-OPTIONS.
Would it help?

> 	- interfaces to partial read()/write()

partial read is available for buffered streams, I think.
I an add partial write too.

> I am quickly coming to the conclusion that CMUCL is a better platform
> for this application as these interfaces already exist.

I would be aggrieved to see you go.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
"A pint of sweat will save a gallon of blood."          -- George S. Patton

---

From: Sam Steingold <[email-masked]>
Date: Fri Sep 27 08:13:07 2002

> * In message <[email-masked]>
> * On the subject of "[clisp-list] TCP-based Network Server in CLISP"
> * Sent on Fri, 27 Sep 2002 15:41:55 +0200
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>
> programmers in other Lisps or languages do and which is portable for
> sockets: in non-blocking mode, you can read _as much as is available_
> using a single read(). I think you can also write() a large number,
> and the OS will tell you that it couldn't write all and tell you how
> much it actually wrote. It's then up to you/the application to deal
> with sending the rest at a later time.
> 
> So this E-Mail is a feature request for Lisp-level access to a single
> OS-level read() and write. READ-SOME-BYTE/SEQUENCE et al.

right now, _buffered_ stream _input_ works as you describe (a single
read).
output works the same way for buffered and unbuffered streams (this can
be fixed).

So, Dave, could you please try buffered streams?
(you will need the latest CLISP, of course).

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Two wrongs don't make a right, but three rights make a left.

---

From: Arseny Slobodjuck <[email-masked]>
Date: Fri Sep 27 23:52:02 2002

Hello Sam,

> I can add FIONREAD, FIONWRITE & FIONBIO to SOCKET-OPTIONS.

No! Don't add FIONBIO! This may break something, firstly ctrl-c
interruptability.

FIONWRITE seems not available in winsock.

I'm going to set all sockets operations to nonblocking and change
everywhere

operation() to

{ do while (!select());
  operation();
  }

to get rid of WSACancelBlockingCall.

> I an add partial write too.

What do you mean by partial read ? It should be something like
(read-to-string socket-stream) -> string, which always return
immediately. There also can be
(write-from-string socket-stream string frompos count) ->
count-written.

-- 
Best regards,
 Arseny

---

From: "Dave Richards" <[email-masked]>
Date: Sat Sep 28 09:36:01 2002

> >>       - an interface to select()

I have two concerns with SOCKET-STATUS as written:

	- SOCKET-STATUS does not allow one to select() on exceptional (what CLISP
calls error) conditions.
	- SOCKET-STATUS retries upon reception of EINTR.

Errors in the sense of ECONNRESET, ETIMEDOUT, etc. select() true for read
(not exceptional conditions).  Exceptional conditions in TCP, for example,
is used to signal the existence of urgent (out-of-band) data.  select()
won't set a bit in the result fd_sets unless that bit was set in the input
fd_sets.  The :direction value in SOCKET-STATUS can be one of :input,
:output or :io.  Therefore, it is not possible to select for urgent TCP data
using SOCKET-STATUS.  It shuld be possible to specify any combination of
:input, :output, and :exception/:error (whatever) and :io for convenience.
Btw, kudos if you handle server sockets differently and use the :accept
direction for them (which, of course, maps toe read).

Upon reception of a signal, SOCKET-STATUS retries.  The timeval structure
passed into select() is normally read-only and, thus, doesn't reflect the
passage of time between the select() and the signal.  (Some operating
systems implement a version of select() where this isn't true, but they are
a minority.)  Therefore, upon delivery of a signal a loss of temporal
continuity exists.  If select() is re-started then select() may have waited
for much longer than the application had intended.

My server application, for example, is really a loop which handles two types
of events: (1) network traffic events and (2) time passage events.  There
are three network traffic events: (1) new input data has arrived, (2) the
application has buffered output and there is now room on the socket to write
some/all of it to the socket, and (3) a new TCP connection has arrived.
We're close to being able to handle these network traffic events.

Time passage events are orthoginal, for the most part, but do impact how we
handle select() EINTRs.  Time passage can be handled in one of two ways: (1)
polling, or (2) via signals.  Either way, this impacts how we handle EINTR
in select().  My current C code uses polling.  The last set of gprof data I
collected showed me in gettimeofday() 3% of the time!  My goal/desire would
be to use an interval timer in the next version.

One day I want to address SIGALRM in CLISP (as part of the passage of time
event handling) but that's a different discussion.

	Dave

---

From: Sam Steingold <[email-masked]>
Date: Sat Sep 28 17:11:04 2002

> * In message <[email-masked]>
> * On the subject of "RE: Re[2]: [clisp-list] TCP-based Network Server in CLISP"
> * Sent on Sat, 28 Sep 2002 09:35:14 -0700

is does!
look at the sources (and docs): SOCKET-STATUS can return :ERROR for a
socket.

> The :direction value in SOCKET-STATUS can be one of :input, :output or
> :io.  Therefore, it is not possible to select for urgent TCP data
> using SOCKET-STATUS.  It shuld be possible to specify any combination
> of :input, :output, and :exception/:error (whatever) and :io for
> convenience.

CLISP always adds the socket so errorfds fd_set, i.e., CLISP always
checks for errors.
(sock . :error) would mean to wait _only_ for errors.
when is this useful?

> be to use an interval timer in the next version.

I am not sure I understand this.
would you prefer CLISP to signal an error on EINTR?
C-c does interrupt SOCKET-STATUS already, right?

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
If you think big enough, you'll never have to do it.

---

From: "Dave Richards" <[email-masked]>
Date: Sat Sep 28 18:01:02 2002

> look at the sources (and docs): SOCKET-STATUS can return :ERROR for a

Good grief!  I missed that when I was reading the code.  You're right.
:input (in_sock) implies :error *pre-select()).  My apologies.

> (sock . :error) would mean to wait _only_ for errors.

The way it's coded is perfectly adequate.

> C-c does interrupt SOCKET-STATUS already, right?

Well, rather than dictate a solution (since there are several possibilities)
let me state the problem:

	1. select() for each socket until network traffix events occurs, i.e.
:input, :output, :io, :error.
	2. Every second execute a LISP function which handles time-based work.

In a single-threaded environment like CLISP, network traffic and time events
must be multiplexed in such a way that both needs are met (more or less
efficiently).  One approach (the one my C code uses) is to pass the time
value (1 second) to select().  This ensures that we'll exit select() in time
to meet our next time event.  Unfortunately, with many users connected, we
can exit select() many times per second.  Which means we must call
gettimeofday() every loop iteration to see how much longer before we should
process our time events.  We subtract the "next time period" from the
current time and pass the difference to select().  This approach, although
simple and seemingly elegant, is also pretty expensive.  The second approach
would be to use a 1 second interval timer.  When the timer expires, the
signal handler would increment a variable.  At the next loop iteration we
can use the variable to decide whether to handle time events (and how many).

If we poll, EINTR handling isn't very important.  In general, CLISP doesn't
receive signals.  Therefore, who really cares how we handle EINTR?

However, in the second (interval timer) case, signals are important.  We use
signals to keep track of time.  If socket-status re-select()s in the EINTR
case, our time event processing may lag.

So, the bottom line is, if we poll for time, re-select()ing isn't a big
deal.  But if we use setitimer() to handle time, EINTR becomes an issue.

Make sense?

	Dave

---

From: Sam Steingold <[email-masked]>
Date: Sat Sep 28 19:10:03 2002

> * In message <[email-masked]>
> * Sent on Sat, 28 Sep 2002 18:00:14 -0700
> simple and seemingly elegant, is also pretty expensive.

in CLISP you will be using GET-INTERNAL-REAL-TIME, right?

> The second approach would be to use a 1 second interval timer.  When
> the timer expires, the signal handler would increment a variable.  At
> the next loop iteration we can use the variable to decide whether to
> handle time events (and how many).
> 

yes.

but since I vaguely recall that CLISP uses alarms for other things, I
think the first way is the right way.

besides, the only reasonable thing to do on EINTR is to signal an error.

I hope more OS-savvy people will weigh in here.
Bruno, Arseny, please!

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Growing Old is Inevitable; Growing Up is Optional.

---

From: Arseny Slobodjuck <[email-masked]>
Date: Sat Sep 28 21:21:02 2002

Hello Dave,

> is used to signal the existence of urgent (out-of-band) data.
Do you want OOB data handling ? My familiarity with OOB was ended up with
this (MSDN)

"To minimize interoperability problems, applications writers are advised not
to use OOB data unless this is required to interoperate with an
existing service."

If you want to know if recv or send operation will succeed you can use
select and filter some situations, but there can be no guarantee that
such operation will succeed. I think only way is to use handler-case
or something like it. Do you want to avoid handler-case ?

> a loss of temporal continuity exists.
please!

-- 
Best regards,
 Arseny

---

From: Arseny Slobodjuck <[email-masked]>
Date: Sat Sep 28 21:21:03 2002

Hello Sam,

> C-c does interrupt SOCKET-STATUS already, right?
I think Dave wants SOCKET-STATUS never wait longer than directed by
time parameter.

I don't know about UNIX, in windows WSAEINTR in select
means WSACancelBlockingCall was called on socket.

-- 
Best regards,
 Arseny

---

From: "Dave Richards" <[email-masked]>
Date: Sat Sep 28 21:49:03 2002

> Do you want OOB data handling ? My familiarity with OOB was ended up with

The TELNET protocol defines a sequence called "synch" which does use urgent
data.  My server is a TELNET server.  So, yes, I need OOB to handle "synch"
correctly (on input).  I never generate a "synch" back to the client.

> to use OOB data unless this is required to interoperate with an

It is.  However, I intend to use the (:so-oobinline t) socket option, so the
OOB data will be delivered "in-band".

> or something like it. Do you want to avoid handler-case ?

My hope was that I'd mark the socket non-blocking and do partial
reads/writes.  I have not started reading the I/O functions yet.  I think
Sam has successfully demonstrated that socket-status ought to meet my needs
for select().  Sadly, I have to do time-differencing, but it works in the
current server, so...  And besides, if that's the only down-side to porting
to CLISP, it's hardly worth mentioning.

	Thanks!

	Dave

---

From: "Dave Richards" <[email-masked]>
Date: Sun Sep 29 11:59:03 2002

> > 	- an interface to select()

Yes.

> I can add FIONREAD, FIONWRITE & FIONBIO to SOCKET-OPTIONS.

FIONREAD "doubles" the round-trips through the kernel.  The first time
through you determine how much to read, the second time you actually acquire
the data.  It's an expensive way to do I/O.

I am not aware of an FIONWRITE.

FIONBIO is really what is needed.  In modern days we typically use
O_NONBLOCK.  This is an fcntl, not a socket option, btw.

> I an add partial write too.

I have been working through stream.d and unixaux.d today.  I have not yet
found code anywhere that handles partial reads or writes in either buffered
or unbuffered streams.  Where should I be looking?

	Dave

---

From: Sam Steingold <[email-masked]>
Date: Sun Sep 29 12:15:02 2002

> * In message <[email-masked]>
> * Sent on Sun, 29 Sep 2002 11:58:29 -0700
> O_NONBLOCK.

Arseny said that FIONBIO will break other things.
What about O_NONBLOCK?

> This is an fcntl, not a socket option, btw.

I know.

> I have been working through stream.d and unixaux.d today.  I have not
> yet found code anywhere that handles partial reads or writes in either
> buffered or unbuffered streams.  Where should I be looking?

unixaux.d has read_helper() which serves full_read and safe_read.

IIUC, the net result is that when read() does not real all the data
requested, an error is raised (this is with buffered streams).

I can do the same thing for write.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Daddy, what does "format disk c: complete" mean?