Subject: select&threads (Was: running CLISP as an Internet daemon?)

From: "J`o'rg-Cyril H`o'hle" <[email-masked]>
Date: Thu, 07 Oct 1999 17:25:17 +0200

Hi,

Russell McManus <[email-masked]> wrote in September:
>Does anyone know how to run CLISP as an internet daemon, using either
>select or threads to handle multiple file descriptors?

Short answer (not from me, but Bruno Haible and Guido van Rossum of
Python fame when I asked him something related): why go for select()
hacks when the real answer is threads (not yet available in CLISP...)?

My answer to the answer: threads impose a granularity on programs that
may not be adequate for the problem at hand.

Yet many programs, algorithms and computational model see fit in the
single source-sink, producer-consumer or client-server model,
therefore threads are nice to handle multiple such relations in a
program.

If you still think you want to wait for several sources at the same
time (I'm fully with you here), I'll tell you that for several years,
I've failed to build a reasonable OS-independant (i.e. portable)
abstraction of the unique knot waiting for all possible sources for
CLISP (e.g. files, ARexx messages, etc.).

You'll soon feel the so-called "curse of the X event loop" or
experience the unability to merge several components into one
application because those all want to provide their own main loop or
waiting function and don't allow for others to get in.

select(3) is an abstraction adapted to UNIX, since UNIX maps almost
everything to files (/dev/tty, pipes, disk files, sockets). "Almost"
because select() also integrates signals, the other source: the call
will return EINTR and the application will know how to handle the
signals.  So select() integrates and can query all sources in one.

On AmigaOS, the unique knot is just Wait()'ing for signals (everything
happens via messages and signals, window events generate messages and
signal this, and so on), so it's conceptually closer to pause(2).

I don't know what would be used on MS-*.

Sadly, most packages I've seen ignore this unicity and provide
interfaces like (KAPI-GET-OR-WAIT one-channel), (SOCKET-WAIT
one-socket-server), GTk main, X main, xyz main loop.  These just don't
interoperate.  Even worse, few of these are thread safe or reentrant
so you're doomed if you wanted to do threading.

Using one interface precludes using the other.  Polling every 0.01s,
as recently suggested by Bruno Haible, is not a solution but probably
the best you can do given the current APIs (not only CLISP's).

The API design problem might be solved by writing all applications
like in a continuation passing style (CPS).  I also feel I'm
approaching what's been called stack groups and coroutines here.
Gilbert Baumann once had a CLISP with coroutines.  There's a nice
paper somewhere about elegantly escaping the X event loop and dispatch
problem by using continuations (see also Haskell's "fudgets",
functional widgets).  However, CSP would be best if it were
transparent.

Providing an *anchor* for merging into the knot/dispatcher instead of
providing a function that only considers one source is maybe what
every API should do.  E.g. on UNIX, which are the file descriptors or
signals that my component will react to so I can tell the dispatcher
provided by the application programmer/merger?  But how to stack such
knots when combining SW components?

Maybe I failed to find a solution because I should really prove that
select() is both a least upper and a most upper bound of the required
functionality, namely integrating all sources into a single knot from
which can then be dispatched.  If that should be the case, it seems
clear that other models like AmigaOS' cannot be unified.

Regards,
	Jorg Hohle.

---

From: Russell McManus <[email-masked]>
Date: 07 Oct 1999 14:02:47 -0400

> may not be adequate for the problem at hand.

Thanks for investing the time to compose such a well considered
response.  I agree that the threads model can be overused.  Of course
in some cases it is the most elegant way to approach a problem.

  <horrible description of portability problems snipped>

I agree that there is probably no simple way to do this portably, btw.
One way to approach the problem is for each platform, to build a thin
layer on top of the multiplexing primitives that are accessible from
lisp, and post the availability of the interface to *features*.  Then
people can have a stab at writing a unifying interface on the lisp
level.

It would also allow people to write OS-specific code (which is
somewhat contrary to the CLISP philosopy).  So to confess my sins, I
guess I was asking for a "Worse is better" solution, and you just
provided a "Worse is _not_ better" answer!

-russ

--

We're too busy mopping the floor to turn off the faucet.

---

From: [email-masked] (Paolo Amoroso)
Date: Thu, 07 Oct 1999 20:09:13 GMT

> problem by using continuations (see also Haskell's "fudgets",

Can you provide a reference to the paper?

Paolo
-- 
EncyCMUCLopedia * Extensive collection of CMU Common Lisp documentation
http://cvs2.cons.org:8000/cmucl/doc/EncyCMUCLopedia/

---

From: Don Cohen <[email-masked]>
Date: Thu, 07 Oct 1999 21:55:15 -0700

Polling every 0.01s, as recently suggested by Bruno Haible, 
  is not a solution but probably the best you can do given the
  current APIs (not only CLISP's).

I would not want to discourage processes (or threads), which I've
long wished for in clisp, but I'd like to mention that I've used
the sleep/poll solution for some time and found it very satisfactory.
I think (sleep .01) is going a bit overboard - I wouldn't really 
expect a response in .01 sec. even in the case where we were doing
the normal wait for input on one stream.  (sleep 1) has been fine
for me, and at that rate I see negligible cpu usage when nothing is
happening.

---

From: Joerg Hoehle <[email-masked]>
Date: Fri, 8 Oct 1999 12:11:08 +0200 (MET DST)

> Can you provide a reference to the paper?

The paper:

Escaping the event loop: an alternative control structure for
multi-threaded GUIs, M. Fuchs,
http://cs.nyu.edu/phd_students/fuchs/index.html
http://cs.nyu.edu/phd_students/fuchs/gui.ps.gz

Fudgets: probably below www.Haskell.org/
coroutines: ask Gilbert Baumann

	Jo"rg Ho"hle.

---

From: Erik Arneson <[email-masked]>
Date: Fri, 8 Oct 1999 08:41:36 -0700 (PDT)

> for me, and at that rate I see negligible cpu usage when nothing is

I have a piece of software which is polling up to a large number of
sockets, some of which are expected to be doing high-throughput file
transmissions.  I tried using no sleep at all, which caused the CPU
usage to leap up to around 80% and sit there.  Finally I've been
experimenting with calling `system::%sleep' directly, so I could give it 
a wait time in microseconds.  This has helped the file transfer
throughput, although on 100mb/s networks it's still painfully slow.

In any case, it has kept CPU utilization down to 0% most of the time.

-- 
# Erik Arneson      [email-masked]    Webring Software Engineer #
# Yahoo! Inc.      PGP ID: 2048/84413E19        (541) 482-3000x114 #
# "There's such a fine line between stupid and clever." Spinal Tap #

---

From: Joerg Hoehle <[email-masked]>
Date: Tue, 12 Oct 1999 17:57:13 +0200 (MET DST)

On Fri, 8 Oct 1999, Erik Arneson wrote:

> experimenting with calling `system::%sleep' directly,

Why do you use %SLEEP when you could call SLEEP on a fraction or rational 
number?

I have the following in a server application:

;; proper format avoids consing in sleep (CLISP: .s0, CMU rational)
(defconstant *kapi-sleep-time* #+(or CLISP) .3s0 #-(or CLISP) 3/10)

Regards,
	Jo"rg Ho"hle.