Subject: [clisp-list] TCP-based Network Server in CLISP

From: "Dave Richards" <[email-masked]>
Date: Thu Sep 19 09:52:04 2002

I am trying to implement a TCP-based network server in CLISP.  This server
accepts requests, processes the request and generates what could potentially
be a large amount of data, then sends this response via the requesting
socket.  The socket remains open for subsequent requests from clients.

Thus, input processing is straightforward.  A socket-status can be performed
while awaiting a request.  By using a read-byte-no-hang I can read data
until  the entire request has been read or until the socket runs out
temporarily, i.e. until the next socket-status loop.

I am unsure how to address the write case.  I can (and currently do)
accumulate the output data into an unsigned-byte array and when
socket-status indicates the socket can be :outpt to, write the data.  The
question is, however, how much?  By examing the steam code I discovered that
when you write to a stream, CLISP assumes you'll write the requested amount.
Which means, if I write too much, I'll block waiting for the socket to
drain.  The problem with this, however, is that a maligicious client can
write a request to the socket (ensuring a large response) and then never
read the response.  This should successfully hang the server.

This could be alleviated if there was a write-byte-no-hang or a way to known
in advance, how much can be written without blocking.  Another approach
would be to implement "atomic" I/O functions, i.e. a version of
read-sequence or write-sequence which returned the requested amount or
actually transferred amount of data.

Have other people written network servers in CLISP?

	Thanks in advance!

	Dave

---

From: Sam Steingold <[email-masked]>
Date: Thu Sep 19 12:38:03 2002

> * In message <[email-masked]>
> * On the subject of "TCP-based Network Server in CLISP"
> * Sent on Thu, 19 Sep 2002 09:50:58 -0700
> * Honorable "Dave Richards" <[email-masked]> writes:
>
> This could be alleviated if there was a write-byte-no-hang

(defun write-byte-no-hang (b o)
  (case (socket-status o 0)
   ((:io :output) (write-byte b o) b)
   (t nil)))

> or a way to known in advance, how much can be written without
> blocking.

I don't think this is possible - OS does not tell that.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
OK, so you're a Ph.D.  Just don't touch anything.

---

From: "Dave Richards" <[email-masked]>
Date: Sat Sep 21 17:54:01 2002

I am enclosing a patch which adds socket-get-option and socket-set-option
subrs.  This patch was generated against 2.28.  In addition, a set of
keyword symbols were also added, :so-keepalive, :so-oobinline, etc.

To enable keepalives on a socket, for example, you can:

	(socket-set-option <socket> :so-keepalive t)

The current value would be obtained by:

	(socket-get-option <socket> :so-keepalive)

which returns two values.  The first is the option value.  The second
indicates whether the first value is "valid", i.e. did the underlying
getsockopt() routine actually return a value?

socket-get/set-option works on both server and stream sockets, although
server sockets are normally used for inheritance purposes only.

I thought by implementing these functions, more specifically support for the
:so-sndlowat option, I would be able to solve the problem I addressed to
this forum.  The send low-water mark is the minimum amount of free space
that must be available on the send socket buffer before select() will return
true for write.  It defaults to 1.

Unfortunately, Linux 2.2 only provides a read-only interface (although 2.4
provides a read/write interface) and Windows 2000 provides a read-only
interface which lies.  Thus, this patch, although it may be useful, doesn't
solve my problem.

	Dave

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Sam
Steingold
Sent: Thursday, September 19, 2002 12:38 PM
To: [email-masked]
Subject: [clisp-list] Re: TCP-based Network Server in CLISP

-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
clisp-list mailing list
[email-masked]
https://lists.sourceforge.net/lists/listinfo/clisp-list

---

From: Sam Steingold <[email-masked]>
Date: Mon Sep 23 15:52:05 2002

> * In message <[email-masked]>
> * On the subject of "RE: [clisp-list] Re: TCP-based Network Server in CLISP"
> * Sent on Sat, 21 Sep 2002 17:53:31 -0700
> I am enclosing a patch which adds socket-get-option and
> socket-set-option subrs.  This patch was generated against 2.28.

could you please work off the CVS head?

> getsockopt() routine actually return a value?

this is not "the Lisp way".
could you please change your patch so that...

1. you should export just one symbol SOCKET-OPTION and use
        (setf (socket-option socket :so-keepalive) t)
   instead of the separate setter function

2. you should write a ChangeLog entry

3. when setsockopt or getsockopt fails, you should signal an error
   instead of returning a second value.

4. you should export the symbol SOCKET-OPTION in init.lisp

5. do not wrap the whole function body in
   begin_system_call/end_system_call.
   
otherwise the patch looks good.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Let us remember that ours is a nation of lawyers and order.

---

From: Sam Steingold <[email-masked]>
Date: Thu Sep 26 21:01:02 2002

I added SOCKET:SOCKET-OPTIONS to the CVS head.
please do "cvs up" and try it.
<http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/clisp/clisp/doc/impnotes.html?rev=HEAD&content-type=text/html#so-opt>

> Unfortunately, Linux 2.2 only provides a read-only interface (although
> 2.4 provides a read/write interface)

well, I would think that few people use 2.2 now (and even fewer will be
using it when CLISP with SOCKET-OPTIONS will be released).

> Windows 2000 provides a read-only interface which lies.

yuk!
maybe Arseny can suggest something here?

> Thus, this patch, although it may be useful, doesn't solve my problem.

maybe you will invent another solution?

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Time would have been the best Teacher, if it did not kill all its students.

---

From: Arseny Slobodjuck <[email-masked]>
Date: Fri Sep 27 05:14:01 2002

Hi,

> maybe Arseny can suggest something here?
Well, SO_RCVLOWAT and SO_SNDLOWAT are nonsupported in winsock. One
thing I can suggest to solve a problem is to make socket nonblocking.
But it is only first step. One can use SOCKET-STATUS aka select()
to determine if he/she may send the data to it. But it doesn't
guarantees send() will succeed and all the more lisp output operator
will succeed. Suppose I wrote (format sockstream "~D ~D" 111 222) and
it result in two send()s. A half of resulted string, say, "111 " will
succesfully being sent, but "222" will stuck. Nonblocking socket returns
WSAENOBUFS from send(). We won't wait and raise an exception in lisp.
How to recover on this error at lisp level ?

Another way is to use SO_SNDBUF to set buffer for sends, but I'm not
sure about what it means. If it guarantees that when select() allows
to send data and data is not longer than sndbuf then send succeeds ?
We then can get a buffer to hold all the data or check if
data will fit in buffer. Again, we should know data size generated by
lisp statement before actual send. What to do if OS doesn't allow big
buffer sizes and the data is big ? Doubtful way I think.

-- 
Best regards,
 Arseny

---

From: "Dave Richards" <[email-masked]>
Date: Fri Sep 27 07:06:17 2002

> thing I can suggest to solve a problem is to make socket nonblocking.

I agree with your subsequent analysis.  The CLISP stream logic pre-supposes
that partial read()s/write()s are errors, and thus, non-blocking is only the
beginning of a solution.

> sure about what it means.

SO_SNDBUF really isn't helpful in this context.  It may be useful from a
performance perspective, but since CLISP has no way of knowing the current
windowing state, knowing the send buffer size isn't helpful.

In previous lives I've implemented server socket abstractions for
non-blocking/single-threaded environments.  It's not conceptually difficult
to do, but it's going to have to be done in C.  I'm not going to attempt it
until I have a better feel for the CLISP/C environment.

	Dave