Subject: [clisp-list] CLISP Apache module

From: "John K. Hinsdale" <[email-masked]>
Date: Tue Dec 31 07:18:01 2002

I'm looking for an Apache module to embed a CLISP interpreter into my
Apache server, much the same way perl is embedded via mod_perl.  I
want this so as to speed up my lisp-based dynamic database driven Web
server.  Does such a thing exist?  If not I can write one.  Are there
substantive examples of embedding a CLISP interpreter in a "C" program
and calling Lisp code from the "C" program?

---
John Hinsdale, Alma Mater Software, Inc., Tarrytown, NY 10591-3710 USA
[email-masked] | http://www.alma.com/staff/hin | +1 914 631 4690

---

From: "Johannes Grødem" <[email-masked]>
Date: Tue Dec 31 08:03:02 2002

* "John K. Hinsdale" <[email-masked]>:

> Apache server, much the same way perl is embedded via mod_perl.

I know this isn't quite what you're asking for, but there is a plugin
called mod_lisp that lets Apache communicate with a running Lisp (or
whatever, really), thus avoiding the need to fork a new clisp for
every request, as you would with CGI-scripts.  (Oh, and also with the
added bonus that you can keep state in the running Lisp image.)

I've done some simple stuff with it, it's quite fast.  2-300 requests
per second on modest hardware.

See the mod_lisp-link on
http://www.fractalconcept.com/

-- 
Johannes Grødem <OpenPGP: 5055654C>

---

From: "John K. Hinsdale" <[email-masked]>
Date: Tue Dec 31 08:09:04 2002

> whatever, really),

Yes, that is what I want, but I was more looking for an interpreter
actually embedded in the Apache server itself (ala mod_perl) ...
fewer moving parts.  For example, when I bounce the Apache server, the
Lisp stuff gets reloaded as well.  w/ a separate pool of processes for
the Lisp servers I have to deal w/ all that as well.

 > thus avoiding the need to fork a new clisp for

right, I want to re-use my database connections (and avoid recreating
lisp garbage etc.)

 > I've done some simple stuff with it, it's quite fast.  2-300 requests

OK I'll give this a try and if it doesn't work for me, maybe I will
write a truly embedded CLISP apache module

---

---

From: Edi Weitz <[email-masked]>
Date: Tue Dec 31 09:04:02 2002

"John K. Hinsdale" <[email-masked]> writes:

>  > I know this isn't quite what you're asking for, but there is a
>  > plugin called mod_lisp that lets Apache communicate with a
>  > running Lisp (or whatever, really),
> 
> fewer moving parts.  For example, when I bounce the Apache server,
> the Lisp stuff gets reloaded as well.  w/ a separate pool of
> processes for the Lisp servers I have to deal w/ all that as well.

With mod_lisp you only have one Lisp image that you don't have to shut
down an restart when you bounce Apache. You can have it run as long as
you wish and (if it's on a remote server) re-connect to it with
something like Dan Barlow's detachtty.

>  > thus avoiding the need to fork a new clisp for every request, as
>  > you would with CGI-scripts.  (Oh, and also with the added bonus
>  > that you can keep state in the running Lisp image.)
> 
> right, I want to re-use my database connections (and avoid
> recreating lisp garbage etc.)

You should really try Marc's mod_lisp. I've just rewritten an
application that was originally done in PHP with CMUCL and mod_lisp
and I'm extremely happy with the result. The old app had to read
thousands of database rows for _each_ page and compute some values out
of them. Granted, PHP has persistent database connections and some
session handling but that didn't help. The fact was that only maybe
one out of 100 page views really altered the database content but you
nevertheless had to re-compute each time.

With mod_lisp the whole database is read once on startup and then kept
in memory (in special variables). If a page alters the database
content the new information is sent back to the database but that's
about all with respect to database connections.

I'm an old mod_perl hacker but there are things you simply can't do
with interpreters embedded in Apache - I'll use mod_lisp (or a
complete Lisp web server like portableaserve) for all new projects.

(Not to mention that you get a _real_ programming language,
incremental development, a compiler, yada yada yada. You already know
that.)

> write a truly embedded CLISP apache module

I think that "truly embedding" a Common Lisp into a C program is not a
simple task. Maybe this will be easier with ECL than with CLISP or
CMUCL.

Cheers,
Edi.

---

From: "John K. Hinsdale" <[email-masked]>
Date: Tue Dec 31 10:59:03 2002

> and I'm extremely happy with the result.

Alrighty then!  I'll let you know how it works out for me.

Best wishes to all for peace, prosperity and happy Lisp hacking in the
coming new year.

---

---

From: "John K. Hinsdale" <[email-masked]>
Date: Tue Dec 31 12:18:17 2002

> down an restart when you bounce Apache. You can have it run as long as

Wondering: how are concurrent requests handled w/ this setup then?  I
don't see any code anywhere that forks off a separate concurrent
thread of activity (e.g., a new Unix process) while the "Lisp
application server" goes back to listening for the next request.
Rather it appears that all the work for any individual Web request
must finish before it will accept another connection.  In other words,
only one Lisp request is being processed at a time.

Am I missing something?  What I need is to pool, say, 10 live database
connections to handle peak activity.  W/ a mod_perl type setup those
connections would exists as perl state in each of the ten Apache
processes that had the perl language embedded in them.

If this is not an appropriate thread for the CLISP list, let me know
and I'll take it up w/ the mod_lisp devleoper(s) - but it doesn't have
a mailing list.

---

---

From: Edi Weitz <[email-masked]>
Date: Tue Dec 31 12:38:02 2002

"John K. Hinsdale" <[email-masked]> writes:

>  > With mod_lisp you only have one Lisp image that you don't have to
>  > shut down an restart when you bounce Apache. You can have it run
>  > as long as
> 
> Wondering: how are concurrent requests handled w/ this setup then?
> I don't see any code anywhere that forks off a separate concurrent
> must finish before it will accept another connection.  In other
> words, only one Lisp request is being processed at a time.

With CMUCL, the multi-threading code is in this function (taken from
the example on the mod_lisp webpage) - note the 'MP' functions. I
haven't used mod_lisp with CLISP yet so I don't no any details for
this implementation.

  (defun make-apache-listener (port)
    (let ((socket (ext:create-inet-listener port)))
      (unwind-protect
           (loop
            (mp:process-wait-until-fd-usable socket :input)
            (multiple-value-bind (new-fd remote-host)
                (ext:accept-tcp-connection socket)
              (let ((stream (sys:make-fd-stream new-fd :input t :output t)))
                (mp:make-process #'(lambda () (apache-listen stream))))))
        (unix:unix-close socket))))

Here are some benchmark results done on my laptop - also with the
mod_lisp example code and CMUCL. This is admittedly a very simple,
static example but you'll see that mod_lisp handles 500 request with a
concurrency level of 10 without any problems - in less than 0.3
seconds.

  edi@bird:~ > /usr/local/apache/bin/ab -n 500 -c 10 'http://localhost/lisp'
  This is ApacheBench, Version 1.3d <$Revision: 1.65 $> apache-1.3
  Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
  Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

  Benchmarking localhost (be patient)
  Completed 100 requests
  Completed 200 requests
  Completed 300 requests
  Completed 400 requests
  Finished 500 requests
  Server Software:        Apache/1.3.26
  Server Hostname:        localhost
  Server Port:            80

  Document Path:          /lisp
  Document Length:        196 bytes

  Concurrency Level:      10
  Time taken for tests:   0.294 seconds
  Complete requests:      500
  Failed requests:        0
  Broken pipe errors:     0
  Total transferred:      206412 bytes
  HTML transferred:       98196 bytes
  Requests per second:    1700.68 [#/sec] (mean)
  Time per request:       5.88 [ms] (mean)
  Time per request:       0.59 [ms] (mean, across all concurrent requests)
  Transfer rate:          702.08 [Kbytes/sec] received

  Connnection Times (ms)
                min  mean[+/-sd] median   max
  Connect:        0     0    0.2      0     3
  Processing:     2     5    2.9      5    16
  Waiting:        1     4    2.9      5    16
  Total:          2     5    2.9      5    16

  Percentage of the requests served within a certain time (ms)
    50%      5
    66%      6
    75%      6
    80%      6
    90%      9
    95%     13
    98%     14
    99%     16
   100%     16 (last request)

> Am I missing something?  What I need is to pool, say, 10 live
> database connections to handle peak activity.  W/ a mod_perl type
> setup those connections would exists as perl state in each of the
> ten Apache processes that had the perl language embedded in them.

The db connection pool should be no big deal although you'll probably
have to implement it yourself. I think the main problem is that many
(most?) CL implementations will block during a db call (because it's
FFI stuff). [Please correct me if I'm wrong.]

This can't be resolved with threading and forking isn't very Lisp-y...

> and I'll take it up w/ the mod_lisp devleoper(s) - but it doesn't
> have a mailing list.

Marc is usually very helpful - you can contact him directly. There's
also the lispweb mailing list.

Cheers,
Edi.

---

From: "John K. Hinsdale" <[email-masked]>
Date: Tue Dec 31 13:01:07 2002

> the example on the mod_lisp webpage) - note the 'MP' functions. I

yah, but I want to use CLISP :)

Also, even in CMUCL it looks like it creates a new process/thread (and
presumably has to init a new DB connection) w/ each request - but
that's the overhead I'm trying to avoid.  What I need is a pool of
processes each already "primed" w/ an active DB connection and
possibly some cached data.  If I embed my language interpreter into
Apache, I get all this for "free" via the process pooling that's
already being done by Apache.  This is what I've been enjoying w/
mod_perl for some years.

It's not obvious to me yet that it won't be easier just to embed CLISP
into Apache w/ a module of its own (e.g., "mod_clisp")

 > I think the main problem is that many (most?) CL implementations will
 > block during a db call (because it's FFI stuff). [Please correct me if
 > I'm wrong.]

yes, they'll block as they'll have to wait on network I/O as w/ any
type of client-side software.  Some of my Web request will hit an
Oracle server w/ a sequence of queries that can take 30 secs or more
so the concurrency is a big issue for me.

 > also the lispweb mailing list.

i'll run my situation by marc and let you know how it works out.

---

From: Edi Weitz <[email-masked]>
Date: Tue Dec 31 17:17:01 2002

"John K. Hinsdale" <[email-masked]> writes:

>  > With CMUCL, the multi-threading code is in this function (taken
>  > from the example on the mod_lisp webpage) - note the 'MP'
>  > functions. I
> 

OK, that's your choice which I won't comment on - especially not on
this list... :)

> Also, even in CMUCL it looks like it creates a new process/thread
> (and presumably has to init a new DB connection) w/ each request -
> but that's the overhead I'm trying to avoid.

No, it doesn't. With mod_lisp's "Keep-socket" feature your thread will
stay alive and keep serving more requests until you willfully shut it
down - pretty similar to mod_perl.

> What I need is a pool of processes each already "primed" w/ an
> active DB connection and possibly some cached data.  If I embed my
> language interpreter into Apache, I get all this for "free" via the
> process pooling that's already being done by Apache.  This is what
> I've been enjoying w/ mod_perl for some years.

As I said, I think this can be done with mod_lisp with some manual
work except for the fact that you have 'threads' which are maintained
by your CL implementation as opposed to processes maintained by your
OS as in mod_perl. (As I detailed in previous mails this also has a
lot of advantages.)

> It's not obvious to me yet that it won't be easier just to embed
> CLISP into Apache w/ a module of its own (e.g., "mod_clisp")

Yes, if that's feasible...

>  > I think the main problem is that many (most?) CL implementations
>  > will block during a db call (because it's FFI stuff). [Please
>  > correct me if I'm wrong.]
> 
>  > Marc is usually very helpful - you can contact him
>  > directly. There's also the lispweb mailing list.
> 

OK, good luck.

[Please note that this and subsequent messages might be spoiled by too
much Champagne so don't take them too serious...]

Cheers,
Edi.

---

From: "Johannes Grødem" <[email-masked]>
Date: Tue Dec 31 19:17:01 2002

[...]
> write a truly embedded CLISP apache module

What's nice about mod_lisp is that you can use any other Lisp, in case
CLISP doesn't do what you want.  (Or an entirely different language,
for that matter.)  Oh, and check out CLISP's #'socket:socket-status if
you haven't already.

(If someone has already mentioned this, I missed it.)

---

From: "John K. Hinsdale" <[email-masked]>
Date: Wed Jan 1 08:36:13 2003

> No, it doesn't. With mod_lisp's "Keep-socket" feature your thread will

Edi - my mistake; you are absolutely right.  I was confused as to
where the pooling was happening (it happens in Apache which has a
mod_lisp "client" in each of its processes, each of which has a
kept-alive connect to a thread inside the Lisp server).

Great.

So, now my next question:

Has anyone done any kind of multi-threaded TCP server in CLISP?  i.e.,
the traditional listen/accept/fork loop kind of thing?  Then I can
just "port" the CMUCL example and be on my way.  I'm also waiting to
hear back from Marc Battyani.

I looked up "CL-HTTPD" but CLISP is not listed as one of the CL
implementations it is known to run on.

Thanks for all the help so far.

---

From: Edi Weitz <[email-masked]>
Date: Wed Jan 1 09:08:07 2003

"John K. Hinsdale" <[email-masked]> writes:

>    >> Also, even in CMUCL it looks like it creates a new
>    >> process/thread (and presumably has to init a new DB
>    >> connection) w/ each request -
> 
> Has anyone done any kind of multi-threaded TCP server in CLISP?
> i.e., the traditional listen/accept/fork loop kind of thing?  Then I
> can just "port" the CMUCL example and be on my way.  I'm also
> waiting to hear back from Marc Battyani.

The mod_lisp website has example code for CLISP by Rachel Richard and
Nils Kassube. Isn't that what you want?

  <http://www.fractalconcept.com/fcweb/download/modlisp-clisp.lisp.gz>

Edi.

---

From: "John K. Hinsdale" <[email-masked]>
Date: Wed Jan 1 09:10:13 2003

> Nils Kassube. Isn't that what you want?

no, if you look at that code, it ignores the case where you want
concurrency.

---

From: Edi Weitz <[email-masked]>
Date: Wed Jan 1 10:55:16 2003

"John K. Hinsdale" <[email-masked]> writes:

>  > The mod_lisp website has example code for CLISP by Rachel Richard
>  > and Nils Kassube. Isn't that what you want?
> 

I see. Sorry, I didn't look at the code, I'm not a CLISP expert.

Cheers,
Edi.

---

From: Ng Pheng Siong <[email-masked]>
Date: Wed Jan 1 17:59:01 2003

> no, if you look at that code, it ignores the case where you want

Well, look at more than the CLISP example:

mod-lisp.lisp (LispWorks, it seems):

(defun make-apache-instream (handle)
  (mp:process-run-function (format nil "apache socket ~d" handle) '()
                            'apache-listen
                            (make-instance 'comm:socket-stream :socket handle
					  :direction :io
                                          :element-type 'base-char)))

modlisp-cmucl.lisp:

        [...]

modlisp-clisp.lisp:

    A bit more stuff to quote, so I won't. Invokes 'apache-listen
    eventually.

And, finally, clisp-2.30/src/threads.lisp:

;; multithreading for CLISP

(defpackage "THREADS"
  (:nicknames "MT")
  (:use "COMMON-LISP" "EXT")
  (:export "MAKE-THREAD" "THREAD-WAIT" "THREAD-WAIT-WITH-TIMEOUT"
           "WITHOUT-INTERRUPTS" "THREAD-YIELD" "THREAD-KILL"
           "THREAD-INTERRUPT" "THREAD-RESTART" "THREADP" "THREAD-NAME"
           "THREAD-ACTIVE-P" "THREAD-STATE" "CURRENT-THREAD" "LIST-THREADS"
           "MAKE-LOCK" "THREAD-LOCK" "THREAD-UNLOCK" "WITH-LOCK"
           "Y-OR-N-P-TIMEOUT" "WITH-TIMEOUT"))

So perhaps you just need to do something like:

    blah blah 
    (mt:make-thread blah-blah 'apache-listen blah-blah) 
    blah blah

Let us know how it goes. Cheers.

-- 
Ng Pheng Siong <[email-masked]> * http://www.netmemetic.com

---

From: "John K. Hinsdale" <[email-masked]>
Date: Thu Jan 2 07:22:19 2003

> ... And, finally, clisp-2.30/src/threads.lisp:

thanks, this looks like what I would want ...

---

From: Ng Pheng Siong <[email-masked]>
Date: Thu Jan 2 08:47:13 2003

> thanks, this looks like what I would want ...

I just tried building 2.30 with Posix threading, and make craps out:

ngps@applause:/usr/local/src/lisp/clisp-2.30/src$ make
test -d bindings || mkdir -p bindings
./lisp.run -B . -N locale -Efile UTF-8 -Eterminal UTF-8 -norc -m 750KW -x "(load \"init.lisp\") (sys::%saveinitmem) (exit)"

[ snipped banner ]

;; Loading file defseq.lisp ...
;; Loaded file defseq.lisp
;; Loading file backquote.lisp ...
;; Loaded file backquote.lisp
;; Loading file defmacro.lisp ...
;; Loaded file defmacro.lisp
;; Loading file macros1.lisp ...
;; Loaded file macros1.lisp
;; Loading file macros2.lisp ...
;; Loaded file macros2.lisp
;; Loading file defs1.lisp ...
;; Loaded file defs1.lisp
;; Loading file places.lisp ...
*** - Program stack overflow. RESET

Without threading, "make check" passes.

Googling around, it looks like multithreading in CLISP is unfinished
business.

---

From: "John K. Hinsdale" <[email-masked]>
Date: Thu Jan 2 10:50:21 2003

> Googling around, it looks like multithreading in CLISP is unfinished

i did some more thought, and it turns out my Oracle library is not
thread safe (uses global conneciton cache), also w/ the mod_lisp
approach I can't use it as a drop-in replacement for my scripts that
write the response to the STDOUT stream as per CGI.  W/ CLISp inside
apache, each request has its own copy of CLISp (which is a downside,
too - it eats a heap of memory per process) But for those reasons I
may just go ahead and try the Apache module.

---

From: Edi Weitz <[email-masked]>
Date: Thu Jan 2 11:34:30 2003

"John K. Hinsdale" <[email-masked]> writes:

> also w/ the mod_lisp approach I can't use it as a drop-in
> replacement for my scripts that write the response to the STDOUT
> stream as per CGI.

  (let ((html (with-output-to-string (*standard-output*)
                (my-old-script))))
    (send-data-to-mod-lisp html))

Wouldn't that work?

Edi.

---

From: "John K. Hinsdale" <[email-masked]>
Date: Thu Jan 2 13:03:17 2003

> Wouldn't that work?

well, almost ... except that it's not always allowed in my case to
collect up the entire output before beginning to send back to the Web
client.  (I have some pages that need to begin producing displayed
output before the request is completely processed

---

From: Edi Weitz <[email-masked]>
Date: Thu Jan 2 15:00:02 2003

> output before the request is completely processed

Hmm, that also rules out using mod_lisp's "Keep-Socket" feature which
is one of its highlights. You _must_ send a correct Content-Length
header before you send the body - kind of like HTTP/1.1 Keep-alive.

Cheers,
Edi.

---

From: Edi Weitz <[email-masked]>
Date: Wed Jan 8 11:00:02 2003

Edi Weitz <[email-masked]> writes:

> Hmm, that also rules out using mod_lisp's "Keep-Socket" feature
> which is one of its highlights. You _must_ send a correct
> Content-Length header before you send the body - kind of like
> HTTP/1.1 Keep-alive.

For the sake of completeness let me add that in 2.31 beta Marc has
re-introduced the feature that you can "stream" your content to
mod_lisp without the need of a correct Content-Length header. As I
said, you can't re-use the Lisp/Apache socket this way (that'll need
something more sophisticated as outlined by Jörg) but at least it is
possible again. It wasn't in the latest released version (2.2).

Cheers.
Edi.