Subject: [clisp-list] clisp, mod_lisp and lockups

From: Adam Warner <[email-masked]>
Date: Sat Jul 13 01:30:06 2002

Hi all,

Marc Battyani suggested I ask the clisp mailing list to see if anyone
else has isolated lockups that they have been having with mod_lisp. I'm
the third person that has experienced lockups with clisp and mod_lisp. I
also concur that they do not happen with cmucl:

http://groups.google.co.nz/groups?hl=en&lr=&ie=UTF-8&threadm=6b4aa54a.0206060137.3739b9a4%40posting.google.com&rnum=1&prev=/groups%3Fselm%3D6b4aa54a.0206060137.3739b9a4%2540posting.google.com

mod_lisp is a little Apache module that allows you to serve content
using clisp and still leave Apache to serve other content reliably and
at high speed (after all it's Apache).

I've found that the lockups can be avoided by creating a new socket each
time. But this induces very strange behaviour where the page is served
to Mozilla every second time (zero content is returned every other
time). This is where I'm up to on the Debian bug database:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=152742

The bug is either in clisp (unlikely), mod_lisp.c or modlisp-clisp.lisp.
The content of
http://www.fractalconcept.com/fcweb/download/modlisp-clisp.lisp.gz
is listed below. Since the wrong MIME type is set on the web site it's
not easy to view without downloading and gunzipping it.

Thanks for any help. mod_lisp+clisp is just not popular enough for the
clisp problem to have been found and fixed by someone who understands
interfacing with Apache.

If you want to try mod_lisp+clisp out and are stuck then post your
question here and I should be able to answer it. The meta install
process is:

(1) make sure you have the mod_lisp.so module installed. If you have the
(Debian) apache-dev tools compiling the modules is as easy as typing
"apxs -c mod_lisp.c" and placing mod_lisp.so in /usr/lib/apache/1.3/

(2) Add a LoadModule line into httpd.conf under the current LoadModule
lines:

LoadModule lisp_module /usr/lib/apache/1.3/mod_lisp.so

(3) Make sure at least some content is going to be served by the module.
For example add this to the end of httpd.conf:

<IfModule mod_lisp.c>
  LispServer  127.0.0.1 3000 "test"
    <Location /modlisp>
      SetHandler lisp-handler
    </Location>
</IfModule>

(4) Restart apache (e.g. "/etc/init.d/apache restart")

(5) load modlisp-clisp.lisp into clisp:

(load "modlisp-clisp.lisp")

(6) and run the server:

(modlisp:modlisp-server)

If you try and access http://<server root>/modlisp/fixed you should then
see a table of key values. If you repeat long enough you might get a
lockup. wget seems very good at inducing lockups/timeouts.

Type CTRL-C to break the running clisp program.

modlisp/fixed is the default path to serve the key values (refer
modlisp-clisp.lisp code below)

Regards,
Adam

;;; -*- Mode:Lisp; Syntax:Common-lisp; Package:MODLISP; Base:10 -*-

;; modlisp-clisp.lisp
;; adapted from modlisp-acl.lisp

;; Original CLISP adaption by Rachel Richard <[email-masked]>
;; Changes by Nils Kassube <[email-masked]> on 2001-11-24:
;;
;; - replaced :lisp by :ext for CLISP's new naming convention,
;;   works now with recent CLISP distributions
;; - added modlisp-server to the :export line 

(in-package :user)

(provide :modlisp)

(eval-when (compile load eval)
  (defpackage :modlisp
    (:nicknames :lph)
    (:use :cl)
    (:export modlisp-server)
    ))

(in-package :modlisp)

;;;;;
;;;
;;; First part adapted from server.cl
;;;
;;;;;

(defvar *server-count* 0)	; used to name servers

(defun make-socket-server (&key (name (format nil "socket server ~d "
					      (incf *server-count*)))
				port 
				function 
				wait 
				(format :text))
  ;;
  ;; create a server process with the given name, listening on the
  ;; given port, running the given function on each connection that
  ;; comes in, and possibly waiting for that function's completion before
  ;; accepting a new connection.
  ;;
  ;; name - a string naming the server process -- if nil, then this 
  ;;	function will create a name.
  ;; port - if nil then an internet domain port number will be chosen
  ;;	    by the operating system.   If a number is given then that
  ;;	    port will be used (or an error will be signalled if it
  ;;	    is already in use).    If port is a string then a unix
  ;;	    domain port will be used.  (this will not work on Windows).
  ;; function - the function to run when a connection is made.  This
  ;;	    function must take one argument which is the stream used
  ;;	    used for reading from and writing to the process that connected
  ;;	    to this socket. 
  ;; wait - if true, then the function will be run in the server process
  ;;	    and thus the server won't accept a new connection until
  ;;	    the function finishes.
  ;; format  - :text (the default) or :binary.   This determes what kind
  ;;	    of data can sent to and read from the socket stream.
  ;;	    
  ;;
  ;;
  ;; The return value is the port number on which the server is
  ;;	listening.
  ;;
  (let ((passive-socket (ext:socket-server port) ;

))
	
	(start-socket-server passive-socket
			     :function function
			     :wait wait)
(ext:socket-server-port passive-socket)))

(defun start-socket-server (passive-socket &key function wait)
  ;; internal function run in the server lightweight process 
  ;; that continually processes the connection.
  ;; This code is careful to ensure that the sockets are 
  ;; properly closed something abnormal happens.
  (unwind-protect
      (loop (let ((connection  (ext:socket-accept passive-socket)))
	      (if wait
		 (progn 
		   (unwind-protect
			  (funcall function connection)
			  (handler-case (values-list (cons t
							   (multiple-value-list
							    (close connection))))
					(error (condition) (declare (ignore-if-unused condition))
					       nil))))

			   (unwind-protect
			       (funcall function connection)
			    	  (handler-case (values-list (cons t
							   (multiple-value-list
							    (close connection))))
					(error (condition) (declare (ignore-if-unused condition))
					       nil)))

		 )))
   
    	  (handler-case (values-list (cons t
							   (multiple-value-list
							    (close passive-socket))))
					(error (condition) (declare (ignore-if-unused condition))
					       nil))))

;;;;;
;;;
;;; Second part adapted from mod-lisp.lisp
;;;
;;;;;

(defconstant +apache-port+ 3000)
(defvar *apache-socket* nil) ;the socket to apache
(defvar *close-apache-socket* nil) ;set to t if you want to close the socket to apache
(defvar *apache-nb-use-socket* 0) ;the number of requests sent in this socket

(defun modlisp-server (&optional (port +apache-port+))
  
  (make-socket-server :name "test"
		      :port port
		      :function 'apache-listen))

(defun apache-listen (*apache-socket*)
   (let ((*close-apache-socket* t))
    (unwind-protect
	(loop for *apache-nb-use-socket* from 0
	    for command = (get-apache-command)
	    while command 
	    do (process-apache-command command)
	  (force-output *apache-socket*)
	    until *close-apache-socket*)
      (close *apache-socket*))))

(defun get-apache-command ()
  (ignore-errors
   (let* ((header (loop for key = (read-line *apache-socket* nil nil)
		    while (and key (string-not-equal key "end")) 	
		    collect (cons key (read-line *apache-socket* nil nil)) )))
     (let* ((content-length (cdr (assoc "content-length" header :test #'equal)))
	  (content (when content-length (make-string (parse-integer content-length :junk-allowed t)))))

     (when content
       (read-sequence content *apache-socket*)
       (push (cons "posted-content" content) header))
     header))))
				
(defun process-apache-command (command)
  (let ((html (if (equal (cdr (assoc "url" command :test #'string=)) 
			 "/modlisp/fixed")
		  (debug-table command)
		(fixed-html))))
    (write-header-line "Status" "200 OK")
    (write-header-line "Content-Type" "text/html")
    (write-header-line "Content-Length" (format nil "~d" (length html)))
    (write-header-line "Keep-Socket" "1")
    (write-string "end" *apache-socket*)
    (write-char #\NewLine *apache-socket*)
    (write-string html *apache-socket*)
    (setf *close-apache-socket* nil)))

(defun debug-table (command)
  (with-output-to-string (s)
   (write-string "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML//EN\">
<HTML><HEAD></HEAD><BODY><TABLE bgcolor=\"#c0c0c0\">
<TR bgcolor=\"yellow\"><TH COLSPAN=2>CLISP + mod_lisp 2.0 + apache + Linux</TH></TR>
<TR bgcolor=\"yellow\"><TH>Key</TH><TH>Value</TH></TR>" s)
   (format s "<TR bgcolor=\"#F0F0c0\"><TD>apache-nb-use-socket</TD><TD>~a</TD></TR>"  *apache-nb-use-socket*)
   (loop for (key . value) in command do
	 (format s "<TR bgcolor=\"#F0F0c0\"><TD>~a</TD><TD>~a</TD></TR>" key value))
   (write-string "</TABLE></BODY></HTML>" s)))

(defun fixed-html ()
  "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML//EN\">
<HTML><HEAD></HEAD><BODY><H1>mod_lisp 2.0</H1><P>This is a constant
  html string sent by mod_lisp 2.0 + CLISP + apache + Linux</P>
</BODY></HTML>")

(defun write-header-line (key value)
  (write-string key *apache-socket*)
  (write-char #\NewLine *apache-socket*)
  (write-string value *apache-socket*)
  (write-char #\NewLine *apache-socket*))

---

From: John Jorgensen <[email-masked]>
Date: Sat Jul 13 06:56:03 2002

I've been seeing similar problems here. The fast way to make mod_lisp
"lock up" is to access a page under /modlisp that doesn't exist, for
instance; http://localhost/modlisp/foobar . At least this does it for
me. Once I access /modlisp/foobar, I can't access /modlisp/fixed .

I may adapt bits of modlisp-clisp.lisp to the socket server in clocc and
see if I get the same problems.

Like you, I'm not convinced that this is a clisp bug but I haven't done
much research into it yet.

J*
----------------------------------------------

On 13 Jul 2002, Adam Warner wrote:

> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> clisp-list mailing list
> [email-masked]
> https://lists.sourceforge.net/lists/listinfo/clisp-list
>

---

From: Adam Warner <[email-masked]>
Date: Sat Jul 13 07:30:01 2002

> much research into it yet.

BTW just another idea: mod_lisp can be used as a drop in replacement for
(libapache-)mod-webapp from http://alpha.onshored.com/lisp-software/.
Thus it appears mod-webapp provides similar functionality to mod_lisp
and could be used to determine whether the mod_lisp apache module is the
cause of the problem.

The downside is that libapache-mod-webapp integrates with cl-imho and
cl-imho only runs on cmucl to date. So there is no clisp specific code
to test/use. And the configuration directives are different (and I
haven't managed to get them to work--but I'm out of my depth):
http://freesw.onshored.com/wwwdist/imho/doc/mod-webapp.html

Regards,
Adam

---

From: Sam Steingold <[email-masked]>
Date: Sun Jul 14 11:20:02 2002

> * In message <1026549146.2319.44.camel@work>
> * On the subject of "[clisp-list] clisp, mod_lisp and lockups"
> * Sent on 13 Jul 2002 20:32:26 +1200
> * Honorable Adam Warner <[email-masked]> writes:
>

as per ANSI CL, DEFPACKAGE need NOT be enclosed in EVAL-WHEN.

>       (loop (let ((connection  (ext:socket-accept passive-socket)))

note that both clauses are almost identical.
the difference is what is being closed.
are you sure this is correct?

> 					(error (condition) (declare (ignore-if-unused condition))

there is no declaration "ignore-if-unused", only IGNORE and IGNORABLE.
these HANDLER-CASE forms are equivalent to IGNORE-ERRORS.

in general, at least reporting error would have been _very_
enlightening!

> 					(error (condition) (declare (ignore-if-unused condition))

passive-socket is not a socket in CLISP, but a socket-server, so CLOSE
will barf on them.
use CLOSE-SOCKET-SERVER instead.

>   (write-char #\NewLine *apache-socket*)

this is weird: I would have expected
    (write-string ": " *apache-socket*)
instead

>   (write-char #\NewLine *apache-socket*))

if you create a test case reproducible without apache, I can try to
debug it.

thanks!

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
The difference between genius and stupidity is that genius has its limits.

---

From: Adam Warner <[email-masked]>
Date: Mon Jul 15 19:27:01 2002

> if you create a test case reproducible without apache, I can try to

Thanks for the offer of help Sam. I spent a while looking at how I would
go about attempting to create my own version of mod_lisp for Apache 2
and learned a lot about apache module development in the process. But in
the process I also realised that clisp is great for plain CGI
development!

Initial testing of using clisp as a CGI scripting language is
impressive. I can load a simple initialisation program, access the
environment variables within a shell and return the result to wget via
Apache 2.0.39 in 0.18s on a Pentium 166 with 64MB of RAM.

The initialisation file, init.lisp just contains:

(format t "Content-type: text/html~2%")
(princ "Hello from init.lisp")

And is compiled with "clisp -c init.lisp" and is placed in the document
root directory.

The CGI file contains:

#!/usr/bin/clisp
(load (concatenate 'string (getenv "DOCUMENT_ROOT") "init.fas"))

(princ "<pre>")
  (princ (shell "set"))
(princ "</pre>")

And returns the list of environment variables to the client.
(NB to access environment variables individually refer
http://clisp.sourceforge.net/clash.html)

I consider I will be able to go a long way in providing rich content
before I need to load the initialisation file persistently in a
continually running clisp environment.

It annoys me that each script has to contain:

#!/usr/bin/clisp
(load (concatenate 'string (getenv "DOCUMENT_ROOT") "init.fas"))

To allow each CGI access to the required macros, etc. It would be nice
if I could hack mod_cgi so that this initialisation text was included by
the module (if for example the filename suffix was "lisp").

Regards,
Adam

---

From: Sam Steingold <[email-masked]>
Date: Tue Jul 16 07:04:04 2002

> * In message <1026786537.2031.30.camel@work>
> * On the subject of "Re: [clisp-list] clisp, mod_lisp and lockups"
> * Sent on 16 Jul 2002 14:28:57 +1200
> Thanks for the offer of help Sam. I spent a while looking at how I
> would go about attempting to create my own version of mod_lisp for
> Apache 2 and learned a lot about apache module development in the
> process. But in the process I also realised that clisp is great for
> plain CGI development!

indeed, CLISP is the perfect tool for any job (except heavy FP
computations, and even there FFI can get you a long way :-)

> (load (concatenate 'string (getenv "DOCUMENT_ROOT") "init.fas"))

do you know about memory images?
<http://clisp.cons.org/impnotes.html#image>

> http://clisp.sourceforge.net/clash.html)

hmmm....
I don't like that you refer to clash.html for information on CLISP
internals and I don't like that you have to resort to SHELL to list
your environment.

I just fixed that.
Now (getenv nil) returns all the environment as an alist and it is
documented in the impnotes.

thanks for bringing this to my attention.

> the module (if for example the filename suffix was "lisp").

dump your own image and use

#!/usr/bin/clisp -M my-cgi-image.mem

see <http://clisp.cons.org/clisp.html#opt-memfile>

--
Sam Steingold (http://www.podval.org/~sds) running RedHat7.2 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
All extremists should be taken out and shot.

---

From: Adam Warner <[email-masked]>
Date: Tue Jul 16 18:57:01 2002

> documented in the impnotes.

Thanks for implementing this!

> see <http://clisp.cons.org/clisp.html#opt-memfile>

It is a great idea and it will mean an initialisation file will be able
to greatly increase in complexity without a corresponding increase in
startup times.

One annoyance is that the memory image should be loaded from a different
directory (e.g. DOCUMENT_ROOT--not the current directory where it will
not exist). The Catch 22 is that before a clisp or bash shell is loaded
I cannot obtain the value of DOCUMENT_ROOT. I don't like hard wiring
paths (NB the CGI path only contains "/sbin:/bin:/usr/sbin:/usr/bin" but
a symlink from within this path is not obeyed in the #! line. One
workaround is to place a symlink to the memory image within each CGI
working directory).

BTW Joerg-Cyril, no I've only been looking at clisp startups at this
stage. And they are impressive. Even though perl will probably have
faster startup times I suspect that the ability to dump clisp memory
images is a huge bonus, especially as program complexity increases.

Regards,
Adam