Subject: [clisp-list] CGI & binary i/o with stdio: solution proposal

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Wed May 14 07:55:14 2003

Hi,

I wrote
>What do you expect to read from the following stream?
>(ext:make-pipe-io-stream "clisp -x '(describe 
>*standard-output*)(exit)'")

Since I am convinced that an extra command line option is needed, I investigated that path. It works.

Without actually adding a switch, I just hacked my CLISP to always use a file-stream for standard-input and output. On such a stream, (SETF (STREAM-ELEMENT-TYPE) works, so I can do binary i/o on stdio!

Witness the session log:

[8]> *standard-output*
#<OUTPUT UNBUFFERED FILE-STREAM CHARACTER #P"/dev/fd/1" @1>
[9]> *standard-input*
#<INPUT UNBUFFERED FILE-STREAM CHARACTER #P"/dev/fd/0" @13>

[1]> *error-output*
#<OUTPUT UNBUFFERED FILE-STREAM CHARACTER #P"/dev/fd/1" @1>
[2]> *debug-io*
#<IO SYNONYM-STREAM *TERMINAL-IO*>
[3]> *query-io*
#<IO SYNONYM-STREAM *TERMINAL-IO*>
[4]> (setq *error-output* *debug-io*) ; errors still to terminal
#<IO SYNONYM-STREAM *TERMINAL-IO*>
[5]> (unwind-protect
 (progn (setf (stream-element-type *standard-input*) '(unsigned-byte 8))
        (read-byte *Standard-input*))
 (setf (stream-element-type *standard-input*) 'character))
abcdef
97	; ASCII for a
[6]>
*** - EVAL: variable BCDEF has no value ; what was left on the line
The following restarts are available:
STORE-VALUE    :R1      You may input a new value for BCDEF.
USE-VALUE      :R2      You may input a value to be used instead of BCDEF.
1. Break [7]> :a

[10]> (file-position *standard-output*)
NIL
[11]> (file-position *standard-input*)
NIL

Here's the trivial diff
***************
*** 15330,15339 ****
--- 15333,15344 ----
  # UP: Return the default value for *terminal-io*.
  # can trigger GC
  local object make_terminal_io (void) {
    # If stdin or stdout is a file, use a buffered stream instead of an
    # unbuffered terminal stream. For the ud2cd program used as filter,
    # this reduces the runtime on Solaris from 165 sec to 47 sec.
    var bool stdin_file = regular_handle_p(stdin_handle);
    var bool stdout_file = regular_handle_p(stdout_handle);
+  # JCH HACK to experiment with setf stream-element-type
+  stdin_file = true; stdout_file= true;
    if (stdin_file || stdout_file) {
      var object stream;
      # Allocate stream for stdin:
      if (stdin_file) {
       #ifdef UNIX

The hack needs a little more polishing, because *error-output* is set EQ to *standard-output*, which for the demo & debugging I don't like. That explains my step 4.
[4]> (setq *error-output* *debug-io*)

global void init_streamvars (bool unixyp) # src/stream.d
    stream = Symbol_value(S(standard_output));
    define_variable(S(error_output),stream);     # *ERROR-OUTPUT*

The remaining questions now are:

1. What should *standard-error* become in such a case?
It would be wise *not* to have it bound to the file stream object as *standard-output*. If it's used in binary mode, an infinite loop results of this, and a Lisp STACK overflow. I prefer to have *standard-error* still point to *terminal-io* (which remains a pure character stream) (or a handle for file descriptor #2, if exists). That means that error and trace messages would be mixed as ASCII into the binary output -- at least you get to see something.
Of course, a CGI application needs to play safe and direct its *trace- etc -output* to something reasonable.

2. What should the command-line option be called?
  -noI comes to mind (readline would have to be disabled as well).

BTW, do we really need -repl? I thought, like -i for sh/csh, something to force interactive mode might be enough, together with -noi?

I feel this binary i/o issue is related to forced non-interactive (aka batch) mode. Normally, you wouldn't use a file-stream on an interactive terminal, like I did. You only want that in pipes or sockets (e.g. the cgi application case). There, you have to define what entering the debugger means... Do you really want a Lisp error message go into HTML output? Do you want the debugger to wait for commands from your HTTP-1.1 stream? -- no!

And yes, I feel like CLISP misses an option to indicate (force) batch mode. Currently, it guesses:
  if ((argv_compile || argv_expr_count || argv_execute_file != NULL)
      && !argv_interactive_debug && !argv_repl) {
    # '-c' or '-x' or file specified -> LISP runs in batch-mode:
But it can guess wrong. I could build an image and need neither -i file nor -x to start since the image's startup function could do all I want (e.g. obtain data from ext:getenv instead of via -x): typical cgi.

Regards,
	Jorg Hohle.

---

From: Sam Steingold <[email-masked]>
Date: Thu May 15 11:27:14 2003

> * In message <[email-masked]>
> * On the subject of "CGI & binary i/o with stdio: solution proposal"
> * Sent on Wed, 14 May 2003 16:54:13 +0200
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>

we already have -repl, so maybe -norepl?

> BTW, do we really need -repl? I thought, like -i for sh/csh, something
> to force interactive mode might be enough, together with -noi?

that's what -repl does - forces interaction after -c/-x/script

>     # '-c' or '-x' or file specified -> LISP runs in batch-mode:

argv_norepl should do the job.

(check out the new EXT:MAKE-STREAM, it should Thibault Langlois with his
problem)

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
There's always free cheese in a mouse trap.