Subject: [clisp-list] *standard-input* encoding

From: Gabor Melis <[email-masked]>
Date: Thu Jul 22 04:30:06 2004

I'm writing a traditional UNIX filter that works with binary input as 
well.ISO-8859-1 looks to be the safe bet, so the script that invokes clisp 
looks like this:

#!/usr/bin/clisp -Eterminal iso-8859-1 -Efile iso-8859-1 ...

The problem is that the whole line is too long for the linux kernel and it 
gets truncated. I would like to set the encodings in the lisp code, but the 
*standard-input* stream already exists.

Is there a way out of this?

Thanks, Gabor

---

From: "John K. Hinsdale" <[email-masked]>
Date: Thu Jul 22 06:35:12 2004

> The problem is that the whole line is too long for the linux kernel

Well I have a solution, it DOES work for me, but there may be better
ways.  But I post this in case you are desperate.

Yes, on my Linux box it seems to be limited to 128 characters ...
Surprisingly lame of Linux/Unix really ...

 > I would like to set the encodings in the lisp code, but the 

Right, so I think you need to set the encodings via the CLISP command
line arguments, before CLISP sets up *STANDARD-INPUT*, and freezes its
character encoding for the remainder of the program (developers, is
that correct?).

At any rate, a way around the silly 128-char limit of the "!" line, is
to make a "C" program wrapper that inserts the extra CLISP arguments
for you.  [I tried to do this hack with a plain /bin/sh shell wrapper
and it did not work -- I do not think you can "nest" #! interpreters].

 > Is there a way out of this?

YES, you can use the method below, unless Sam has a better idea ;)

This hack allows a CLISP program to read binary data from
*standard-input* even when fed as a Unix pipe ...

Create the following two files, "myclisp.c" and "test.sh" by cutting
and pasting what is attached below.  Edit "prepend_args" array in
"myclisp.c" to add as many extra "standard" CLISP options you want
(i.e., the ones that are taking up all the room ;)   Note each option
must be a separate string in the array (don't put spaces inside the
strings).  Then compile it with:

     gcc -o myclisp myclisp.c 

Install "myclisp" somehere accessible to you (in your home, or
/usr/local/bin for systemwide visibility, whatever)

Edit "test.sh" to replace the first line (/path/to/myclisp) with where
you installed the compiled "myclisp" ...

Run test.sh, feeding it some binary data.  Note that you can do BOTH
	./test.sh <my-binary-data
AND
	cat my-binary-data | ./test.sh

This second form (Unix pipe/filter) is I believe what you are asking
to do.  CLISP opens *STANDARD-INPUT* differently in each of these two
cases.  In the first case CLISP sees a "file" type of stream, in the
second a "terminal' type of stream.  In the second case, when CLISP
opens the *standard-input* as a terminal, you _cannot_ use READ-BYTE
on it.  See:

   http://clisp.cons.org/impnotes/stream-dict.html#bin-stdio

However, you _can_ use READ-CHAR, which, when you use an 8-bit char
set like ISO-8859-1, you can get the binary data.  You may then need
to further do something like:

     (setf binary-char (coerce (char-code c) '(unsigned-byte 8)))

to convert the ISO-8859-1 characters to raw bytes.  Note that the
ISO-8859 meaning of the character is irrelevant here.

Also note that you can invoke "test.sh" with additional arguments that
will show up in (ARGV) and which you can access in the Lisp code:

	binary-generator-program | test.sh arg1 arg2

I tested this out and it seems to work for me on Linux to read binary
(8-bit octets) from *standard-input*, whether from file or a pipe.

Hope this helps ... ugly but it works; code follows...

	Cheers,

-------  Cut below here and save as myclisp.c ---------
/* myclisp.c - Execute CLISP with extra options, bypassing 128-char
   limit of the Unix "#!" command-interpeter facility */

#include <unistd.h>

/* Execute this interpreter ... */
static char * interpreter = "/usr/local/bin/clisp";

/* ... with these arguments pre-pended to the args given to this program.
   There is no siginificant limit to the number or size of args here.
 */
static char * prepend_args[] = {
  "-Efile",     "iso-8859-1",
  "-Epathname", "iso-8859-1",
  "-Eterminal", "iso-8859-1",
  "-Eforeign",  "iso-8859-1",
  "-Emisc",     "iso-8859-1",
  "-q",
};

/* Re-exec a new interpreter, inserting arguments above */
int main(int argc, char **argv) {

  int nprepend = sizeof prepend_args / sizeof *prepend_args;
  char ** newargv = (char **) malloc((nprepend + argc + 1) * sizeof *newargv);
  int i;
  char **p;
  
  newargv[0] = interpreter;

  for (i=0; i < nprepend; i++)
    newargv[i+1] = prepend_args[i];

  for (i=nprepend; i < nprepend + argc; i++)
    newargv[i+1] = argv[i - nprepend + 1];

  argv[nprepend + argc] = 0;
  
  execvp(*newargv, newargv);
  return 1;
}
-------  Cut above here and save as myclisp.c ---------

-------  Cut below here and save as test.sh ---------
#!/u/hin/myclisp

; Replace the #! line above with the path to the compiled program "myclisp"

; Use READ-CHAR on a stream encoded in ISO-8851-1 or other
; 8-bit character set to get binary data
(defun read-binary (s)
  (print "Beginning of stream ...")
  (do (char eof)
      (eof)
      (setf char (read-char s nil))
          (if (not char) (setf eof t)
                (print (char-code char))))
  (print "... end of the stream"))

; Show the encodings we are using
(mapcar #'print (list
        "Here are the format encodings ..."
        *default-file-encoding*
        *foreign-encoding*
        *misc-encoding*
        *pathname-encoding*
        *terminal-encoding*
        "... end of the format encodings:"))

; Show the command arguments
(mapcar #'print (list
        "Beginning of the command arguments ..."
        (argv)
        "... end of the command arguments"))

; Read the binary from the standard input
(read-binary *standard-input*)
-------  Cut above here and save as test.sh ---------

---

From: Sam Steingold <[email-masked]>
Date: Thu Jul 22 07:23:19 2004

> * John K. Hinsdale <[email-masked]> [2004-07-22 09:34:33 -0400]:
>

how about just "-E iso-8859-1"?
<http://clisp.cons.org/clisp.html#opt-enc>

> Yes, on my Linux box it seems to be limited to 128 characters ...

<http://clisp.cons.org/impnotes/quickstart.html>:
On some platforms, the first line which specifies the interpreter is
limited in length:

    * max. 32 characters on SunOS 4,
    * max. 80 characters on HP-UX,
    * max. 127 characters on Linux.

> character encoding for the remainder of the program (developers, is

1. first, you can do (ext:make-stream :input)
   <http://clisp.cons.org/impnotes/stream-dict.html#make-stream>

2. IIRC, when CLISP runs as a script interpreter, *standard-input* is a
   usual buffered handle stream, so you can change its encoding at will.

> and it did not work -- I do not think you can "nest" #! interpreters].

yes.

> In the first case CLISP sees a "file" type of stream, in the second a
> "terminal' type of stream.  In the second case, when CLISP opens the
> *standard-input* as a terminal, you _cannot_ use READ-BYTE on it.
> See:

*STANDARD-INPUT* should be a synonym to *TERMINAL-IO* only when user
interaction is expected.  Please report everything else as a bug (but
first test against CVS head because I think it has been fixed).

> ISO-8859 meaning of the character is irrelevant here.

this is not a good idea, both conceptually and performance-wise.

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Bill Gates is great, as long as `bill' is a verb.

---

From: Gabor Melis <[email-masked]>
Date: Thu Jul 22 08:22:25 2004

> <http://clisp.cons.org/clisp.html#opt-enc>

Hmm. I didn't realize that's possible. Something is strange, though:

$ clisp -q "-E iso-8859-1"

doesn't work (prints usage), but

$ clisp "-E iso-8859-1" -q

and

$ clisp -q -Efile iso-8859-1

does.

>     * max. 127 characters on Linux.

That's so low that depending on the length of the path where the xxx program 
is installed my script can run into trouble. What about this wasteful 
solution?

#!/bin/sh
F=/tmp/xxx-invoker$$
echo '(xxx)' > $F
# "-E iso-8859-1" doesn't work here either
/home/mega/lib/clisp/xxx/lisp.run -Efile iso-8859-1 -Eterminal iso-8859-1 \
    -M/home/mega/lib/clisp/xxx/lispinit.mem $F "$@"
rm -f $F

---

From: "John K. Hinsdale" <[email-masked]>
Date: Thu Jul 22 08:24:21 2004

> Something is strange, though:
 > doesn't work (prints usage),

CLISP wants the "-E" and the "iso-8859-1" to be separate Unix command
arguments.  When you put the quotes around it, gets squashed into
a single argument.

 > but
 > does

that is weird

 > does

as expected (the "-Efile" and its parameters are separate)

---

From: "John K. Hinsdale" <[email-masked]>
Date: Thu Jul 22 08:26:29 2004

>    usual buffered handle stream, so you can change its encoding at will.

see i knew Sam would have a better idea. ;)

I think you still need something like the "C" wrapper hack if you
really must use a lot of other command options.  (Setting the stdin
encoding and skirting the 128-char limit are really two different
issues).

I suppose in principle anything you can do on the command line you
should also be able to do from Lisp (like set memory usage, lanauage,
Lisp dialect etc.) so that use of lots command options can be avoided.

Cheers and good luck.

---
John Hinsdale, Alma Mater Software, Inc., Tarrytown, NY 10591-3710 USA
[email-masked] | http://www.alma.com/staff/hin | +1 914 631 4690

---

From: Sam Steingold <[email-masked]>
Date: Thu Jul 22 13:07:15 2004

> * Gabor Melis <[email-masked]> [2004-07-22 13:24:42 +0200]:
>
> well.ISO-8859-1 looks to be the safe bet, so the script that invokes
> clisp looks like this:
>
> and it gets truncated. I would like to set the encodings in the lisp
> code, but the *standard-input* stream already exists.

1. '#!/usr/bin/clisp -E iso-8859-1'
   should help you avoid the kernel limit
   <http://clisp.cons.org/clisp.html#opt-enc>

2. you do not need these in the command line at all:
   set the default encodings explicitly
   <http://clisp.cons.org/impnotes.html#enc-dflt>
   or pass the :external-format argument to open:
   <http://clisp.cons.org/impnotes.html#extfmt>

3. if you want binary input, you should do
   (setf (stream-element-type *standard-input*) 'unsigned-byte)
   <http://clisp.cons.org/impnotes.html#open>
   note that *STANDARD-INPUT* will be a synonym for *TERMINAL-IO* (and
   thus the above will fail) when CLISP expects an interactive session.
   Otherwise (e.g., when running as a script) *STANDARD-INPUT* will be
   a normal buffered handle stream [at least in the CVS HEAD].
   At any rate, you can always follow the advice in the Fine Manual:
   <http://clisp.cons.org/impnotes.html#bin-stdio>

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
There are two kinds of egotists: 1) Those who admit it  2) The rest of us