Subject: [clisp-list] ((SETF STREAM-ELEMENT-TYPE) new-element-type stream)

From: Adam Warner <[email-masked]>
Date: Fri Sep 6 22:40:03 2002

Hi all,

I'm hoping you can help me understand how to create a Gray binary output
stream.

First I create the stream (CLISP 2.29):

(setf bstream (make-instance 'FUNDAMENTAL-BINARY-OUTPUT-STREAM))
#<FUNDAMENTAL-BINARY-OUTPUT-STREAM #x204490E5>

Then I attempt to set the STREAM-ELEMENT-TYPE according to the CLISP
specific impnotes (what's the cross-platform way?):

((SETF STREAM-ELEMENT-TYPE) '(unsigned-byte 8) bstream)   

*** - NO-APPLICABLE-METHOD: When calling #<GENERIC-FUNCTION (SETF
STREAM-ELEMENT-TYPE)> with arguments ((UNSIGNED-BYTE 8)
#<FUNDAMENTAL-BINARY-OUTPUT-STREAM #x204490E5>), no method is
applicable.

Thanks,
Adam

---

From: Sam Steingold <[email-masked]>
Date: Sat Sep 7 09:47:03 2002

> * In message <1031377382.2222.10.camel@work>
> * On the subject of "[clisp-list] ((SETF STREAM-ELEMENT-TYPE) new-element-type stream)"
> * Sent on 07 Sep 2002 17:43:02 +1200
> * Honorable Adam Warner <[email-masked]> writes:
>
> I'm hoping you can help me understand how to create a Gray binary
> output stream.

see, e.g., describe.lisp in the CLISP sources for inspiration.

> #<FUNDAMENTAL-BINARY-OUTPUT-STREAM #x204490E5>

what do you think this stream should do?

> #<FUNDAMENTAL-BINARY-OUTPUT-STREAM #x204490E5>), no method is

before you can call a function, you have to define it.
before you can write
        (setf (stream-element-type bstream) '(unsigned-byte 8))
you have to write
        (defmethod (setf stream-element-type) ((bstream fundamental-binary-output-stream)))

what are you trying to accomplish?

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
If abortion is murder, then oral sex is cannibalism.

---

From: Adam Warner <[email-masked]>
Date: Sat Sep 7 16:21:02 2002

> see, e.g., describe.lisp in the CLISP sources for inspiration.

Thanks. I really have to work to understand the CLOS system.

> what are you trying to accomplish?

This is what I have accomplished. It decodes URI encoded URL's/HTML form
substrings:

(defparameter *faithful* (ext:make-encoding :charset 'charset:iso-8859-1
:line-terminator :unix))

    (defun unescape-uri (string)
      (let* ((newstring (substitute #\Newline #\+ string))
             (start-pos 0)
             (list (loop for pos = (position #\% newstring :start start-pos) until (eq pos nil)
                         unless (= pos start-pos) collect (subseq newstring start-pos pos)
                         collect (subseq newstring pos (+ pos 3))
                         do (setf start-pos (+ pos 3)))))
        (when (<= start-pos (1- (length newstring)))
          (setf list (append list (list (subseq newstring start-pos)))))

        ;;now that the string is broken up I need to turn all the
        ;;escaped characters into actual characters.
        ;;I encode the string as faithful character then convert it to binary.
        ;;It is then converted to the default terminal encoding (which should be UTF-8).

        (let ((current-encoding *terminal-encoding*))
          (setf  *terminal-encoding* *faithful*)
          (let ((binary
                 (convert-string-to-bytes
                  (with-output-to-string
                    (stream)
                    (dolist (item list)
                      (if (string= (subseq item 0 1) "%")
                          (write-char (code-char (parse-integer (subseq item 1) :radix 16)) stream)
                        (write-string item stream)))) *terminal-encoding*)))
            (setf *terminal-encoding* current-encoding)
            (convert-string-from-bytes binary *terminal-encoding*)))))

e.g. (unescape-uri
"English+%28Braille%29%3A+%E2%A0%8A%E2%A0%80%E2%A0%89%E2%A0%81%E2%A0%9D%E2%A0%80%E2%A0%91%E2%A0%81%E2%A0%9E%E2%A0%80%E2%A0%9B%E2%A0%87%E2%A0%81%E2%A0%8E%E2%A0%8E%E2%A0%80%E2%A0%81%E2%A0%9D%E2%A0%99%E2%A0%80%E2%A0%8A%E2%A0%9E%E2%A0%80%E2%A0%99%E2%A0%95%E2%A0%91%E2%A0%8E%E2%A0%9D%E2%A0%9E%E2%A0%80%E2%A0%93%E2%A0%A5%E2%A0%97%E2%A0%9E%E2%A0%80%E2%A0%8D%E2%A0%91")

Will return line 45 of the UTF-8 sampler if the current terminal
encoding is UTF-8: http://www.columbia.edu/kermit/utf8.html

The kludge is that I currently have to simulate a binary stream using
with-output-to-string, convert that to a vector and then convert that to
a UTF-8 encoded string. If I had a vector stream of (unsigned-byte 8) I
could bypass the conversion from the simulated binary stream to the
actual vector stream (plus writing to a vector stream should be faster
than a string stream).

For a speedup I realise I could bypass the intermediate list
representation of the original string.

Regards,
Adam

---

From: Sam Steingold <[email-masked]>
Date: Sat Sep 7 17:44:02 2002

> * In message <1031440965.1510.43.camel@work>
> * On the subject of "Re: [clisp-list] ((SETF STREAM-ELEMENT-TYPE) new-element-type stream)"
> * Sent on 08 Sep 2002 11:22:44 +1200
> with-output-to-string, convert that to a vector and then convert that
> to a UTF-8 encoded string. If I had a vector stream of (unsigned-byte
> 8) I could bypass the conversion from the simulated binary stream to
> the actual vector stream (plus writing to a vector stream should be
> faster than a string stream).

why use streams at all?
doesn't VECTOR-PUSH-EXTEND do what you want?

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
God had a deadline, so He wrote it all in Lisp.

---

From: Adam Warner <[email-masked]>
Date: Sat Sep 7 18:51:02 2002

> doesn't VECTOR-PUSH-EXTEND do what you want?

Why yes it does. Thanks for pointing out this functionality!

Speed wise--since all non-escaped (%xx encoded) string elements are
simply printed to the stream using write-string--any byte-by-byte
approach would probably be slower (since I can't compile to native
code).

The way to make it faster would be to avoid converting the plain ASCII
portions to vectors in the first place. The output string would a
concatenation of the plain ASCII portions and the VECTOR-PUSH-EXTENDed
portions converted back to strings.

Thanks for the advice Sam. It was much appreciated.

CLISP's great Unicode support was one of the clear reasons for choosing
to develop using CLISP instead of CMUCL.

Regards,
Adam

(Note from the impnotes: UTF-8 only supports Unicode-16 encoding up to
three bytes. Longer UTF-8 sequences do not yet appear to be supported:

"UTF-8, the 16-bit UNICODE character set. Every character is represented
as one to three bytes. ASCII characters represent themselves and need
one byte per character. Most Latin/Greek/Cyrillic/Hebrew characters need
two bytes per character, and the remaining characters need three bytes
per character. This is therefore, in general, the most space-efficient
encoding of all of Unicode-16."

UTF-8 encoded characters may be up to six bytes long. The 16-bit
character subset is up to three bytes long as stated above.)

---

From: Bruno Haible <[email-masked]>
Date: Mon Sep 9 11:06:04 2002

Adam Warner writes:

> three bytes. Longer UTF-8 sequences do not yet appear to be supported

This is corrected current in the CVS sources of clisp: All of Unicode,
up to U+10FFFF, will be supported in the next clisp release. (And no
changes to your programs will be needed when switching from the 16-bit
Unicode subset clisp to the full Unicode clisp.)

The characters that need more than three bytes when encoded as UTF-8
are rarities like Etruskian alphabet, musical symbols, and rare
Chinese ideographs.

Bruno