Subject: [clisp-list] character sets

From: [email-masked] (Don Cohen)
Date: Mon Mar 15 10:58:26 2004

This must be a frequently asked question - I must have asked it myself.
Is there some character set that contains 256 8 bit bytes?
All I really want to do is read characters without any errors.

Somehow I got the impression from impnotes that this could be done
with
 (with-open-file 
  (f "/root/http/log" :external-format 
    (make-encoding :input-error-action :ignore)) ...)
but I still get
 *** - READ from #<INPUT BUFFERED FILE-STREAM CHARACTER #P"/root/http/log" @2>: \illegal character #\U008D 

It would be even better if I could then figure out what the bytes had
been from the characters (1-1 mapping).

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Mon Mar 15 12:31:28 2004

> Is there some character set that contains 256 8 bit bytes?

You  are mixing  two different  things.   You can  have a  set of  256
character.  And you can have a character encoding using 8-bit bytes.

Normally, clisp is compiled to have unicode characters.  There is more
than  256 unicode  characters.  The  number of  bits used  to  store a
unicode  character  is  implementation  dependent, and  most  probably
variable (a unicode character may need various code points).

When you write characters, you must choose an encoding that is able to
store the characters you want to  write.  And when you read an encoded
character  stream, you  must decode  it,  and not  all byte  sequences
correspond to valid character.

> All I really want to do is read characters without any errors.

Then provide an error-free file!

> been from the characters (1-1 mapping).

You cannot because it depends on the encoding used.

I have the feeling that what you want to do is to actually read binary
bytes instead of reading characters.

   (with-open-file (f path :element-type '(unsigned-byte 8))
      (do ((byte (read-byte f nil nil)(read-byte f nil nil)))
          ((null byte))
         (format t "read byte ~D~%" byte)))

You can always try to convert these bytes to characters with
(code-char byte), but beware that some byte value correspond to no
defined characters:

     (code-char #16r81) --> #\U0081
vs.:
     (code-char #16rC1) --> #\LATIN_CAPITAL_LETTER_A_WITH_ACUTE

This is probably unrelated, but  clisp can be compiled to support only
256 characters instead of all unicode characters.

 --without-unicode       no UNICODE support: character=8bit

I don't know  what it would do if  you tried to read in  such a dumbed
down beast an ISO-8859-1 file and a ISO-8859-2 file.

-- 
__Pascal_Bourguignon__                     http://www.informatimago.com/
There is no worse tyranny than to force a man to pay for what he doesn't
want merely because you think it would be good for him.--Robert Heinlein
http://www.theadvocates.org/

---

From: [email-masked] (Don Cohen)
Date: Mon Mar 15 12:41:27 2004

> variable (a unicode character may need various code points).

I think it was clear what I want.
I want a character set with 256 characters, and an encoding that
maps those 1-1 with all of the 8 bit combinations.
I want no multibyte characters.

 > Then provide an error-free file!
I consider a file containing 256 different bytes to be error free.

 > You cannot because it depends on the encoding used.

But make-encoding should be able to make an encoding that does
whatever I want, shouldn't it?

 > bytes instead of reading characters.

That's what I have been forced to do so far.
But that seems absurd.  Why not create the obvious encoding and use
characters?

 > down beast an ISO-8859-1 file and a ISO-8859-2 file.

It is not unrelated.  If I can do it without unicode (which I've not
actually checked) then I should be able to do it with unicode.

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Mon Mar 15 12:55:06 2004

> I want no multibyte characters.

There is  no encoding that I know  of that maps 1-1  256 8-bit values.
For example, ISO-8859-* does not  map values between 128 and 159. Even
the various IBM-PC OEM encoding leave some codes unmapped.

You could invent your own mapping though.

> whatever I want, shouldn't it?

You did not pass it a :charset argument with a set of 256
characters.  Why should it not return an error?

Note that since there exist no character with a code equal to 141,
YOU will have to choose an existing character to put in the slot 141
of the :charset argument.

Remember that  in C, char  is actually an  integer type!  So  it's not
surprizing that you  can store 141 in a char  variable.  But that does
not mean that there exist any  character that have a code equal to 141
in any official encoding.  You still can invent your own encoding, but
you have to specify it (with the :charset key)!

> But that seems absurd.  Why not create the obvious encoding and use

A good reason why not would be that it is not portable accros
Common-Lisp implementations... Reading bytes is pure Common-Lisp.

---

From: [email-masked] (Don Cohen)
Date: Mon Mar 15 13:38:05 2004

> You could invent your own mapping though.
I thought so, but have not yet succeeded.

 > characters.  Why should it not return an error?
The code I sent did not attempt to generate a 1-1 mapping, just
a mapping that would not generate an error.  And even that did
not work.
Notice that make-encoding did not complain.  It was when I read
an illegal character that it complained, and I thought that's what
:ignore was supposed to prevent.  If I supply a default character
I still get an error.

 > of the :charset argument.
And is this not what the argument
(make-encoding :input-error-action #\space) [or :ignore]
is supposed to do?

 > you have to specify it (with the :charset key)!

Which one should I use?  I've now tried a few - same result.

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Mon Mar 15 14:36:01 2004

> :ignore was supposed to prevent.  If I supply a default character

There is no invalid characters.  Characters are.  There may be invalid
codes, codes that correspond to no character, and there may be invalid
byte sequences, byte sequences that encode no character.

> Which one should I use?  I've now tried a few - same result.

Now, I must excuse me, I thought that the string that can be passed as
:charset argument to  make-encoding was a set of  character, but it is
not.  It  is expected  to be  the name of  an existing  character set.
There  does not  appear  to be  any public  (implementation-dependend)
function to create your own character set.

That  means  that you'll  have  to go  the  portable  way, write  pure
Common-Lisp code,  and implement your own encoding  yourself.  This is
not difficult at all:

(defvar *my-encoding*
  (let ((cs (make-string 256  :initial-element (character "?"))))
    (loop for i from 32 upto 127
          do (setf (aref cs i) (code-char i)))
    cs));;*my-encoding*

(defun read-char-with-my-encoding
  (stream &optional (eof-error-p t) eof-value)
  (assert (equal '(unsigned-byte 8) (stream-element-type stream)))
  (let ((byte (read-byte stream eof-error-p eof-value)))
    (if (typep byte '(unsigned-byte 8))
      (aref *my-encoding* byte)
      byte)));;read-char-with-my-encoding

(defun write-char-with-my-encoding
  (character &optional (output-stream *standard-output*))
  (assert (equal '(unsigned-byte 8) (stream-element-type output-sstream)))
  (let ((byte (position character *my-encoding*
                        ;; A little hack depending on *my-encoding*:
                        :start 32)))
    (write-char (aref *my-encoding* byte) :output-stream output-stream))
  );;write-char-with-my-encoding

(with-open-file (in "/tmp/a" :element-type '(unsigned-byte 8))
  (do ((char (read-char-with-my-encoding in nil nil)
             (read-char-with-my-encoding in nil nil))
       (i 0 (1+ i)))
      ((null char))
    (when (= 0 (mod i 17)) (terpri))
    (format t "~C " char)))

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
?   ! " # $ % & ' ( ) * + , - . / 
? 0 1 2 3 4 5 6 7 8 9 : ; < = > ? 
? @ A B C D E F G H I J K L M N O 
? P Q R S T U V W X Y Z [ \ ] ^ _ 
? ` a b c d e f g h i j k l m n o 
? p q r s t u v w x y z { | } ~ _ 
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
? 
NIL

$ od -t x1 /tmp/a
0000000 0a 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e
0000020 0f 0a 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d
0000040 1e 1f 0a 20 21 22 23 24 25 26 27 28 29 2a 2b 2c
0000060 2d 2e 2f 0a 30 31 32 33 34 35 36 37 38 39 3a 3b
0000100 3c 3d 3e 3f 0a 40 41 42 43 44 45 46 47 48 49 4a
0000120 4b 4c 4d 4e 4f 0a 50 51 52 53 54 55 56 57 58 59
0000140 5a 5b 5c 5d 5e 5f 0a 60 61 62 63 64 65 66 67 68
0000160 69 6a 6b 6c 6d 6e 6f 0a 70 71 72 73 74 75 76 77
0000200 78 79 7a 7b 7c 7d 7e 7f 0a 80 81 82 83 84 85 86
0000220 87 88 89 8a 8b 8c 8d 8e 8f 0a 90 91 92 93 94 95
0000240 96 97 98 99 9a 9b 9c 9d 9e 9f 0a a0 a1 a2 a3 a4
0000260 a5 a6 a7 a8 a9 aa ab ac ad ae af 0a b0 b1 b2 b3
0000300 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf 0a c0 c1 c2
0000320 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf 0a d0 d1
0000340 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df 0a e0
0000360 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef 0a
0000400 f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff
0000420 0a
0000421

---

From: [email-masked] (Don Cohen)
Date: Mon Mar 15 18:12:14 2004

I now realize that this is not a problem with encodings but a problem 
with READ.
 > >  *** - READ from #<INPUT BUFFERED FILE-STREAM CHARACTER #P"/root/http/log" @2>: \illegal character #\U008D 

It had not occurred to me that certain characters would not be allowed
by READ.  I imagined that all those that were not otherwise specified
would be treated like alphabetic characters.
How foolish of me.

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Mon Mar 15 18:37:43 2004

> would be treated like alphabetic characters.

There is no character with code 141.  The error message is misleading.

On the other hand, if you're trying to apply the lisp reader on random
input, it's not surprizing that some lexical error may occur.

Actually, I've  got no problem  in reading strings containing  all the
byte values from 0 to 255.

[15]> (with-open-file (in "/tmp/a" :direction :input 
                          :external-format (ext:make-encoding 
                                                :charset "iso-8859-1"
                                                :line-terminator :unix)) 
        (dotimes (i 17) (read-line in)))
NIL

I'm realizing  that you're using the  LISP reader to  read random data
files.  Why are  you doing that?  For all  I know, /root/http/log does
not  contain   any  LISP  data  (symbols,   strings,  numbers,  lists,
structures),  does it?   Of course,  the  LISP reader,  which use  the
*readtable* will find  that the (pseudo- IMO) character  with code 141
has no valid syntax, so it calls it "illegal", under _its_ law.

[18]> (set-syntax-from-char #\% #\u008d)
T
[19]> '%oto
*** - READ from #<INPUT CONCATENATED-STREAM #<INPUT STRING-INPUT-STREAM> #<IO TERMINAL-STREAM>>: illegal character #\%

If you really have LISP data in  this file and want to be able to read
unescaped  symbols,  the you  should  copy  a  legal syntax  to  these
characters.  But what syntax?  Is  it used as a symbol constituent, as
a whitespace, as a macro character, or as an escape?

Assuming you want it as a constituent, you could do:

         (set-syntax-from-char  #\u008d #\a)
and be happy.

[22]> (set-syntax-from-char #\% #\a)
T
[23]> '%toto
%TOTO

On the  other hand, as is most  probable, you just have  plain data in
this  file,  then  you don't  want  to  use  READ  on it.   Stay  with
READ-STRING, READ-CHAR, or READ-BYTE.  With READ-STRING and READ-CHAR,
you still have to ensure  that your file contains valid byte sequences
for  the encoding it  uses.  If  you want  to be  able to  read random
bytes, use READ-BYTE.  See my previous messages.

---

From: [email-masked] (Don Cohen)
Date: Mon Mar 15 19:58:23 2004

> There is no character with code 141.  The error message is misleading.

What do you think #\U008D is?

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Mon Mar 15 20:28:36 2004

> What do you think #\U008D is?

A figment of clisp unicode's imagination. :-)

While there is  only one encoding and only one  character set, you can
always  add imaginary  characters to  get  a 1-1  mapping between  the
encoding and the  character set.  But once you  want to manage various
encoding, while you can always recognize and unify an upper case latin
a letter,  and associate it  with the corresponding byte  sequences in
the various encodings, you would have much more difficulties doing the
same  starting  from  unassigned   byte  sequences  from  the  various
encodings. Does 141  in ISO-8859-1 correspond to 63  or 137 in EBCDIC?
You  have  a different  number  of  unassigned  codes in  the  various
encodings. How  many imaginary characters  will you create?   How will
you map the various encodings since there's not the same number?

On  the other  hand,  you  could manage  internally  codes instead  of
managing characters.   That's what  C does.  But that's not  what Lisp
does (or wants to do).

---

From: Phil Chu <[email-masked]>
Date: Mon Mar 15 23:07:18 2004

Hi Don,

This might be of no practical use, but note that unicode is just an 
assignment of numbers to characters, and there are a variety of 
encodings available, e.g. latin-1 and utf-8 both encode the characters 
0-255 with one byte. As I remember, Java is defined to support unicode 
internally but specific encodings can be used with the I/O methods. I 
don't know what the situation is with CL.

-Phil

Don Cohen wrote:

> -------------------------------------------------------
> This SF.Net email is sponsored by: IBM Linux Tutorials
> Free Linux tutorial presented by Daniel Robbins, President and CEO of
> GenToo technologies. Learn everything from fundamentals to system
> administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
> _______________________________________________
> clisp-list mailing list
> [email-masked]
> https://lists.sourceforge.net/lists/listinfo/clisp-list
> 

-- 
Phil Chu
[email-masked]
http://www.technicat.com/