Subject: [clisp-list] character sets

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Tue Mar 16 02:11:11 2004

Hi,

Don Cohen writes:
>This must be a frequently asked question - I must have asked it myself.
>Is there some character set that contains 256 8 bit bytes?
>All I really want to do is read characters without any errors.

I wrote an entry for the CL cookbook:
"Faithful Output with Character Streams"
http://cl-cookbook.sourceforge.net/io.html#faith

Pascal Bourguignon wrote:
>Actually, I've  got no problem  in reading strings containing  all the
>byte values from 0 to 255.
>[15]> (with-open-file (in "/tmp/a" :direction :input 
                          :external-format (ext:make-encoding 
                                                :charset "iso-8859-1"
                                                :line-terminator :unix)) 
That proves nothing except that you were able to read one particular sequence of bytes as characters.

Basically, my article shows how to do 8bit writes using characters.
For input (read), you're doomed w.r.t. faithful input because there's no way to get around CLISP's CRLF translation at end-of line. If that difference is not important to you, you can use the same settings as for output: iso-8859-1 and :line-terminator :unix (as Pascal shows). You should then be able to read all 8-bit values without errors, as you requested (yet beware when some inputs end in CR, CLISP will probably wait for a possible LF before it returns a #\Newline, or it may return #\Newline immediately and remember to goble a possible consecutive LF, I forgot the exact behaviour).

BTW (yet OT), the XML standard mandates a similar conversion of line-endings. Application processing XML do not see the original line-terminator, they get a single UNIX-line #\Newline.

Regards,
	Jorg Hohle.

---

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Tue Mar 16 02:31:12 2004

Hi,

Pascal replied to Don Cohen:
>I'm realizing  that you're using the  LISP reader to  read random data
>files.

1. That file is not guranteed to contain pure ASCII, e.g. it could embded URL-requests as is, which might very well contain Umlauts in DOS or Latin-1 or whatever format -- I just don't know, and you probably don't know either!

2. Therefore, and for typical purposes, it's reasonable to just be able to ask for being able to read such a file without errors, as Don did, under the following hypotheses:
 - the file is a collection of text lines.
 - the data is mostly ASCII, which means that [CR]LF can be used to detect the end-of-line.
 - the data is mostly ASCII, and the first positions of each line will be readable under that assumption (e.g. date, time, etc.)

For such a purpose, I recommend using ISO-8859-1 and :line-terminator :unix.
It may not be the encoding used to write the data, but
 + it's an 8-bit superset of ASCII (unlike UTF-8) and
 + it provides means in CLISP to most likely process the extra characters and write them to other files, even though the actual data may be cyrillic, Hangul, UTF-8 or whatever.

The point where it might break is that I don't know whether Hangul (Korea), Japanese and other countries encodings which use multi-byte sequences can embed something which could be mistaken as CR or LF when read bytewise and which could find its way into the log files.

BTW, I recommend using READ-CHAR-SEQUENCE to process log files. I talked about that here when I mentioned my "10 times faster than perl" firewall log-processing application a year or so ago. You need buffering to achieve performance.

Regards,
	Jörg Höhle

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Tue Mar 16 05:24:20 2004

oehle, Joerg-Cyril writes:
> 1. That file is not guranteed to contain pure ASCII, e.g. it could 
> embded URL-requests as is, which might very well contain Umlauts in DOS 
> or Latin-1 or whatever format -- I just don't know, and you probably 
> don't know either!
> 
> 
> 2. Therefore, and for typical purposes, it's reasonable to just be able 
> to ask for being able to read such a file without errors, as Don did, 
> under the following hypotheses:
>  - the data is mostly ASCII, and the first positions of each line will 
>    be readable under that assumption (e.g. date, time, etc.)
> 
>  + it provides means in CLISP to most likely process the extra 
>    characters and write them to other files, even though the actual data 
>    may be cyrillic, Hangul, UTF-8 or whatever.

Ok. I was in  error when I wrote that there is  no character with code
ISO-8859-1 equal  to 141.   I thought that  between 128 and  159 codes
were not assigned (and that  an implementation would thus be justified
in  returning  an error  when  reading  these  code in  an  ISO-8859-1
encoding as it does when it  encounters bytes 192,193 in UTF-8), but I
just found that ISO-8859 assigns these codes to specific control codes
(control "characters"), so  that all 256 codes are  defined and should
correspond to characters.

> The point where it might break is that I don't know whether Hangul 
> (Korea), Japanese and other countries encodings which use multi-byte 
> sequences can embed something which could be mistaken as CR or LF when 
> read bytewise and which could find its way into the log files.

The file format  is broken anyway.  It should  have been written takin
special care  of these exotic  encodings.  For example, it  could have
been encoded in UTF-8.

> BTW, I recommend using READ-CHAR-SEQUENCE to process log files. I 
> talked about that here when I mentioned my "10 times faster than perl" 
> firewall log-processing application a year or so ago. You need 
> buffering to achieve performance.

Personnaly, I stay  with my advice and would  rather use READ-BYTE (or
READ-BYTE-SEQUENCE) for this  kind of files.  This would  allow you in
addition to implement  heuristics to detect the encoding  of any range
of bytes and decode them correctly.

-- 
__Pascal_Bourguignon__                     http://www.informatimago.com/
There is no worse tyranny than to force a man to pay for what he doesn't
want merely because you think it would be good for him.--Robert Heinlein
http://www.theadvocates.org/

---

From: [email-masked] (Don Cohen)
Date: Tue Mar 16 09:18:16 2004

> http://cl-cookbook.sourceforge.net/io.html#faith
Thank you.  Maybe one of these days I can even go back and
change my http/smtp/... servers to use characters again.
I think something about this ought to be in clisp impnotes and faq.
Also something about the syntax class of the extra characters.
Also something about the influence of LANG.
Also *print-pretty* - these things have bitten me again and again
in clisp.
I know, I should write something and submit it.

 > For input (read), you're doomed w.r.t. faithful input because
 > there's no way to get around CLISP's CRLF translation at end-of
 > line.
Oh, right, THIS is why I have to use binary input.
I really wish there were a way around this.
Why are the line terminator modes not used for input?
Perhaps there could be some backward compatible addition
to support this.  An extra flag for encodings that says to
use the line terminator mode on input?

 > > or Latin-1 or whatever format -- I just don't know, and you probably 
Turns out this guess is correct - I'm trying to read my http logs
to find out which IPs deserve to be filtered.

---

From: Sam Steingold <[email-masked]>
Date: Tue Mar 16 10:39:17 2004

> * Don Cohen <[email-masked]> [2004-03-16 09:10:50 -0800]:
>

did you actually look in the impnotes?!

> Also something about the syntax class of the extra characters.

<http://clisp.cons.org/impnotes.html#graph-char> &c.

> Also something about the influence of LANG.

<http://clisp.cons.org/clisp.html#environment>
<http://clisp.cons.org/clisp.html#opt-lang>
<http://clisp.cons.org/clisp.html#opt-enc>

> Also *print-pretty* - these things have bitten me again and again

<http://clisp.cons.org/impnotes/printer.html#multi-rep>

> Why are the line terminator modes not used for input?

<http://sourceforge.net/tracker/index.php?func=detail&aid=731952&group_id=1355&atid=351355>

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
In every non-trivial program there is at least one bug.

---

From: [email-masked] (Don Cohen)
Date: Tue Mar 16 11:36:03 2004

> did you actually look in the impnotes?!

Many times, often, every time I have such problems.
Whenever I post to the list about such a problem I have
searched impnotes for something about it.  
That gives you a lower bound.

Of course, just cause something is there doesn't mean it's 
easy to find!

 > <http://clisp.cons.org/impnotes.html#graph-char> &c.

I don't see how this was supposed to help me.

 > <http://clisp.cons.org/clisp.html#opt-enc>

This is the sort of thing that it's easy to read and not realize
how it affects you.  Basically I have to learn all about unicode
and what the locale and language mean/affect/control.
I know, I'm just a US-centric (maybe even ascii-centric) old fogey.
And for some reason you don't want to make it easy for me remain so.

 > <http://clisp.cons.org/impnotes/printer.html#multi-rep>

The lesson I have learned (over and over - sort of like claiming to be
really good at quitting smoking cause you've done it so many times!) 
is that I should turn off *print-pretty* before I start.
I really wish the default were different.

 > <http://sourceforge.net/tracker/index.php?func=detail&aid=731952&group_id=1355&atid=351355>

I'm glad to see it.  I'd like to vote for it.
I realize that the vote that would really count is a patch.

---

From: Sam Steingold <[email-masked]>
Date: Tue Mar 16 12:52:00 2004

> * Don Cohen <[email-masked]> [2004-03-16 11:30:03 -0800]:
>

the status of a character is determined by its Unicode properties.
what is your question, specifically?
(and what is "syntax class"?)

> And for some reason you don't want to make it easy for me remain so.

_I_ am an ASCII bigot.
I think that l12n and i18n are wrong.
People who want to use a computer should
1. learn English
2. pass a test, similar to a driving test to get a driver's license.
[no, I am not _completely_ serious - but you are not paying me for
being serious, so you cannot expect me to be serious, right?]

That said, I have never had a problem with CLISP Unicode &c.
I always run "clisp -E utf-8" and I never have any problems.
if I get an encoding error, it is a welcome indication that I have bad data.

> is that I should turn off *print-pretty* before I start.  I really
> wish the default were different.

which default do you want to change?
we can have a vote.
if the majority wants a different default, it will be changed.
(not in 2.33, that's too close, but in 2.34).

> I realize that the vote that would really count is a patch.

before a vote or a patch can be discussed, we need a clear proposal.
At this time I just do not see a way to make a change which would...

1. be generally consistent with Common Lisp standard and culture

2. would be helpful to the users in any way.

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
The early worm gets caught by the bird.

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Tue Mar 16 13:36:04 2004

> (not in 2.33, that's too close, but in 2.34).

When I, as a user, see in the standard specifications such as:

    Variable *PRINT-PRETTY*
    Value Type:     a generalized boolean.
    Initial Value:  implementation-dependent.
                    ^^^^^^^^^^^^^^^^^^^^^^^^

I understand it to mean that I must put a:

   (SETQ *PRINT-PRETTY* T) 

[or NIL, YMMV] in my ~/.common.lisp file that is loaded by my
various common-lisp rc files.

When I see the same, as an implementer, I undestand that I can put:

     (SETQ *PRINT-PRETTY*  
           (< (IF (NEW-MOON-P (GET-UNIVERSAL-TIME)) 0.3 0.7) (RANDOM 1.0)))
     
in my implementation initialization.

I think it's quite futile to have users vote at one time to change the
arbitrary default implemented, whatever it  may be. Next year the user
set will have changed and  their preference too probably.  It would be
more  useful to  prepare and  distribute a  template  of .clisprc.lisp
containing assignments  to all the COMMON-LISP variables  that have an
implementation-dependant initial value.

---

From: Sam Steingold <[email-masked]>
Date: Tue Mar 16 13:51:04 2004

> * Pascal J.Bourguignon <[email-masked]> [2004-03-16 22:35:36 +0100]:
>

I was talking about clisp-specific CUSTOM:*PPRINT-FIRST-NEWLINE*, not
*PRINT-PRETTY*.
the initial value of *PRINT-PRETTY* is not up to a discussion.

> It would be more useful to prepare and distribute a template of
> .clisprc.lisp containing assignments to all the COMMON-LISP variables
> that have an implementation-dependant initial value.

(apropos "" "CUSTOM")
<http://clisp.cons.org/impnotes/customize.html>.

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
What's the difference between Apathy & Ignorance? -I don't know and don't care!

---

From: [email-masked] (Don Cohen)
Date: Tue Mar 16 14:06:08 2004

> (and what is "syntax class"?)
What I expected (from many earlier lisp implementations, which of
course did not know about unicode) was that all characters other than 
a special few would act like #\A.  So it took a while to realize that
the error message claiming to be from read was really from read!

 > if I get an encoding error, it is a welcome indication that I have bad data.
My data is not under my control in this case.  It's really bytes that
others (mostly "attackers") send to my server.  But can you blame me
for wanting to view it as 8bit characters?

 > which default do you want to change?
*print-pretty* should be nil
(as it is in every other lisp I have used)

 > 2. would be helpful to the users in any way.

I propose a new datum on an encoding that controls how newline data is
read.  At least one possible value should be that cr and lf result in 
separate characters.  I suppose, depending on the line terminator
value one of them could return #\newline and the other could be either
#\return or #\linefeed.  I'd actually like one to be #\return and the
other to be #\linefeed.

---

From: Sam Steingold <[email-masked]>
Date: Tue Mar 16 14:53:11 2004

> * Don Cohen <[email-masked]> [2004-03-16 14:00:02 -0800]:
>

yes.
bytes are bytes, characters are characters.
when attackers are trying to exploit a vulnerability (such as a buffer
overflow), what they send is bytes, not characters.

OTOH, if what is being written into the log file is actually characters
(encoded by your application in, say, utf-8), then you should read they
as such, using the encoding which the application which created the file
used.

> (as it is in every other lisp I have used)

I will leave this to Bruno.

> #\return or #\linefeed.  I'd actually like one to be #\return and the

you appear to have missed the whole point of mine.
please read my two messages referred to in the RFE above and explain
here - or in a comment to the RFE - what behavior, and _why_(!), you
consider correct in the specific cases I mention there.

specifically, is it OK for READ-LINE to return a string with an
embedded newline?  why?  why not?
how do you propose to avoid it if you do not consider to OK?

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
A computer scientist is someone who fixes things that aren't broken.

---

From: [email-masked] (Don Cohen)
Date: Tue Mar 16 16:00:00 2004

> how do you propose to avoid it if you do not consider to OK?

To me the issue is what exactly is a newline.
I don't think it has to be the same as #\return or #\linefeed.
I think that if the line terminator mode is :unix then it should
be ok for READ-LINE to return a string with an embedded #\return
and if line terminator mode is :mac then it should be ok to return
a string with #\linefeed, and if :dos then it should be ok to 
return a string with both of those (but not #\return followed
immediately by #\linefeed).

I can't tell for sure but it appears to me that currently
in clisp (eq #\linefeed #\newline).
I suggest that this not be the case, that newline, return and
linefeed be three different characters, or alternatively, 
that whether (eq #\linefeed #\newline) is true should depend
on the "current" line terminator mode.
This gets into implementation issues that I don't know about.

http://article.gmane.org/gmane.lisp.clisp.general/6970

 with-open-file (s "foo.dos" :direction :output
		  :element-type '(unsigned-byte 8))
   (write-sequence
    (mapcar #'char-code
	    '(#\f #\o #\o #\Newline #\b #\a #\r #\Return #\Newline))
   s))

 now, what should

 (with-open-file (s "foo.dos" :direction :input
		  :element-type 'character
		  :external-format :dos)
   (read-line s))

 return?

There's a problem here, which is that you write and read with
different element types and different external format.
I therefore think that lots of different answers are permissible.
I think the first form should be interpreted according to some
translation of newline and return into unsigned byte 8, and the
second should be interpreted according to to some understanding
of what the element type and external format mean for whatever
was produced by the first form.

 You might reasonably argue that the right string to return is
 "foo\fbar".  Unfortunately, "\f" (== (code-char 10)) is #\Newline in
 CLISP, so READ-LINE would return a string with an embedded newline,
 which, if not outright non-compliant, would be quite surprising to a
 user.

 Because of this problem, CLISP reads CR, LF and CRLF as #\Newline.

http://article.gmane.org/gmane.lisp.clisp.general/4718

 It has been requested on many occasions that CLISP provide an option to
probably mostly by me
 treat CR/LF/CR+LF differently on character input (right now all three
 are read as #\Newline STREAM-ELEMENT-TYPE is CHARACTER).
 The answer to these requests has been to use binary i/o.
 6 months ago it was suggested that a :LINE-TERMINATOR-STRICT-P option
 be added to the ENCODING object.

 The problem is that this feature will produce unexpected results:
 READ-LINE will return strings with embedded #\Newline!
 ANSI does not appear to forbid it.

 In CLISP, #\Newline is identical to #\Linefeed (which is specifically
 permitted by <http://www.lisp.org/HyperSpec/Body/sec_13-1-7.html>).
 Therefore, if the file is exactly this string:

 (concatenate 'string "foo" (string #\Linefeed) "bar"
	      (string #\Return) (string #\Linefeed))

 and we open it with

 (setq e (make-encoding :charset "ascii" :line-terminator :dos
			:line-terminator-strict-p t))
 (setq s (open "foo" :external-format e))

 then the string returned by (READ-LINE s) will contain an embedded
 #\Newline between "foo" and "bar" (because a single #\Linefeed is not
 a
 #\Newline in the specified encoding, it will not make READ-LINE
 return,
 but it _is_ a CLISP #\Newline!)

I'd like it to be #\return and for that not to be the same as
#\newline. 

 Therefore, files "foo" and "bar", written with

 (with-open-file (o "bar" :direction :output :external-format e)
   (with-open-file (i "foo" :external-format e)
     (write-line (read-line i) o)))

ok, so they both use e which is strict dos
So the read-line should return only one line containing a linefeed
in the middle.
Then the write-line should write only one line containing a linefeed
in the middle.  
In both cases there will be a crlf at the end.
Or am I missing something here?

 will be different:

 ---- foo ----
 foo^Jbar
 -------------
 ---- bar ----
 foo
 bar
 -------------

 We already have this behavior (unless the ENCODING's LINE-TERMINATOR
 is
 :UNIX), the point here is that :LINE-TERMINATOR-STRICT-P does _not_
 fix
 this.

 Is anyone still interested in this :LINE-TERMINATOR-STRICT-P feature?
 Do you see any problems with the behavior I just described?

---

From: Bruno Haible <[email-masked]>
Date: Wed Mar 17 04:33:48 2004

> > (as it is in every other lisp I have used)

You have enough customizable variables, from *print-pretty* to the
symbols in the CUSTOM package, that you can set in your .clisprc.lisp file.

The default is set so that clisp becomes most useful to newbies, without
customization. If all other Lisps look frightening to newbies, that's
not a reason why clisp should look the same.

Bruno

---

From: Bruno Haible <[email-masked]>
Date: Wed Mar 17 04:43:45 2004

Don Cohen wrote:
> separate characters.

This muddles up the distinction between bytes and characters, thereby
creating portability problems.

> immediately by #\linefeed).

This is the way it's done in Java. And it's shitty: although the standard
way to designate a string with an embedded newline is "foo\n", when you
write a string to a file stream, you find out that
    stream.print("foo\n");
does not the same thing as
    stream.println("foo");

Even Microsoft, since the 1980's, has in its C library a hack in the
fgetc() function that converts 0x0D 0x0A to "\n". Why? Because different
external representations of some thing on different systems must not
lead to different internal (in-memory) representations of it, otherwise
programs become unportable.

Bruno

---

From: Bruno Haible <[email-masked]>
Date: Wed Mar 17 12:16:07 2004

Don Cohen wrote:
> it seems inevitable that if you try to map
> multiple file representations to the same memory representation then
> the reverse mapping will (and should) be one to many.  The best you
> can do is provide some way to control both the input and output
> mappings.

Yes. And in clisp this mapping is the ENCODING with its :LINE-TERMINATOR
accessor. It allows you to accomodate different external representations
for the same in-memory representation of #\Newline = (code-char 10).

What you wanted in the last posting, was *different* in-memory representations
of #\Newline. Which leads to unportable programs. This way, you would
write Lisp programs that are less portable between Unix and Windows than
the equivalent C programs!

> Clearly my current solution is exactly what I want.
> I read 8 bit bytes and then call code-char, using an encoding that
> gives me 256 different characters for the 256 bytes.

Such encodings are disappearing rapidly from the landscape. The use of
code-char and char-code makes your program unable to support even the
standard repertoire of characters required for English (because of the
curly quotes...).

> It just seems silly that this should be the best (maybe only) way
> to get the desired result.

You missed the functions CONVERT-STRING-TO-BYTES and CONVERT-BYTES-TO-STRING.

Bruno

---

From: Sam Steingold <[email-masked]>
Date: Wed Mar 17 12:31:03 2004

> * Bruno Haible <[email-masked]> [2004-03-17 21:09:30 +0100]:
>
> What you wanted in the last posting, was *different* in-memory
> representations of #\Newline. Which leads to unportable programs.

why?
because (format nil "~%") will depend on the current value of some
global encoding variable?

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Winword 6.0 UNinstall: Not enough disk space to uninstall WinWord

---

From: [email-masked] (Don Cohen)
Date: Wed Mar 17 13:02:14 2004

Bruno Haible writes:
 > representations

I don't know about the equivalent c programs.
I also don't know which programs you view as more or less portable,
but my view is that you should be able to write the program you
want, and preferably in the most straight forward way.
I suppose encodings default to different line terminator modes on
different OS's.  In that case, if you use the default then you
get (what I consider to be) different behavior on different OS's, 
but I guess that's what you want.
If you specify the mode then you get the same behavior on different
OS's.
I argue that if you want the same behavior on different OS's then
you should not use #\newline but either #\return or #\linefeed.
There are plenty of other things that differ from one platform to
another, such as *features* - the differences are viewed by some
people as making it easier to write portable programs, even though
on a small scale they lead to different behavior on different
platforms.

 > Such encodings are disappearing rapidly from the landscape. The use of

Does this mean that the encoding I now use is likely to disappear?

 > standard repertoire of characters required for English (because of the
I don't know what you you consider the standard repertoire.
(What are the curly quotes?)
And why does char-code not work on them?

 > You missed the functions CONVERT-STRING-TO-BYTES and
(You mean STRING-FROM-BYTES)
Yes I did miss those.
But I don't think they do what I need.
They might if they allowed the result to be put into an arbitrary
position of an existing vector.  Right now I do
   (vector-push-extend (code-char (aref *byte-io-vector* i)) ...)

So these depend on encodings but code-char and char-code do not?
I'd appreciate an explanation of that.
It must be related to the curly quote question above.

---

From: Bruno Haible <[email-masked]>
Date: Wed Mar 17 13:34:01 2004

> global encoding variable?

Because

  1) So many operations will give subtly different results, starting
     from (length string), (map 'list string), the hash code of a string
     and thus also the order of traversal of a hash table containing
     strings as keys, up to all kinds of string manipulation functions
     that work by scanning a string.

  2) The developer tests his program in one mode and not in the other.
     That was Don Cohen's point about *print-pretty*: one more variable
     that can be switched on or off, means more testing.

In short, in application programming (as opposed to low level system
programming) system dependencies are best handled at the border between
the program and the outside world, and the program logic is kept free of
system dependencies.

Bruno

---

From: Bruno Haible <[email-masked]>
Date: Wed Mar 17 13:49:05 2004

> And why does char-code not work on them?

Oh, char-code works fine on them. It's only when you attempt to feed
this char-code into a byte stream that expects values between 0 and 255
that you will get a nice error.

> (write-string "“‘Ha ha’ said the clown” from Manfred Mann")
“‘Ha ha’ said the clown” from Manfred Mann
"“‘Ha ha’ said the clown” from Manfred Mann"
> (map 'list #'char-code *)
(8220 8216 72 97 32 104 97 8217 32 115 97 105 100 32 116 104 101 32 99 108 111
 119 110 8221 32 102 114 111 109 32 77 97 110 102 114 101 100 32 77 97 110 110)

> position of an existing vector.

SETF SUBSEQ will copy a vector into an existing one.

> So these depend on encodings but code-char and char-code do not?

Yes, CONVERT-STRING-TO/FROM-BYTES work with any character and any encoding.

Whereas the assumption of the 1980ies, that each character is a byte,
works only for half the encodings of the world and for less than 1/10th
of the characters of the world.

Bruno

---

From: Sam Steingold <[email-masked]>
Date: Wed Mar 17 13:51:10 2004

> * Bruno Haible <[email-masked]> [2004-03-17 22:28:14 +0100]:
>
> Sam Steingold wrote:

actually, no, the value of (format nil "~%") is just (string #\Newline)
independent on any global state.

>   that work by scanning a string.

why?
the only change is that (char-code #\Newline) is not 10 anymore.

when you read from a file with the an encoding with a
:LINE-TERMINATOR-STRICT-P NIL (the current behavior), you will have
only #\Newline, no #\Linefeed or #\Return.
When :LINE-TERMINATOR-STRICT-P is T, you will get #\Newline for the
appropriate line terminator and the original control character
otherwise.
(length (format nil "foo~%bar")) is always 7.

>   that can be switched on or off, means more testing.

again, what is that mode?

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
MS DOS: Keyboard not found. Press F1 to continue.

---

From: Sam Steingold <[email-masked]>
Date: Wed Mar 17 14:09:05 2004

> * Bruno Haible <[email-masked]> [2004-03-17 22:43:01 +0100]:
>

I think he wants to re-use the vector (a la READ-SEQUENCE).
CONVERT-STRING-TO-BYTES will always allocate a fresh vector.
Maybe an :OUTPUT + :START-OUTPUT combination is needed:

(EXT:CONVERT-STRING-FROM-BYTES byte-vector encoding
                               :START 10 :END 33
                               :OUTPUT string :START-OUTPUT 17)

will write characters into STRING starting at position 17.
(and using ADJUST-ARRAY if STRING is shorter than 40)

> works only for half the encodings of the world and for less than
> 1/10th of the characters of the world.

still, each character is an integer, right?
(even 21 bit integer!)

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Time would have been the best Teacher, if it did not kill all its students.

---

From: [email-masked] (Don Cohen)
Date: Wed Mar 17 15:24:04 2004

>      that work by scanning a string.

If I use the encoding that generates separate #\cr and #\lf
and maps one-to-one between bytes 0-255 and characters then 
I get the same result on any system.  (Assuming I don't write
#\newline, but stick to #\cr and #\lf.)
It is when I use the current default which changes from one system
to another that I get different results for different systems.

I should use #\newline and the current encoding stuff when I want
different output on different systems.

 > > (map 'list #'char-code *)

Interesting, when I copy your string I get
(map 'list 'char-code "“‘Ha ha’ said the clown” from Manfred Mann")
(226 128 156 226 128 152 72 97 32 104 97 226 128 153 32 115 97 105 100 32 116
 104 101 32 99 108 111 119 110 226 128 157 32 102 114 111 109 32 77 97 110 102
 114 101 100 32 77 97 110 110)

 > SETF SUBSEQ will copy a vector into an existing one.

Can I get that to also do vector-push-extend ?

 > of the characters of the world.

All I need is one encoding that covers bytes 0-255 and allows me to
read each byte as one character.  I don't care whether the one I use
represents 100% or 1e-9% of the total.

 > (and using ADJUST-ARRAY if STRING is shorter than 40)
Ah, with the adjust yet!!

How code-char/char-code get away without an encoding:
I gather the point is that the encoding relates bytes on a file to 
characters, and code-char/char-code relates characters to integers, 
which evidently do NOT have to be the same integers that you would 
get if you read or wrote those bytes to/from a file!
I assume that (= x (char-code (code-char x))) for x in the appropriate
range.  I've also been assuming that the ascii characters correspond
to the "right" codes.  That's probably all that matters to me so far.

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Wed Mar 17 16:53:01 2004

Don Cohen writes:
> and maps one-to-one between bytes 0-255 and characters then=20

Imagine  you copy  your lisp  program  to three  different systems:  a
Macintosh system, a MS-Windows system and a unix system, and that with
you same lisp  program on these three systems, you try  to read a TEXT
file named example.txt transfered thru  FTP as a text file between the
three systems.

In clear, on the Macintosh you'll have in the file: "texttext[CR]texttext[CR]"
on the unix system you'll have "texttext[LF]texttext[LF]"
and on MS-Windows you'll have  "texttext[CR][LF]texttext[CR][LF]".

Then with your scheme your three identical copies of your program wont
read the same data of the same text file.

> I gather the point is that the encoding relates bytes on a file to=20
> characters, and code-char/char-code relates characters to integers,=20
> which evidently do NOT have to be the same integers that you would=20
> I assume that (=3D x (char-code (code-char x))) for x in the appropriat=
> e

You're assuming too much. The  "ASCII" characters have not the "right"
codes on an EBCDIC system.  And EBCDIC systems are far from dead, they
even do web CGI on CICS...

---

From: [email-masked] (Don Cohen)
Date: Wed Mar 17 17:37:46 2004

> file named example.txt transfered thru  FTP as a text file between the
Right, because ftp happens to be changing the file contents.
If, on the other hand, I use scp, or ftp in binary mode,
then I get the same results with the program that I propose, which
I consider to be system indedpendent, and different results with 
the program that I must write in clisp now, which I regard as
system dependent.  I don't mind that I *can* write a program that
is system dependent.  I just want to be *able* to write one that is
system independent.

 > even do web CGI on CICS...
Ok, so if I want to use ebcdic then I should use some other encoding.
As it turns out, the Internet protocols pretty much stick to ascii,
so that's what I mostly want to use.

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Wed Mar 17 17:56:05 2004

> If, on the other hand, I use scp, or ftp in binary mode,

But once  you copy your files  in BINARY mode, they're  no longer TEXT
files. They are  now binary file, and you  cannot read CHARACTERS from
them only BYTES.   If you want to have the exact  same behavior on the
three  systems  BYTE-FOR-BYTE, you  must  read  and  write BYTES,  not
LINES of CHARACTERS.

> is system dependent.  I just want to be *able* to write one that is

Then  don't   use  CHARACTERS  and   CHAR-CODE/CODE-CHAR  since  these
functions  ARE IMPLEMENTATION  DEPENDENT!   Which is  even worse  than
system dependent,  since different implementations on  the same system
can  have  different  idea  of  what  CHAR-CODE  or  CODE-CHAR  should
return.  Even the  SAME implementation  can be  compiled  with options
giving different  results, such  as clisp compiled  in 8-bit  chars or
with unicode support!

 system independent       <=> BYTE, READ-BYTE, WRITE-BYTE
 implementation dependent <=> CHAR, READ-CHAR, WRITE-CHAR

> so that's what I mostly want to use.

My  point,  and  what you  don't  understand,  is  that when  you  are
expecting TEXT data, your program could be running on an EBCDIC system
and receive  URLs and  HTML textual data  in EBCDIC!  Of  course, what
runs  on the wire  is always  ASCII, but  what code  you find  in core
memory  can be  anything the  system likes  to have.   Read  again the
COMMON-LISP standard
http://www.lispworks.com/reference/HyperSpec/Body/02_ac.htm
and you'll see that the only thing that is prescribed is a minimum set
of characters,  but that  the corresponding codes  can be  anything as
long as some constaints are respected:
http://www.lispworks.com/reference/HyperSpec/Body/13_af.htm

You should really study the whole chapter 2 and chapter 13 of CLHS.

---

From: [email-masked] (Don Cohen)
Date: Wed Mar 17 18:52:01 2004

> three  systems  BYTE-FOR-BYTE, you  must  read  and  write BYTES,  not

What determines whether they are binary or text files in the first
place?  If I create them emacs are they text?
If I copy them with scp do they remain text?
If I name them .txt does that make them text?
I have to say that I don't see any well defined boundary.
I'm not so sure I really see any boundary at all.
Do you think that a file could possibly contain
part text and part binary data?

 > functions  ARE IMPLEMENTATION  DEPENDENT!
So you think that when I look at some RFC that talks about the
"US-ASCII coded character set" that I should be reading in binary.
You think that when it talks about "GET" this should be viewed as
a sequence of bytes.  I should not use the character string "GET"
in a lisp program to find out what type of http request is arriving.

I think these are characters, and a programming language should be
able to deliver them to me as characters.  If I use an ebcdic machine
then perhaps I should use some non-default encoding to do that.  There
is, after all, an ascii character set that is supported by clisp.  I
don't know what it does with bytes>127, but I suggest that it would be
useful to support a character set that agrees with ascii up to 127 and
assigns some (I don't really care which) characters to 128-255.  I
believe that whatever character set I now use does satisfy these
requirements.  I should then be able to use that character set to read
whatever bytes are in my file, whether or not you happen to view it as
a "text" file.

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Wed Mar 17 19:07:00 2004

Don Cohen writes:
> place? 

It depends on the operating system you're using.

On unix, it's entirely up to the user and the library and applications
you're using.

On  MS-DOS  and  Macintosh,   it's  the  applications  that  tell  the
OS/libraries how  to consider the file  at open time. (The  "b" in the
mode argument of fopen).

On other operating  system the file system knows  exactly what kind of
file it stores and how they are structured.

> If I create them emacs are they text?

You should ask to users of emacs  on VMS.  But I guess that most files
created with emacs are indeed text files.

> If I name them .txt does that make them text?

That would be irrelevant on the OS I know.

> I have to say that I don't see any well defined boundary.

Because  you seem  to know  only  operating systems  where there's  no
boundary.  Common-Lisp is  designed  to  be able  to  run portably  on
operating systems that make  the distinction between text files, fixed
record  binary files,  variable  record binary  files, indexed  files,
sequential access files, etc.

> part text and part binary data?

Not from the point of view of Common-Lisp, and the OS that I know that
make the distinction  between text and binary. 

But  why do  you  think clisp  developer  took the  time to  implement
EXT:CONVERT-STRING-FROM-BYTES  and  EXT:CONVERT-STRING-TO-BYTES?  They
are useful to embed text into a binary file.

And note that these OS are not only legacy OS.  PalmOS for example has
only  record-structured files,  and text  can be  stored one  line per
record (no CR/LF problem there!).

> "US-ASCII coded character set" that I should be reading in binary.

Indeed, if you  want to manipulate the packets  or the streams defined
by the Internet  RFC, you'd better do  it in binary.  If you  do it in
text,  on unix  or on  macintosh you'd  have problems  with  CRLF line
terminations that are mandatory in Internet protocols.

> in a lisp program to find out what type of http request is arriving.

What you get  for "GET" depends on the encoding of  the source of your
lisp program!

> is, after all, an ascii character set that is supported by clisp.  

By clisp but NOT by COMMON-LISP!  
That's where the portability enters the scene.

> I don't know what it does with bytes>127, but I suggest that it
> would be useful to support a character set that agrees with ascii up
> to 127 and assigns some (I don't really care which) characters to
> 128-255.  I believe that whatever character set I now use does
> satisfy these requirements.  I should then be able to use that
> character set to read whatever bytes are in my file, whether or not
> you happen to view it as a "text" file.

---

From: zera holladay <[email-masked]>
Date: Wed Mar 17 19:20:03 2004

> part text and part binary data?

You're right there is no boundry, a file is just a
bunch of ones and zeros on your hard disk, a byte is
really just a byte is a byte.  What you you do with
the data is what makes the difference, ie allow hex
0x41 to represent what humans call "A".

-zh

__________________________________
Do you Yahoo!?
Yahoo! Mail - More reliable, more storage, less spam
http://mail.yahoo.com

---

From: [email-masked] (Don Cohen)
Date: Wed Mar 17 23:07:05 2004

> symbols in the CUSTOM package, that you can set in your .clisprc.lisp file.

Init files are really not adequate for things that affect programs
that are run by different people on different machines.
You might imagine that *print-pretty* should not affect programs, but 
I've found many times that this is just not true.  The last case
(provoking a recent message to this list) was a program that was
trying to manage the space on a screen.  I wanted to print something
on the top line of the screen.  I position the cursor to the top of
the screen and do (princ "...") and the text comes out on the second
line!
I think the only solution is that this program should include in its
code (setf *print-pretty* nil).  Considering the amount of time I've
spent tracking down such problems it seems clear that I really ought 
to do that in every lisp program I write.
The unicode stuff is similar.  I've started to put in my batch files
 export LANG=en_US
but I now begin to think that this ought to be replaced by another 
form in every lisp program I write.
I guess this means that my web server only runs in English.
At least for clisp.

 > not a reason why clisp should look the same.

I think I've heard this argument before.
There's certainly room for argument about what's useful or frightening
to newbies.  (They probably know better what frightens them than what
is useful or good for them.) 
In any case, I gave up long ago on this one.  I just have to keep
learning to set it.  I should perhaps thank you for making me fix
my programs to not rely on this implementation dependent value.
(Is there a list of other global variables with implementation
dependent initial values?)

 > does not the same thing as

Knowing practically nothing about java, I will not attempt to explain 
or defend it.  However, it seems inevitable that if you try to map
multiple file representations to the same memory representation then
the reverse mapping will (and should) be one to many.  The best you
can do is provide some way to control both the input and output
mappings.  That's what I want to do.  If you like the current behavior
then let that be the default.

---

From: Bruno Haible <[email-masked]>
Date: Thu Mar 18 04:50:16 2004

> (and using ADJUST-ARRAY if STRING is shorter than 40)

While it's theoretically possible to add options like this, you will note
that functions which do complex operations _and_ store the result
destructively somewhere are rare in Lisp. The only one that comes to mind
are MAP-INTO and the BIT array operations. There is no RPLACBOTH, no
NSUBSEQ, no NCONCATENATE, etc. (The reason is, of course, that destructive
operations kill the functional programming style.)

Therefore I would add such options only after an investigation would show
that the garbage collection overhead due to the strings created by
EXT:CONVERT-STRING-FROM-BYTES is not bearable.

> still, each character is an integer, right?

Still, for creating portable programs, you best view a character just as
the contents of a string of length 1, and forget about the mapping to
integer.

Bruno

---

From: Sam Steingold <[email-masked]>
Date: Thu Mar 18 05:44:09 2004

> * Pascal J.Bourguignon <[email-masked]> [2004-03-18 01:53:52 +0100]:
>

yes it will.

(with-open-file (i "foo" :direction :input)
  (list (read-line i) (read-line i)))

will return ("texttext" texttext") on each platform, both now and with
the proposed separation of #\Newline from #\Linefeed.

even when these 3 different files are read on the same platform, this
list ("texttext" texttext") will be read by this form:

(with-open-file (i "foo" :direction :input
                 :external-format (make-encoding :line-terminator-strict-p nil))
  (list (read-line i) (read-line i)))

or by using the right form for each file:

(with-open-file (i "foo" :direction :input :external-format :dos)
  (list (read-line i) (read-line i)))

...

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
I'd give my right arm to be ambidextrous.

---

From: Sam Steingold <[email-masked]>
Date: Thu Mar 18 05:49:23 2004

> * Bruno Haible <[email-masked]> [2004-03-18 13:44:10 +0100]:
>
> Sam Steingold wrote:
> While it's theoretically possible to add options like this, you will
> note that functions which do complex operations _and_ store the result
> destructively somewhere are rare in Lisp. The only one that comes to
> mind are MAP-INTO and the BIT array operations. There is no RPLACBOTH,
> no NSUBSEQ, no NCONCATENATE, etc. (The reason is, of course, that
> destructive operations kill the functional programming style.)

there are READ-SEQUENCE and REPLACE.
I think that CONVERT-STRING-FROM-BYTES _is_ a kind of READ-SEQUENCE,
mentally implemented as (READ-SEQUENCE (make-array) (bytes->stream))

> Therefore I would add such options only after an investigation would
> show that the garbage collection overhead due to the strings created
> by EXT:CONVERT-STRING-FROM-BYTES is not bearable.

it depends on the pattern of usage.
if Don ends up using CONVERT-STRING-FROM-BYTES instead of
READ-SEQUENCE exclusively, then the semantics should be similar.

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Bus error -- driver executed.

---

From: Bruno Haible <[email-masked]>
Date: Thu Mar 18 05:59:15 2004

> it depends on the pattern of usage.

OK, then show me one usage, together with benchmark results, where the GC
does not do its job sufficiently well.

Bruno

---

From: Bruno Haible <[email-masked]>
Date: Thu Mar 18 06:19:03 2004

> the only change is that (char-code #\Newline) is not 10 anymore.

This alone creates portability problems. Around 16 years ago, the conventions
on the Mac were just the converse than on Unix: \n was 0x0D and \r was 0x0A.
And _of course_ when you ported programs from a Mac to Unix you had to
change \r into \n in a lot of places.

> appropriate line terminator and the original control character

What Don proposed, was that on Windows, a line ends with two whitespace
characters, not just one #\Newline. Here is a good example for some innocent
looking code that is broken by such a change:

(let ((s (make-string-input-stream "xy
z"))) (progn (read s) (read-char s)))

would return #\z on Unix and #\LF in Don Cohen's "Windows faithful" mode.
There are lots of examples like this.

Bruno

---

From: [email-masked] (Don Cohen)
Date: Thu Mar 18 09:20:03 2004

> change \r into \n in a lot of places.

Clearly the current implementation choice is useful when you
want to write a text file that will be compatible with the other 
tools on whatever system you're using, and also when you want to
read it back in on any other system.
I don't propose that you get rid of it.

 > There are lots of examples like this.

All of those examples also apply to binary IO which is what I'm 
forced to do now.  I'm not expecting the new faithful mode to
hide/solve these differences.  I just want to deal with characters
instead of bytes.  In fact, I currently do deal with characters.  
I read bytes and convert them to characters.

Let's just consider the question of what's the "right" way to 
deal with input when you write a web server in Clisp.
I see three possible positions so far.
- use only bytes (I think this is Pascal's position)
- read bytes and convert them to characters, then use characters
  internally - this is what I do now
- read characters in "faithful" mode
  This is what I think is best.

As a separate matter, we might consider what the "right" way would
be to do such IO in portable common lisp.  I think Pascal's argument 
is that the spec doesn't even promise that there will 256 characters
so it's pretty much hopeless to do anything other than stick to
bytes.

This issue does not really apply to my server.  It became clear long
ago that this particular program was not going to be possible to write
in portable common lisp, which does not even support network sockets,
let alone determining how much output can be written without blocking
(which would be replaced by multiple threads in an implementation
supporting that).

---

From: Bruno Haible <[email-masked]>
Date: Thu Mar 18 10:16:14 2004

>   This is what I think is best.

I think for a web server the second option is best, for two reasons:

  - The HTTP protocol is defined in terms of bytes. The byte stream of
    an HTTP connection can contain binary data as well, for example after
    a POST request some binary data can be sent, IIRC.

  - For security reasons, you may want to control explicitly all I/O
    conversions. Which is hard if you use the Lisp implementation's
    READ-CHAR / READ-LINE black box.

Of course you can use READ-CHAR and READ-LINE on portions of the stream,
if you use a Gray stream for converting READ-CHAR calls into
READ-BYTE calls.

> so it's pretty much hopeless to do anything other than stick to

Yes, ANSI CL promises 96 characters, not more.

And I don't remember why CLISP's interpretation of ISO-8859-1 contains the
control characters 0x80..0x9F. The standards are not clear on this issue.
ISO-8859-1 doesn't contain them, but ISO-8859-1 is usually viewed as the
8-bit portion of Unicode, and Unicode has control characters at 0x80..0x9F.
Probably I included this range only so that Jörg could use the character
#\U009B on Amiga... So: don't count on it. The details of I/O conversion
in READ-CHAR are up to the implementation.

Bruno

---

From: Sam Steingold <[email-masked]>
Date: Thu Mar 18 10:53:05 2004

> * Bruno Haible <[email-masked]> [2004-03-18 14:52:11 +0100]:
>
> Sam Steingold wrote:

(ext:times
 (with-open-file (in "foo" :element-type 'unsigned-byte)
   (let ((bytes 0) (chars 0)
         (buf (make-array 1024 :element-type 'unsigned-byte)))
     (loop (let ((got (read-sequence buf in)))
             (incf bytes got)
             (incf chars (length (ext:convert-string-from-bytes
                                  buf charset:utf-8)))
             (unless (= got (length buf))
               (return (values bytes chars))))))))

                                            Permanent            Temporary
Class                                  instances   bytes    instances   bytes
-----                                  --------- ---------  --------- ---------
SIMPLE-STRING                                  0         0     435575 265098784
EXT:SIMPLE-8BIT-VECTOR                         1        48      62218  64210044
CONS                                           0         0    2862903  22903224
SYMBOL                                         0         0     124446   3484488
SIMPLE-VECTOR                                  6       272     186748   3241416
BIGNUM                                         2        24      91662   1099944
FUNCTION                                       0         0         42      1668
SYSTEM::ANODE                                  0         0         24       576
SYSTEM::FNODE                                  0         0          3       336
SIMPLE-BIT-VECTOR                              0         0         29       276
SYSTEM::VAR                                    0         0          4       272
HASH-TABLE                                     1        56          3       168
STREAM                                         0         0          3       192
STRING-STREAM                                  0         0          3       180
VECTOR                                         0         0          6       168
FILE-STREAM                                    0         0          1       144
STRING                                         0         0          6       144
SYSTEM::CONST                                  0         0          5       120
PATHNAME                                       0         0          5       100
STANDARD-GENERIC-FUNCTION                      0         0          2        40
BLOCK                                          0         0          1        36
-----                                  --------- ---------  --------- ---------
Total                                         10       400    3763689 360042320

Real time: 51.568 sec.
Run time: 41.54 sec.
Space: 360900936 Bytes
GC: 606, GC time: 15.705 sec.
63707377 ;
63708160

note that over 35% of time is spent on GC!

BTW, why do I get more chars than bytes as the return value?
this is a pure ASCII file!

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Incorrect time syncronization.

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Thu Mar 18 12:01:03 2004

> change \r into \n in a lot of places.

This  is  not  a portability  problem.   This  is  a good  thing  that
assumptions that you should not make be questionned from time to time.

I'd really  like to  see a Common-Lisp  implementation that  would use
random values for all that is implementation dependent. A program that
would run conformably and bug  free on such an implementation would be
really  portable,  so this  kind  of  implementation  would be  really
valuable to application developers.

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Thu Mar 18 12:41:06 2004

Bruno Haible writes:
> Probably I included this range only so that J=F6rg could use the character

You'd  have to check  the official  standard.  There's  some ambiguity
concerning  0x80..0x9f.  Most "ISO  8859" glose  exclude them,  but on
wikipedia,  after giving  a first  table for  "ISO 8859"  with control
ranges marked as  "unused", they note that IANA  defined an "ISO-8859"
(not (string-equal  "ISO 8859"  "ISO-8859")) where the  control ranges
are filled with ASCII control codes and 32 additionnal control codes.

http://en.wikipedia.org/wiki/Latin-1

Note  that  they  call  them  "control  character",  but  I  take  the
precaution to call  them "control codes", not characters.   But on the
other hand, Common-Lisp has characters such as #\newline.  

The control codes  are useful only on the wire,  and generaly can't be
converted  to  other  character  encoding, while  the  characters  are
abstracted and independent  from any encoding, which allow  them to be
encoded with different encoding.

IMO, the undestanding of Common-Lisp  is that a stream that comes from
the wire should be read in binary, codes should be handled as byte and
codes corresponding to characters should be decoded to characters. The
structuring implied by the control codes should traslate to high level
structuring  (objects,  structures, array,  lists).   Clearly this  is
quite different from the point of view of unix/posix.

It's really a shame that ISO standard documents not be more available,
they make a lot of people lose a lot of time on these questions...

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Thu Mar 18 12:52:04 2004

> this is a pure ASCII file!

read-sequence does not set the fill pointer of the array, the more so
if it has no fill pointer!

[71]> (let ((a (make-array '(64) :element-type 'unsigned-byte :initial-element 0))) (with-open-stream (in (make-string-input-stream "BONJOUR")) (values a (read-sequence a in))))
#(#\B #\O #\N #\J #\O #\U #\R 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0) ;

---

From: Sam Steingold <[email-masked]>
Date: Thu Mar 18 13:08:02 2004

> * Pascal J.Bourguignon <[email-masked]> [2004-03-18 21:02:13 +0100]:
>

How about a CLISP option "-r" which would do just that - assign random
valid values to the implementation dependent variables :-)

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Software is like sex: it's better when it's free.

---

From: Pascal J.Bourguignon <[email-masked]>
Date: Thu Mar 18 13:26:02 2004

> valid values to the implementation dependent variables :-)

If you fancy it.  I would definitely use it along with -ansi!

---

From: Bruno Haible <[email-masked]>
Date: Thu Mar 18 13:34:06 2004

> GC: 606, GC time: 15.705 sec.

What kind of system is this on, and what kind of GC does it have?
(GENERATIONAL_GC or not? SPVW_BLOCKS or SPVW_PAGES?)

On Linux/x86 I get (where "foo" is produced as 10 times the concatenation
of all clisp/src/*.d files):

(defun foo (filename)
 (ext:times
  (with-open-file (in filename :element-type 'unsigned-byte)
   (let ((bytes 0) (chars 0)
         (buf (make-array 1024 :element-type 'unsigned-byte)))
     (loop (let ((got (read-sequence buf in)))
             (incf bytes got)
             (incf chars (length (ext:convert-string-from-bytes
                                  buf charset:utf-8)))
             (unless (= got (length buf))
               (return (values bytes chars)))))))))

(foo "in10")
                                            dauerhaft             temporär
Klasse                                 Instanzen   Bytes    Instanzen   Bytes
------                                 --------- ---------  --------- ---------
SIMPLE-STRING                                  0         0      62236 255273008
SIMPLE-8BIT-VECTOR                             0         0      62226  64220304
CONS                                           0         0    1617865  12942920
BIGNUM                                         2        24      91675   1100100
SIMPLE-VECTOR                                  0         0          1      4104
FILE-STREAM                                    0         0          1       148
PATHNAME                                       0         0          4        80
SIMPLE-BIT-VECTOR                              0         0          1        12
------                                 --------- ---------  --------- ---------
Gesamt                                         2        24    1834009 333540676

Real time: 18.288704 sec.
Run time: 18.15 sec.
Space: 334408540 Bytes
GC: 640, GC time: 0.76 sec.
63717750 ;
63693701

So, it's a very similar test to yours; the amount of GCs is nearly the same;
but here only 4% of the time is spent in GC.

> this is a pure ASCII file!

It may be an ASCII file containing CR/LFs...

Bruno

---

From: Bruno Haible <[email-masked]>
Date: Thu Mar 18 13:46:05 2004

Pascal J.Bourguignon wrote:
> You'd  have to check  the official  standard....

The final draft of ISO-8859-15 and -16 is online at
http://www.evertype.com/sc2wg3.html
but it doesn't answer the question about #x80..#x9F.

Bruno

---

From: Sam Steingold <[email-masked]>
Date: Thu Mar 18 14:07:02 2004

> * Bruno Haible <[email-masked]> [2004-03-18 22:28:31 +0100]:
>
> Sam wrote:
>> (ext:times

how do I find this out from a running CLISP?
right now, I had to look at *lib-directory*, "make lispbibl.h" there
and grep through it.  this is not too reliable.
there gotta be a better way!

> (GENERATIONAL_GC or not? SPVW_BLOCKS or SPVW_PAGES?)

no GENERATIONAL_GC!!!  why?!
I do have libsigsegv:

$ grep -i sigsegv config.log
  $ configure --srcdir=../src --with-module=syscalls --with-module=regexp --with
-module=dirkey --with-module=bindings/win32 --with-libsigsegv-prefix=/usr/local/
libsigsegv-cygwin --cache-file=config.cache --no-create --no-recursion
configure:6897: checking for libsigsegv
configure:6964: checking how to link with libsigsegv
configure:6966: result: /usr/local/libsigsegv-cygwin/lib/libsigsegv.a
configure:22449: gcc -o conftest.exe -g -O2 -I/usr/local/libsigsegv-cygwin/inclu
de  conftest.c -lreadline -lncurses  >&5
cl_cv_lib_sigsegv=yes
CPPFLAGS='-I/usr/local/libsigsegv-cygwin/include'
LIBSIGSEGV='/usr/local/libsigsegv-cygwin/lib/libsigsegv.a'
LTLIBSIGSEGV='-L/usr/local/libsigsegv-cygwin/lib -lsigsegv'
#define HAVE_SIGSEGV 1
$ grep SPVW lispbibl.h
#define SPVW_MIXED
#define SPVW_PAGES
$

> GC: 640, GC time: 0.76 sec.

OK, so generational GC does save the day, but even these 4% are not
really needed!

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Lisp is a way of life.  C is a way of death.