Subject: Re: [clisp-list] internationalization problem on clisp-2.27

From: Bernard Urban <[email-masked]>
Date: Tue Oct 2 02:39:02 2001

>>> But in my case (Debian system), the fr locale prexisted 
>>> (in /usr/share/locale) for some GNU utilities, 
>>> so it should have been found ?
>>
>>The locale is in /usr/lib/locale, and in glibc-2.2 there is no "fr"
>>locale. (What kind of monetary information would it contain? French or
>>Swiss or Luxemburgish Franc?)
>>
>>What you see in /usr/share/locale are the translation catalogs, where
>>a catalog named "fr" acts as fallback for all locales starting in
>>"fr_": "fr_FR", "fr_CH" etc.
>>
>>Bruno

But shouldn't clisp installation warn the user
if there is no locale for the proposed messages translation ?

And now another bug related to internationalization:

When there is a pathname in $HOME with an accented letter
(in coding iso-8859-1), clisp 2.27 gives the message:

*** - invalid byte #x81 in CHARSET:ASCII conversion

It is related to looking for $HOME/.clisprc, as the message disappears
when -norc option is given. It disappears also with option
-Epathname iso-8859-1.

Regards.

-- 

Bernard Urban

---

From: Peter Wood <[email-masked]>
Date: Tue Oct 2 06:41:07 2001

> when -norc option is given. It disappears also with option

Hi,

By default Clisp uses charset::ascii, with the result that you
observe.  You can cure this by dumping an image after setting
*terminal-encoding* *default-file-encoding* and *misc-encoding* to
charset:iso-8859-1.

(let ((encoding (make-encoding :charset charset:iso-8859-1 :line-terminator :unix)))
  (setf *terminal-encoding* encoding
        *default-file-encoding* encoding
        *misc-encoding* encoding))

Replace your default image with the one you dumped.

I don't think it is a bug, as Clisp has to choose *some* default, and
ascii is probably the best bet.

Regards,
Peter
 
Ps. If you don't use unix, you obviously don't want the
:line-terminator bit (above).

---

From: Bernard Urban <[email-masked]>
Date: Tue Oct 2 07:01:09 2001

> ascii is probably the best bet.

First, clisp behaviour was ok, then I inadvertently created a file in
the HOME directory with an european accented letter in the name, and
clisp broke, even if clisp does not use at all this new file !
I do not think that it is normal for any software to change its
behaviour when an unrelated file is created.

So thank you for the advice, but for me this is a bug, which obviously
is triggered when clisp searches for $HOME/.clisprc by looking at all
filenames in $HOME.

Regards.

-- 

Bernard Urban

---

From: Dave Pearson <[email-masked]>
Date: Tue Oct 9 23:42:10 2001

[I might be slightly missing the point here so please excuse any incorrect
terminology]

> ascii is probably the best bet.

Is there a reason why a default that allows characters 0-255 can't be used?
I'm finding that with some code I'm not really that concerned with which
charset I use, I simply want to be able to read some random source (a file
perhaps) and have it work without throwing an error because of the content
of that file.

Currently I'm having to prefix all my programs with code like this:

,----
| (let ((encoding (make-encoding :charset charset:iso-8859-1)))
|   (setq *default-file-encoding* encoding
|         *terminal-encoding*     encoding))
`----

I suppose, as you suggested, I should really dump an image with those values
wired in, that'll save me having to prefix all my lisp-scripts. Given that,
is it or might it be possible to specify a charset when building clisp?

-- 
Dave Pearson
http://www.davep.org/

---

From: Sam Steingold <[email-masked]>
Date: Wed Oct 10 06:25:08 2001

> * In message <[email-masked]>
> * On the subject of "Re: [clisp-list] internationalization problem on clisp-2.27"
> * Sent on Wed, 10 Oct 2001 07:40:47 +0100
> * Honorable Dave Pearson <[email-masked]> writes:
>
> Given that, is it or might it be possible to specify a charset when
> building clisp?

yes - just edit config.lisp.

actually, there was a typo in init.lisp - a misspelling of
*terminal-encoding*.
if you fix it and rebuild, you would probably be fine.

-- 
Sam Steingold (http://www.podval.org/~sds)
Support Israel's right to defend herself! <http://www.i-charity.com/go/israel>
Read what the Arab leaders say to their people on <http://www.memri.org/>
Takeoffs are optional.  Landings are mandatory.

---

From: Dave Pearson <[email-masked]>
Date: Wed Oct 10 08:04:11 2001

> yes - just edit config.lisp.

I tried that by adding:

,----
| ;; I live in England.
| (let ((encoding (make-encoding :charset charset:iso-8859-1)))
|   (setq *default-file-encoding* encoding
|         *pathname-encoding*     encoding
|         *misc-encoding*         encoding
|         *terminal-encoding*     encoding))
`----

and then doing a rebuild (this was after doing a "make clean" and so on) but
it doesn't seem to have made any difference:

,----
| davep@hagbard:~$ clisp -K full -norc -q
| 
| [1]> *misc-encoding*
| #<ENCODING CHARSET:ASCII :UNIX>
| [2]>
`----

Is this what you were suggesting or did you mean something else?

> if you fix it and rebuild, you would probably be fine.

Would that be "*T*GERMINAL-ENCODING*"? I fixed that as part of the above
rebuild but:

,----
| davep@hagbard:~$ clisp -K full -norc -q
| 
| [1]> (apropos "erminal-encoding")
| *T*GERMINAL-ENCODING*
| SYSTEM::*TERMINAL-ENCODING*                symbol-macro
| SYSTEM::SET-TERMINAL-ENCODING              function
| SYSTEM::TERMINAL-ENCODING                  function
| 
| [2]>
`----

Would a "make clean" not have been enough?

I fear that I'm missing something obvious and vital regarding this encoding
business. I've read the section in the impnotes but I have to admit that I'm
none the wiser about what's going on here.

What confuses me in all of this is what this encoding business aims to
provide. Is there something within the clisp documentation that outlines
this? All I do know is that there was a time in the past when clisp didn't
have a problem reading "8bit" data "out of the box" and that, these days, it
appears to have "problems" that require some fiddling before my code works.
I'm more than happy to accept that it's my lack of understanding but I'm
failing to find anything in the documentation that says why the default
would be to fail and throw an error.

Any explanations would be greatly appreciated.

All of the above tests were conducted with clisp 2.27.

,----
| davep@hagbard:~$ clisp --version
| GNU CLISP 2.27 (released 2001-07-17) (built 3211713097) (memory 3211713868)
`----

-- 
Dave Pearson
http://www.davep.org/

---

From: Sam Steingold <[email-masked]>
Date: Wed Oct 10 08:26:10 2001

> * In message <[email-masked]>
> * Sent on Wed, 10 Oct 2001 16:02:47 +0100

I assume that you did "make install" too: the above command will invoke
the installed version, not the development version.

> Is this what you were suggesting or did you mean something else?

yes, this is what I meant.

> Would that be "*T*GERMINAL-ENCODING*"? I fixed that as part of the
> above rebuild but:
> 
> ,----

okay, this is bad.  as long as you see this, you have not fixed the
problem. 

> Would a "make clean" not have been enough?

it should have been.  you need to install though.  I always use a
separate build directory and the build/.clisp script there.

> All of the above tests were conducted with clisp 2.27.

Could you please try the current pre-test (cvs up)?

-- 
Sam Steingold (http://www.podval.org/~sds)
Support Israel's right to defend herself! <http://www.i-charity.com/go/israel>
Read what the Arab leaders say to their people on <http://www.memri.org/>
If Perl is the solution, you're solving the wrong problem. - Erik Naggum

---

From: Dave Pearson <[email-masked]>
Date: Wed Oct 10 09:20:12 2001

> the installed version, not the development version.

Yes, I did do a "make install".

> okay, this is bad.  as long as you see this, you have not fixed the

It seems that a "make clean" wasn't enough for this. A "rm -r full" from the
build directory followed by a make seemed to solve this.

,----
| *TERMINAL-ENCODING*                        symbol-macro

Note that this rebuild (after the "rm -r full") made no difference to the
problem of the encoding settings in config.lisp failing to come thru in the
final image. Note, also, that other settings from config.lisp (loads paths,
site names, etc...) do come thru so config.lisp would appear to be read and
used.

> Could you please try the current pre-test (cvs up)?

Unfortunately I'm on the end of a dial-up connection so that makes for a
prohibitively large download because SF's CVS server appears to be
appallingly slow to speak to at the moment. IOW, no, not any time soon
(unless there's a source tarball available from a good download site?).

-- 
Dave Pearson
http://www.davep.org/

---

From: Sam Steingold <[email-masked]>
Date: Wed Oct 10 10:44:10 2001

> * In message <[email-masked]>
> * Sent on Wed, 10 Oct 2001 17:18:07 +0100
> Note that this rebuild (after the "rm -r full") made no difference to
> the problem of the encoding settings in config.lisp failing to come
> thru in the final image. Note, also, that other settings from
> config.lisp (loads paths, site names, etc...) do come thru so
> config.lisp would appear to be read and used.

I was wrong - the encoding variables (actually, symbol-macro places) are
_not_ saved into the memory images.

This is a feature, not a bug: since they are initialized from the
environment (see <http://clisp.cons.org/clisp.html#opt-enc>), they
should not be "set in stone" (if I am wrong again, and this _is_ a bug,
Bruno will correct me :-).

please investigate the values if the environment variables 
LC_ALL, LC_CTYPE, LANG.

I do not set them and I have
$ clisp -q -norc -x '*misc-encoding*'
#<ENCODING CHARSET:ISO-8859-1 :UNIX>
$

-- 
Sam Steingold (http://www.podval.org/~sds)
Support Israel's right to defend herself! <http://www.i-charity.com/go/israel>
Read what the Arab leaders say to their people on <http://www.memri.org/>
((lambda (x) (list x (list 'quote x))) '(lambda (x) (list x (list 'quote x))))

---

From: Dave Pearson <[email-masked]>
Date: Wed Oct 10 23:42:06 2001

On Wed, Oct 10, 2001 at 01:42:18PM -0400, Sam Steingold wrote:

> environment (see <http://clisp.cons.org/clisp.html#opt-enc>), they should
> not be "set in stone" (if I am wrong again, and this _is_ a bug, Bruno
> will correct me :-).

Ok, that sort of makes sense. 

Given this, what's the right thing to do if I want to write some code that
reads a random file that might contain data outside of the range considered
to be ASCII and I want it to run in a random environment? How can I write
such code so that it doesn't blow up if accented characters (for example)
appear in a file it reads?

Am I doing the right thing now by setting the encoding to a random 8bit
encoding (I say random because, although ISO-8859-1 makes sense for me, it
might not make sense for someone else). Is there a plain "8bit" encoding?
Does that question make sense? Should/could the default encodings be
something that can handle 8bit? If not, why not?

> #<ENCODING CHARSET:ISO-8859-1 :UNIX>

It's not clear what you mean when you say "them" here. Are you talking about
the environment variables or are you talking about the encoding variables
within clisp itself?

-- 
Dave Pearson
http://www.davep.org/

---

From: Sam Steingold <[email-masked]>
Date: Thu Oct 11 06:40:30 2001

> * In message <[email-masked]>
> * Sent on Thu, 11 Oct 2001 07:39:35 +0100
> Given this, what's the right thing to do if I want to write some code
> that reads a random file that might contain data outside of the range
> considered to be ASCII and I want it to run in a random environment?
> How can I write such code so that it doesn't blow up if accented
> characters (for example) appear in a file it reads?

(setq *default-file-encoding* charset:utf-8)
or
(open ... :external-format charset:utf-8)

see
<http://clisp.cons.org/impnotes.html#open>
<http://clisp.cons.org/impnotes.html#def-file-enc>
<http://clisp.cons.org/impnotes.html#extfmt>

> Am I doing the right thing now by setting the encoding to a random
> 8bit encoding (I say random because, although ISO-8859-1 makes sense
> for me, it might not make sense for someone else). Is there a plain
> "8bit" encoding?  Does that question make sense? Should/could the
> default encodings be something that can handle 8bit? If not, why not?

charset:utf-8 comes to "plain 8bit" as close as I can imagine.
you should ask someone else though - I am no expert on these matters.

> It's not clear what you mean when you say "them" here. Are you talking
> about the environment variables or are you talking about the encoding
> variables within clisp itself?

both - "clisp -norc" always invokes the default vanilla CLISP image.

$ unset LC_ALL LC_CTYPE LANG
$ clisp -q -norc -x '*misc-encoding*'
#<ENCODING CHARSET:ISO-8859-1 :UNIX>
$

--
Sam Steingold (http://www.podval.org/~sds)
Support Israel's right to defend herself! <http://www.i-charity.com/go/israel>
Read what the Arab leaders say to their people on <http://www.memri.org/>
War doesn't determine who's right, just who's left.

---

From: Dave Pearson <[email-masked]>
Date: Thu Oct 11 08:04:05 2001

> #<ENCODING CHARSET:ISO-8859-1 :UNIX>

Curious, I wonder why I get something different then?

,----
| davep@hagbard:~$ unset LC_ALL LC_CTYPE LANG
| davep@hagbard:~$ clisp -q -norc -x '*misc-encoding*'
| #<ENCODING CHARSET:ASCII :UNIX>
`----

-- 
Dave Pearson
http://www.davep.org/

---

From: Bruce <[email-masked]>
Date: Fri Oct 12 01:20:06 2001

> | #<ENCODING CHARSET:ASCII :UNIX>

I can confirm Dave's result on Debian GNU/LINUX sid. My locale is
normally set to zh_CN.GB2312. I guess that's controlled by glibc setings.