Subject: [clisp-list] read-integer, read-float, write-integer and write-float bug + more

From: Bernard Urban <[email-masked]>
Date: Thu Dec 12 04:28:02 2002

The subject says it all. It is for clisp-2.30.

Consider reading with (read-float ... 'double-float :big) the
following IEEE coding of 1.0d0 in big-endian (as used in XDR
representation): 

3f f0 00 00 00 00 00 00

There is an underflow exception raised.

The culprit is elt_nreverse() in stream.d for byte swapping, 
which does not work as intented. Same is true for the other ones of the
family: read-integer, write-integer and write-float.

In clisp-2.27, the call for elt_nreverse() was specific code. It works
ok. Here is a patch to restore the old code:

I insist on not using elt_nreverse(), which has the bad effect of beeing
very costly (on files of 1Mb size full of double-float, the additional
time cost is about 30% compared to the 2.27 version).

========================================

My second remark is about the impossibility to have a localized clisp-2.30
version by using the standard INSTALL instructions. The key to obtain
such a version is to compile with -DENABLE_NLS, which seems not be
controllable from the configure options. Here is a simple patch:

--- clisp-2.30.orig/src/makemake.in
+++ clisp-2.30/src/makemake.in
@@ -1296,6 +1296,7 @@
   if [ "${with_gettext}" = "0" ]; then
     XCFLAGS=$XCFLAGS' -DNO_GETTEXT'
   else
+    XCFLAGS=$XCFLAGS' -DENABLE_NLS'
     if [ @USE_NLS@ = yes ] ; then
       USE_GETTEXT=yes
       LIBS='@LIBINTL@ '$LIBS

but the maintainers can probably do better. In particular, 
the debian clisp-2.30 in unstable does not have this patch and hence 
is not localized.

Regards.

-- 

Bernard Urban

---

From: Sam Steingold <[email-masked]>
Date: Thu Dec 12 07:28:02 2002

> * In message <[email-masked]>
> * On the subject of "[clisp-list] read-integer, read-float, write-integer and write-float bug + more"
> * Sent on 12 Dec 2002 13:26:57 +0100
> * Honorable Bernard Urban <[email-masked]> writes:
>
> following IEEE coding of 1.0d0 in big-endian

how do I create a file like that?

> (as used in XDR representation):
what is XDR?

> 3f f0 00 00 00 00 00 00

when I type this into a hex editor, I get this:

$ hexdump bin 
0000000 f03f 0000 0000 0000                    
0000008

I am confused.  I use linux/x86.  Can it be that I have a different
endiannes than you do?

> There is an underflow exception raised.

indeed.

But I also see this:

$ clisp -q -norc

[1]> (with-open-file (b "bin1" :element-type 'unsigned-byte :direction :output)
       (write-float 1d0 b  'double-float :big))
1.0d0
[2]> (with-open-file (b "bin1" :element-type 'unsigned-byte :direction :input)
       (read-float b  'double-float :big))
1.0d0
[3]> 
$ hexdump bin1
0000000 0000 0000 0000 3ff0                    
0000008

I.e., we do have i/o consistency here, right?

> The culprit is elt_nreverse() in stream.d for byte swapping, which
> does not work as intented. Same is true for the other ones of the

elt_nreverse() is also used for NREVERSE - could you please give an
example where it produces wrong results there?

> I insist on not using elt_nreverse(), which has the bad effect of
> beeing very costly (on files of 1Mb size full of double-float, the
> additional time cost is about 30% compared to the 2.27 version).

interesting - actually, elt_nreverse() does the exact same thing as the
old code (see the SIMPLE_NREVERSE CPP macro), so all the overhead is
the C function call. can it really be that expensive?

In the futrure, could you please send separate messages for separate
issues?  Thanks.

> My second remark is about the impossibility to have a localized
> clisp-2.30 version by using the standard INSTALL instructions.

Thanks for reporting this.
I fixed this problem in a different way.
Please try the latest CVS head and report whether the probles has been
indeed fixed.  Thanks.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
I don't want to be young again, I just don't want to get any older.

---

From: Sam Steingold <[email-masked]>
Date: Thu Dec 12 07:28:03 2002

what is XDR?

$ hexdump bin 

indeed.

But I also see this:

$ clisp -q -norc

---

From: Bernard Urban <[email-masked]>
Date: Fri Dec 13 00:33:03 2002

> how do I create a file like that?

I found the problem by using an XDR file generated by the Right Thing
(namely a rpcgen generated program). You could use clisp as in your
exemple below, BUT RELEASE 2.27 or before, of course...

> > (as used in XDR representation):

It is the serialized representation of data used by Sun Remote
Procedure Call. An incredible tool, largely forgotten today 
(or reinvented badly, as in XML-RPC).

> I am confused.  I use linux/x86.  Can it be that I have a different

I do found the problem on x86 (which is little endian, and XDR is
big endian, hence the swapping of F0 and 3F). So it is correct.
Try 'od -t x1' if in doubt.

> I.e., we do have i/o consistency here, right?

Yes, but...
Actually, the same error in write-float compensate for the one in
read-float (elt_nreverse being its own inverse).

> example where it produces wrong results there?

No, I think elt_nreverse() is probably correct, but there is a switch
statement, and you probably end up in the wrong case for read-float
and friends.

> the C function call. can it really be that expensive?

Apparently, it is so, for reading and writing, at least on the system I use (K6-2).

> In the futrure, could you please send separate messages for separate

Ok.

> Please try the latest CVS head and report whether the probles has been

I will be able to test this tomorrow.

Regards.

-- 

Bernard Urban

---

From: Sam Steingold <[email-masked]>
Date: Fri Dec 13 06:53:02 2002

> * In message <[email-masked]>
> * On the subject of "Re: [clisp-list] read-integer, read-float, write-integer and write-float bug + more"
> * Sent on 12 Dec 2002 17:17:30 +0100
> (namely a rpcgen generated program).

could you please give some more examples for the regression test?
right now I have

(let ((vec (make-array 8 :element-type '(unsigned-byte 8)
                       :initial-contents '(#x3f #xf0 0 0 0 0 0 0))))
  (with-open-file (foo "./foocl" :direction :output
                       :element-type '(unsigned-byte 8))
    (write-sequence vec foo))
  (prog1
      (with-open-file (foo "./foocl" :direction :input
                           :element-type '(unsigned-byte 8))
        (read-float foo 'double-float :big))
    (delete-file "./foocl")))

can you give me more binary representations of floats and integers in
various endiannesses?

> You could use clisp as in your

actually, CVS head CLISP should do too.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
(lisp programmers do it better)

---

From: Bernard Urban <[email-masked]>
Date: Thu Dec 19 10:02:03 2002

> can you give me more binary representations of floats and integers in

Here is a simplified version of the data which started all this:

And lisp code to test the reading of this data:

Alas, I have no example for little endian and single-float data. 
Notice that cmucl has not read-float, so I use the following, which
could be taken as a reference:

(defmacro to-double-float2 (x y bias mantisse expon modele)
  "Returns a float of type MODELE from 4-bytes unsigned X and Y seen as the
 bit pattern of a IEEE float with parameters BIAS, MANTISSE and EXPON."
  `(let ((f (ldb (byte ,(- mantisse 32) 0) ,x))
         (e (ldb (byte ,expon ,(- mantisse 32)) ,x)))
     (declare (type (unsigned-byte ,(- mantisse 32)) f))
     (declare (type (unsigned-byte ,expon) e))
     (if (and (zerop f) (zerop ,y) (zerop e))
         ,modele
       (scale-float (float (if (< ,(ash 1 (+ expon mantisse -32)) ,x)
                               (- (+ ,y (ash (+ f ,(ash 1 (- mantisse 32))) 32)))
                             (+ ,y (ash (+ f ,(ash 1 (- mantisse 32))) 32)))
                           ,modele)
                    (- e ,bias ,mantisse)))))

(defmacro to-double-float (x y)
  `(to-double-float2 ,x ,y 1023 52 11 0d0))

(defun tcp-getdouble (rpcstream)
  (let* ((v1 (xdr-read-unsigned rpcstream))
         (v2 (xdr-read-unsigned rpcstream)))
    (to-double-float v1 v2))

Notice that I cannot use read-integer, as it is requested to be
'(unsigned-byte 8) or a multiple thereof. Any reason for this
limitation ?

Regards.

-- 

Bernard Urban

---

From: Bernard Urban <[email-masked]>
Date: Thu Dec 19 10:16:12 2002

> '(unsigned-byte 8) or a multiple thereof. Any reason for this

Ah yes, for 4 byte integer, you should use  '(unsigned-byte 32) , not
'(unsigned-byte 4), so it makes sense. And it works ok:

Time for Christmas holydays...

-- 

Bernard Urban

---

From: Bernard Urban <[email-masked]>
Date: Thu Dec 19 10:46:05 2002

Concerning the read-integer, read-float, etc... problem, the
CVS from 17 December seems fine, at least for reading float and
positive integers.

The same code in CVS solves the localization problem. But I forgot to
mention a still unresolved issue, encountered when compiling on a
Debian woody system, without libncurses-dev. You must add manually to CFLAGS
-DNO_TERMCAP_NCURSES, or at linking lisp.run, there will be unresolved
references to termcap functions. libtermcap is not detected (that means
detected "broken"), but
obviously, the following code in makemake.in is not working properly:

if [ "${with_termcap_ncurses}" = "0" -o "${LIBTERMCAP}" = "broken" ] ; then
  XCFLAGS=$XCFLAGS' -DNO_TERMCAP_NCURSES'
  LIBTERMCAP="";
fi

(around line 700)

In addition, you must apply the following patch:
*** ../src/stream.dORI	Thu Dec 19 19:40:37 2002
--- ../src/stream.d	Wed Dec 18 19:03:54 2002
***************
*** 14812,14815 ****
--- 14812,14816 ----
        if (char_p) *char_p = true;
        return;
+ #ifdef KEYBOARD
      case strmtype_keyboard:
        if (in_sock)
***************
*** 14817,14820 ****
--- 14818,14822 ----
        if (char_p) *char_p = true;
        return;
+ #endif
      case strmtype_twoway_socket:
        obj = TheStream(obj)->strm_twoway_socket_input;

Rebuilding from source the .deb clisp-2.30 package 
does not trigger the bug, as package libncurses-dev is required. So there lies
the problem (patch recommended though)

Sincerely.

-- 

Bernard Urban

---

From: Sam Steingold <[email-masked]>
Date: Thu Dec 19 11:51:02 2002

> * In message <[email-masked]>
> * Sent on 19 Dec 2002 19:45:23 +0100
> I forgot to mention a still unresolved issue, encountered when
> compiling on a Debian woody system, without libncurses-dev. You must
> add manually to CFLAGS -DNO_TERMCAP_NCURSES, or at linking lisp.run,
> there will be unresolved references to termcap functions. libtermcap
> is not detected (that means detected "broken"), but obviously, the
> following code in makemake.in is not working properly:
> 

please look at the top of Makefile - what parameters were passed to makemake?

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
MS Windows: error: the operation completed successfully.

---

From: Sam Steingold <[email-masked]>
Date: Thu Dec 19 12:03:04 2002

> * In message <[email-masked]>
> * Sent on 19 Dec 2002 18:59:49 +0100

this is not too useful as is.

could you please do as you did with 1d0, e.g.:

(#x3f #xf0 0 0 0 0 0 0)  <-->  1d0 big endian
(0 0 0 0 0 0 #xf0 #x3f)  <-->  1d0 little

thanks.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Bill Gates is great, as long as `bill' is a verb.