Subject: [clisp-list] fastest (character) based io with CLISP?

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Thu Feb 6 08:23:02 2003

Hi,

I wish to outperform a perl application. It has processing some logfiles (>100MB). I wish to show how using Lisp makes real software fun, somewhat easy to validate (understandable?) and efficient and worthwhile to dumb customers.

What's your experience about fastest possible IO with CLISP?

Is character-based i/o much slower than using (unsigned-byte 8) (i.e. should I work on ASCII codes, emulate read-line etc.?

What's the influence of built-in unicode and encodings on the speed? Should I compile --without-unicode so that strings internally take only 1 byte per character?

Is using READ-CHAR-SEQUENCE much faster than READ-LINE?

On a Linux box, time wc -l foo.log (23MB) takes 0.2 seconds. That's a lower bound on what I can expect :-)

In my test, log.txt has 50.562.702 bytes:

(defun test (&optional (logfile "log.txt"))
  (time (with-open-file (stream logfile :direction :input
				#+CLISP :buffered #+CLISP T
				:element-type 'character
				:external-format charset:iso-8859-1)
	  (loop for line = (read-line stream nil nil)
		for line-count from 0
		while line
		finally return line-count))))

(compile 'test)
(time (test))
;;; Datei mit read-line lesen
Real time: 13.594 sec.
Run time: 13.171875 sec.
Space: 102802424 Bytes
GC: 196, GC time: 0.671875 sec.

Note how Space is roughly twice the size of my logfile.

Please share your experience!

Regards,
	Jörg Höhle.

---

From: Sam Steingold <[email-masked]>
Date: Fri Feb 7 07:45:02 2003

> * In message <[email-masked]>
> * On the subject of "fastest (character) based io with CLISP?"
> * Sent on Thu, 6 Feb 2003 17:21:53 +0100
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>
> Is character-based i/o much slower than using (unsigned-byte 8)
> (i.e. should I work on ASCII codes, emulate read-line etc.?

probably not.

> What's the influence of built-in unicode and encodings on the speed?
> Should I compile --without-unicode so that strings internally take
> only 1 byte per character?

no.

> Is using READ-CHAR-SEQUENCE much faster than READ-LINE?

READ-CHAR-SEQUENCE boils down to a single read(2).
READ-LINE is one char at a time, which should _NOT_ matter for buffered
streams.

> GC: 196, GC time: 0.671875 sec.

the two obvious alternatives:

(let ((buf (make-array 10240 :element-type '(unsigned-byte 8)))
      (nl (char-code #\Newline)))
  (defun test1 (&optional (logfile "/var/log/XFree86.0.log"))
    (time (with-open-file (stream logfile :direction :input
                                  #+CLISP :buffered #+CLISP T
                                  :element-type '(unsigned-byte 8)
                                  :external-format charset:iso-8859-1)
            (loop for bytes = (read-byte-sequence buf stream)
              until (zerop bytes)
              sum (count nl buf :end bytes) into nl-count
              sum bytes into byte-count
              finally return (values nl-count byte-count))))))

(let ((buf (make-array 10240 :element-type 'character)))
  (defun test2 (&optional (logfile "/var/log/XFree86.0.log"))
    (time (with-open-file (stream logfile :direction :input
                                  #+CLISP :buffered #+CLISP T
                                  :element-type 'character
                                  :external-format charset:iso-8859-1)
            (loop for chars = (read-char-sequence buf stream)
              until (zerop chars)
              sum (count #\newline buf :end chars) into nl-count
              sum chars into char-count
              finally return (values nl-count char-count))))))

are about half as fast (take twice as much time!) as your version, but
do almost no consing.

> Note how Space is roughly twice the size of my logfile.

that's bad - but it's the least problem.

you can compile CLISP with profiling and see where it spends all the
time.  comparing the profiling output for your version and mine would
be most instructive.

(still faster that ACL 6.2!)

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
A PC without Windows is like ice cream without ketchup.