Subject: [clisp-list] Re: fastest (character) based io with CLISP?

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Fri Feb 7 07:38:07 2003

Hi,

I asked
>What's your experience about fastest possible IO with CLISP?

I now did some experiments. I found one culprit which is consuming almost all the time: POSITION (and all sequence functions).

>Is using READ-CHAR-SEQUENCE much faster than READ-LINE?
A loop using read-byte-sequence (not -char-) using (unsigned-byte 8) took
3 seconds for reading a 50MB file
Real time: 3.360564 sec.
Run time: 0.67 sec.     
Space: 72212 Bytes

but 17 seconds for read-line (and magnitudes more GC).
Real time: 17.933756 sec.  
Run time: 16.18 sec.       
Space: 203_229_604 Bytes     
GC: 388, GC time: 0.67 sec.

There's some room for improvement here.
I also don't know why reading 50MB from a file consumes 4 times this GC space  (on UNIX, only 2 on MS-Windows).

Now where does POSITION jump in? See below:

In another experiment, I read a 23MB file using read-line:
Real time: 8.689632 sec.   
Run time: 8.12 sec.        
Space: 96764244 Bytes      
GC: 185, GC time: 0.45 sec.

Then I added a call to SPLIT-LOG-LINE (code below) on each result of read-line
Real time: 40.52663 Sec.   
Run time: 40.34 sec.       
Space: 98_104_624 Bytes    
GC: 187, GC time: 0.63 sec.

That's a lot of time. When one looks at the internals, the built-in function POSITION in written in a very generic and operates on sequences. As a result, there's *a lot* of overhead. A specialized strindex() would be much much faster.
Similarly, (concatenate 'string strings...) is much much slower than (ext:string-concat strings) - I believe because of the same apparently immense "step through every element of the sequences one by one" overhead (see src/sequence.d).

Adding the actual log-file analysis (e.g. Top-N) merely added 5 seconds of processing time: that's negligible.

FWIW, I have written the code in a DSL approach which compiles
      (when (and (column-op action = "accept")
		     (column-op type = "account")
down to
(string= line "account" :start1 (column-index 'action) :end1 ...)
I.e. I found it important that no more strings (->GC) be created. 
(Column-index C)are really compile-time constants, since the ordering of columns in the log-file is known after the header-line has been read.

Please share your experience!
There's obviously room for improvement!

Regards,
	Jörg Höhle.

(defun split-log-line (line starts stops &optional (line-start 0) (line-end nil))
  "Stores positions of ;-separated fields in the given arrays."
  (loop for i from 0
	for start = line-start then (1+ stop)
	for stop = (position #\; line :start start :end line-end)
	with max = (length starts)
	when (= i max)
	    do (error "Too many columns")
	do (setf (aref starts i) start
		 (aref stops i) (or stop line-end (length line)))
	;TODO (error "Not enough columns")
	while stop))
;(split-log-line .line. .start-positions. .stop-positions.)
;(split-log-line "a;b" (make-array 1) (make-array 1))