Subject: [clisp-list] On benchmarking (was: TCP-based Network Server in CLISP: read-cha r vs. buffering)

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Tue Oct 1 09:05:07 2002

Hi,

here's a summary of Don' and my recent experiments.

My summary is: thanks to the Linux environment (thank not the application) for making non-buffered application-level i/o work not so much worse than buffered one.[2]
The ratio is incredibly worse with MS-Windows2k.

Dan observed speed up by factor 7 on local files - I presume on Linux.
I hereby report speedups by factor 1600 for sockets on MS-W2k.

Please read below the incredible numbers measured on my MS-W2k system.

Reading 1_000_000 bytes from a larger file (~3MB),
CLISP TIME reported real time/run time for one run:

from source:    local file      Samba/NFS server
READ-SEQUENCE   0.23/0.04       1.23/0.04
buffered        0.78/0.75       2.19/0.72
unbuff'ed       161/132 secs    1866/173 seconds!![1]

[1] CPU use was visually low during that time. I.e., CLISP was constantly waiting.

I.e. the network can showel >875KB/sec, but you better use buffering. CLISP's 8KB buffering less than halves the performance of the presumably fastest solution via READ-SEQUENCE from the NFS server, which I find acceptable - YMMV. Smaller units could be tested to see where the performance seriously degrades.

Unbuffered i/o is so slow that anybody on MS-w2k is going
1) to scream for buffering
2) to hate applications which don't do it.

[2] Maybe the difference has nothing to do with Linux vs MS-W2k but only with configuration settings, i.e. MTU, grouping small packets, urgent vs. high throughput TCP bit or whatever.

BTW, from Don's data:
> Real time: 0.015744 sec.
> Run time: 0.0 sec.
> Space: 0 Bytes
>  414834 chars left to go.  -- infinite [speed]

I learned not to rely on "system" time for end-to-end measurements on Sun UNIX machines years ago. Some applications were dead slow, while they took comparatively little system time. Using system time to benchmark would have been meaningless.
IIRC, CMUCL's GC would swap on our Suns, yet that did not count as system time. GC and swap time should count IMHO.
I consider inclusion of real-time to measure "round-trip" of the whole access - presuming a lightly loaded machine. 

Wall clock (or real time) is what counts.

I also learned to prefer slow CPU for testing, where timing for small runs would not be below measurability level.

But with todays machines, it's hard to avoid looping in simple tests in order to get large enough times. But this gives inaccurate numbers, as all kinds of caching occurs.

> Real time: 0.116746 sec.
> Run time: 0.12 sec.
That's a time warp, isn't it? :-) How can system time be smaller than real clock time? Does your machine have two processors? - I expect not.
We should not believe in two many digits precision...

Regards,
	Jorg Hohle.

(compile (defun f (file n) (with-open-file (s file)
  (setf (stream-element-type s) '(unsigned-byte 8))
  (loop for i below n do 
  (read-byte s)))))

(compile (defun g (file n) (with-open-file (s file :buffered nil)
  (setf (stream-element-type s) '(unsigned-byte 8))
  (loop for i below n do 
  (read-byte s)))))

(defun h (file n) (with-open-file (s file :buffered nil :element-type '(unsigned-byte 8))
  (read-sequence (make-array n :element-type '(unsigned-byte 8)) s)))

---

From: Todd Sabin <[email-masked]>
Date: Tue Oct 1 20:18:23 2002

"Hoehle, Joerg-Cyril" <[email-masked]> writes:

> My summary is: thanks to the Linux environment (thank not the
> application) for making non-buffered application-level i/o work not
> so much worse than buffered one.[2]

> unbuff'ed       161/132 secs    1866/173 seconds!![1]

I suspect this has little to do with Windows, per se, and more to do
with the way clisp does I/O on Windows.  If you look at
DoInterruptible in win32aux.d, you'll notice that clisp creates (and
destroys) a thread every time through DoInterruptible.  Combine that
with an unbuffered stream and the result is that clisp creates and
destroys an OS thread for every character read.  I'm not sure if that
would account for all of the difference, but it seems like a very good
place to start...

Todd