Subject: [clisp-list] task proposal: better integrated regexp (was: modules and FFI _de pendency_)

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Wed Nov 27 07:33:01 2002

Hi,

I wrote:
>I doubt FFI is not included in any typical build of CLISP.
I have to correct myself: I forgot some of the compilation problems with ffcall mentioned by Will Newton et al.
No ffcall => no FFI, thus there are builds of CLISP on UNIX without FFI.

However, Dan's gdi module should still work, since I've recently pointed out that modules do not necessarily depend upon the FFI.

An interesting(?) call for volunteer task could be:
	better integration of regexp into CLISP

a) it depends on the FFI
b) as such, it uses "1:1" C-style strings, whereas CLISP uses some UCS internally. The result for each call is character encoding conversion from CLISP's internal format to custom:*foreign-encoding* (probably iso-8859-1).
c) because of 1:1, it doesn't work with all supported encodings, e.g. not with Chineese, Korean or Japaneese ones
d) the API is not efficient at all (w.r.t. to multiple invocations), as I pointed out in April, subject "About speed, and regexp&FFI in particular".

The better regexp module could operate directly on CLISP strings, without encoding conversion and copying strings on the stack.

i) A version of the regexp code which operates on two-byte characters would integrate much better with CLISP.
ii) While doing so, the regexp interface could be turned into a "pure module", not needing the FFI (for howto see my module cookbook).

Problem areas:
1) I don't know whether the GNU/perl/xyz regexp code would have to be changed for this to work.
2) Furthermore, CLISP also uses some 7bit strings internally (e.g. on some constant symbols names): don't assume everything is UCS-2.

Of course, one could argue this is solely an optimization area (except for the 1:1 restriction). Time and effort might be better devoted to bringing new things to CLISP.

Difficulty: unassessed because of 1)
Interest/Fun factor: not assessed

Any comments?
Is sourceforge's task section a good place for these kinds of proposals?

Regards,
	Jörg Höhle.