Subject: [clisp-list] Re: FFI errors - tracking them closer

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Tue Sep 16 02:20:03 2003

Hi,

>> BTW, I think such a generic name - "buffer" - is a very bad 
>choice for
>> your variable.  It'll get global scope in the object files and may
>> interfere with other places. 

>It is within the package RAWSOCK - so unless someone else modifies
>RAWSOCK to use the same variable we should be safe.

I didn't mean the Lisp variable. I meant the C one. It gets global scope across all objects. You wrote:
>; (setq snaplen 1514)
>(def-c-var buffer (:name "buffer"...

which leads (or lead, until recent past) to
>extern uint8 (buffer)[1518];
extern/global, not static.

Maybe you could use
(def-c-var buffer (:name "packet_analyser_snapshot_buffer")
or whatever?

>> Fred, actually, what do you need it for?
I meant the C variable buffer that you want to access from Lisp.
Could you use dynamically allocated memory instead?
Could you use :out FFI parameter modes instead, which can supply arrays?

Especially for your recvfr() and sendt(), this could make a difference, otherwise all such function stomp on the same single buffer. But that's maybe what you want (maintain performance by not copying twice each packet, which reminds me of a paper written by the guys who reimplemented TCP/IP in SML).

BTW, 16 or 20 bytes timestamp?
>char timestamp[20],
>(def-c-var timestamp (:name "timestamp") (:type (c-array uchar 16)) (:read-only t) (:alloc :none))

So to summarize, I believe (without trying) that the following is necessary to get rawsock to work with clisp-2.31
o provide some rawsock.h which declares the variables and functions
o (FFI:C-LINES "#<include it")
That's compatible with older versions of CLISP.
o get rid of offset macro problem (foreign.d uses offset as a local variable name, so it apparently isn't a macro in the core code?!).
o more?

>getting the file size limit up to 4 Tbytes instead of 2.1Gbytes - which
I have now seen a system with a 2,7GB sized logfile!
I was real happy that "less" worked correctly with it.

Sadly, Emacs doesn't work with huge logfiles, and I miss it's incremental search and built-in grepping facilities.

>It' such a little code snippet I figured it wasn't worth two files.
You're not mainstream :-) I'm joking, and I find it sad that the automated linkkit module mechanism seems to oblige one to provide a one-line '(MAKE-PACKAGE "e.g. LDAP")' file (witness modules/dirkey/preload.lisp), which will result in the build of an intermediate 1MB image file, all of which is wasted space IMHO. But I repeat myself here.
I also find the statement "HD space is cheap these days" false and blinding. Please provide cheap HD space to every computer owner world wide (don't forget China, India etc.). I'm sure you bank will want to talk to you.

Regards,
	Jorg Hohle.

---

From: Fred Cohen <[email-masked]>
Date: Tue Sep 16 07:18:05 2003

> >> BTW, I think such a generic name - "buffer" - is a very bad choice for

> extern/global, not static.

Ah!!! Very bad...  I should probably use rawsock- before all C variables
then.

> (def-c-var buffer (:name "packet_analyser_snapshot_buffer")

Is there a convention here for doing it this way? If not let's make one. 
Which brings me to the offset macro somewhere within FFI (I believe) in
the new clisp which causes by offset function within rawsock (declared
in a lisp thing somewhere) to go awry. 

> Especially for your recvfr() and sendt(), this could make a
> difference, otherwise all such function stomp on the same single buffer. 
> But that's maybe what you want (maintain performance by not copying
> twice each packet, which reminds me of a paper written by the guys who
> reimplemented TCP/IP in SML). 

I want the poerformance - a lot of the things the responder application
does include replacing a few bytes here and there and sending the packet
back out.

> >(def-c-var timestamp (:name "timestamp") (:type (c-array uchar 16)) (:read-only t) (:alloc :none))

An old habit of mine.  I allocate more space than I need to prevent
stupid overruns caused by off-by-one errors from hurting me.  I will
see if this can be safely fixed.

> So to summarize, I believe (without trying) that the following is
> necessary to get rawsock to work with clisp-2.31

> o provide some rawsock.h which declares the variables and functions

This seems reasonable enough.

> That's compatible with older versions of CLISP.

This seems reasonable enough.

> o get rid of offset macro problem (foreign.d uses offset as a local
> variable name, so it apparently isn't a macro in the core code?!). 

So shouldn't foreign.d also use a name like foreign-offset?  I will
change mine as well, but...

> I was real happy that "less" worked correctly with it.

This is a small logfile in my experience.  And I regularly do disk
images of 120Gig disks which I would like to process with clisp.  The
reason less and some other code works is that my team at Sandia (when I
was there) did these fixes and promulgated them into the community.
It was part of making Linux more suitable to forensics work.

> Sadly, Emacs doesn't work with huge logfiles, and I miss it's
> incremental search and built-in grepping facilities. 

If this is a 4Tbyte issue with file I/O we should fix it - of course you
might need a lot of RAM and swap space... 

> You're not mainstream :-) I'm joking, and I find it sad that the
> automated linkkit module mechanism seems to oblige one to provide a
> one-line '(MAKE-PACKAGE "e.g.  LDAP")' file (witness
> modules/dirkey/preload.lisp), which will result in the build of an
> intermediate 1MB image file, all of which is wasted space IMHO.  But I
> repeat myself here.

> I also find the statement "HD space is cheap these days" false and
> blinding.  Please provide cheap HD space to every computer owner world
> wide (don't forget China, India etc.).  I'm sure you bank will want to
> talk to you. 

I don't think I made that assertion - the cost of disk space is in its
use more than the materials. 

I will eventually upgrade rawsock per your suggestion.  But in the
meanwhile because of library issues and a lack of static compilation I
am still at 2.27. 

FC

> 	Jorg Hohle.

-- This communication is confidential to the parties it is intended to serve --
Fred Cohen - http://all.net/ - [email-masked] - [email-masked] - tel/fax: 925-454-0171
Fred Cohen & Associates	- University of New Haven - Security Posture