Subject: [clisp-list] read-char in http log

From: [email-masked] (Don Cohen)
Date: Sat Mar 20 21:38:02 2004

> BTW, since you're reading httpd server log files, why is CR and/or
 > LF a problem at all?? If you stick to CLISP's automatic conversion
 > (which is perfect for text procesing), your code will run equally
 > on httpd well on MS-Windows and UNIX machines. What am I missing?

Since you ask, and I think you're going in the wrong direction,
I'll describe the context.

I've described earlier how my server reads bytes, converts them to
chars with code-char then manipulates them as chars.
My log files are written as characters with (print ...)
So I have no trouble just doing (read <log>) from the beginning
to the end.
I wrote a quick and dirty program to read all the log entries,
collect the requests, group them by IP address, sort the groups by 
size and show the first request for each ip address.
The result looks something like this: 

;; number of requests, ip address then earliest log entry
((153 "24.126.116.41" 
  (:HTTP "3/19/2004 20:40:00" "24.126.116.41" :GET
   "/scripts/root.exe?/c+dir"))  
 (16 "24.126.20.50" 
  (:HTTP "3/19/2004 23:19:17" "24.126.20.50" :GET
   "/scripts/root.exe?/c+dir")) 
 (13 "24.126.165.92" 
  (:HTTP "3/20/2004 00:06:10" "24.126.165.92" :GET 
   "http://www.outwar.com/page.php?x=888132")) 
 ...
The object is to identify the addresses sending the most attacks 
(all of above in this case are attacks) and stop listening to them, as
in 
 iptables -A INPUT -s 24.126.116.41 -j DROP 

A few days later when I had seen a bunch of new attacks I wanted to
do the same computation but starting from the middle of the log
so I didn't have to look at the old data.
My quick and (as it turns out, much too) dirty approximation was to
accept as input a file position and start reading at that position.
If it were a position where a log entry started everything would be
fine.  
I figured (this was the real mistake) I would start at any old input
position, do a read-line, and hope that the next line started at the
begining of a log entry.
Well, it didn't.  The input was a url that contained a newline, so I 
was trying to read some stuff out of the middle of the url.
What confused me was the fact that READ complained about an
illegal character.  It never occurred to me that any characters were
illegal to read.  I figured that all characters not otherwise
specified would act like #\A.  So I mistakenly thought that this was a
problem related to character conversion, just cause that's the problem
that has always bitten me in the past.