Subject: Re[2]: [clisp-list] Separate Unix error stream keyword for ext:ru n-shell-command

From: "Hoehle, Joerg-Cyril" <[email-masked]>
Date: Thu Jun 5 09:18:02 2003

Hi,

here are my (unasked for) comments on this issue:

>> I'm slighltly confused about making it user-level. It is very similar
>> to run-program and run-shell-command but it is more `direct' and does

>We already have a whole bunch of built-in functions that do
>similar things: SHELL, MAKE-PIPE-INPUT-STREAM, MAKE-PIPE-OUTPUT-STREAM,
>MAKE-PIPE-IO-STREAM.
>That's more than enough.
>I want to have just one single built in.
>Let us make LAUNCH user-level only when it has absorbed all the
>functionality of the 4 other commands, and implement them as calling
>LAUNCH and deprecate them.

I'm opposed to such global removal. It's would be a Schemesque approach (i.e. "why do you need catch/throw/block/return/ignore-errors/handler-case when all is needed is call/cc?").

Another argument to this issue is: higher-level stuff (e.g. ignore-errors) is usually easier to mimic than passing through a single primitive (i.e. call/cc).

For instance, a naive (mini) CL on top of a JVM would easily provide IGNORE-ERRORS. It cannot easily provide handler-bind, because exception semantics are really different (cf. unwinding of stack in exceptional situations).

Please leave in a kiss-style (SHELL &optional cmd+args), which does several things:
+ ability to suspend CLISP, portably invoking a user-shell shell
 (please don't tell me that I can just open another X window).
+ ability to trust the programmer or user when s/he knows exactly what
 to pass to the shell instead of having CLISP second-guess quoting issues
 on all systems or situations where the execvlpe() family of calls may
 not be used. Second-guessing is always going to fail in some situations,
 typically where a cygwin-compiled program tries to invoke a native one,
 or vice-versa.

o Similarly, make-pipe-... stream are easy to comprehend and relatively easy to implement, even without a full-featured launch. E.g. On AmigaOS, it could be nothing more than
(open "APIPE:ls mydev:foo sort" :direction :input), on systems where the freeware APIPE device handler is installed.
LAUNCH may possibly allow more features, e.g. have the programmer choose whether stderr goes or not to a single Lisp-input-stream shared with stdout. That's a definitive plus.

>(with-open-file (null "/dev/null" :direction :output :if-exists :append)
o Please support :stream nil or some such convention, because having the programmer to use "/dev/null" in code will immediately loose the portability that CLISP tries to maintain.

Heck, I use CLISP for its portability!
People, please try to avoid gratuitous system dependencies.

o a PID is an abstract data type where portability matters. On UNIX it's known to be a small integer (for better or worse[*]), on MS-Windows it may very well be a pointer (don't know), and on AmigaOS it *is* a pointer. It will be hard for (LAUNCH :asynchronous t) to return a portable thing -- at least return an abstract/opaque thing.

>But I have never created lisp types from scratch.
In some situations (e.g. rexx-send-command) we used to just return a FOREIGN-POINTER object. Formerly, we used a simple bit-vector -- at least it was something else than an integer, and EQ-ness could be checked for, preventing fake arguments!
What I mean here is it can be enough to agree to either document that the exact type is purposely not documented and to be considered opaque, even though it may happen to be an integer on some systems, or document all OS cases.
(dotimes (i 10000) (posix:kill i)) is perfectly fine, since for POSIX, pids are tiny integers (or am I wrong here?).

Enough said, maybe I shall go Python instead.

[*] Please write an essay on the pros and cons of open() etc. using small integers vs. fopen() using FILE pointers. I'd be happy to discuss such design issues with everybody.

What I did not say (yet):
o I consider execxyz() a good thing, because it avoids tcl-style madness (the quoting problem, also known to linguists) to some extent. BTW, did you ever use find -o or -exec on UNIX?
o I also consider having EXT:[:]LAUNCH a good thing (but not that MS-Windows specific :indirect keyword, which is not understandable per se).
o Launch's ability to generate the equivalent of pipe-input/output-streams for use in Lisp is also valuable.
o I'd maybe provide a portable :priority setting (:high/:low/:normal mapping to OS&context dependent values) for portability-aware programmers. It's just cheap to do so.
o Having two distinct Lisp-side streams for reading stdout & stderr command output can result in deadlocks.
o An advantage of MAKE-PIPE-*-STREAM over LAUNCH is that it does not have to deal with producing PID objects/values that the programmer doesn't care about. That's another reason why it's easier to implement or provide portably (cf. call/cc).
o BTW, I think I've seen MAKE-PIPE-*-STREAM produce zombies (or just unusable processes) on Linux (e.g. forgetting about the stream and having the GC get (or not?) rid of it).

Regards,
	Jorg Hohle.

---

From: Sam Steingold <[email-masked]>
Date: Thu Jun 5 12:01:11 2003

> * In message <[email-masked]>
> * On the subject of "Re[2]: [clisp-list] Separate Unix error stream keyword for ext:ru n-shell-command"
> * Sent on Thu, 5 Jun 2003 18:05:07 +0200
> * Honorable "Hoehle, Joerg-Cyril" <[email-masked]> writes:
>

??  comments are always welcome, especially from a CLISP developer with
your experience!!

> I'm opposed to such global removal. It's would be a Schemesque
> approach (i.e. "why do you need
> catch/throw/block/return/ignore-errors/handler-case when all is needed
> is call/cc?").

I am saying that SHELL & MAKE-PIPE-*-STREAM should be implemented on
top of LAUNCH.  This is called "code reuse".

>  (please don't tell me that I can just open another X window).

(LAUNCH "" :INDIRECT T)

>  typically where a cygwin-compiled program tries to invoke a native one,

(LAUNCH "your command line here" :INDIRECT T)

> o Similarly, make-pipe-... stream are easy to comprehend and
> relatively easy to implement, even without a full-featured
> launch. E.g. On AmigaOS, it could be nothing more than (open "APIPE:ls
> mydev:foo sort" :direction :input), on systems where the freeware
> APIPE device handler is installed.

(defun MAKE-PIPE-INPUT-STREAM (cmd &rest options)
  (LAUNCH cmd :INDIRECT T :INPUT `(:STREAM ,options) :OUTPUT NIL :ERROR NIL))

> o Please support :stream nil or some such convention, because having
> the programmer to use "/dev/null" in code will immediately loose the
> portability that CLISP tries to maintain. 

of course.  it's just that it is not clear yet how to do this.

> o a PID is an abstract data type where portability matters. On UNIX
> it's known to be a small integer (for better or worse[*]), on
> MS-Windows it may very well be a pointer (don't know), and on AmigaOS
> it *is* a pointer. It will be hard for (LAUNCH :asynchronous t) to
> return a portable thing -- at least return an abstract/opaque thing. 

FOREIGN-POINTER is an option, then.

> In some situations (e.g. rexx-send-command) we used to just return a
> FOREIGN-POINTER object. Formerly, we used a simple bit-vector -- at
> least it was something else than an integer, and EQ-ness could be
> checked for, preventing fake arguments!
> What I mean here is it can be enough to agree to either document that
> the exact type is purposely not documented and to be considered
> opaque, even though it may happen to be an integer on some systems, or
> document all OS cases.
> (dotimes (i 10000) (posix:kill i)) is perfectly fine, since for POSIX,
> pids are tiny integers (or am I wrong here?).

making a new opaque type PID would mean adding a function
MAKE-PID &c &c.

> Enough said, maybe I shall go Python instead.

huh?
are you jumping the ship?

> o I consider execxyz() a good thing, because it avoids tcl-style
> madness (the quoting problem, also known to linguists) to some
> extent. BTW, did you ever use find -o or -exec on UNIX?
of course!

> o I also consider having EXT:[:]LAUNCH a good thing (but not that
> MS-Windows specific :indirect keyword, which is not understandable per
> se).

it's nor woe32-specific!

> o I'd maybe provide a portable :priority setting (:high/:low/:normal
> mapping to OS&context dependent values) for portability-aware
> programmers. It's just cheap to do so.

problem is that you cannot raise priority on Unix unless you are root.
we can add a table, e.g., :high -> "-5", :low -> "+5"...

> command output can result in deadlocks.

that's what SOCKET-STATUS is for!

> o An advantage of MAKE-PIPE-*-STREAM over LAUNCH is that it does not
> have to deal with producing PID objects/values that the programmer
> doesn't care about. That's another reason why it's easier to implement
> or provide portably (cf. call/cc).

that's trivial compared with the rest.

> o BTW, I think I've seen MAKE-PIPE-*-STREAM produce zombies (or just
> unusable processes) on Linux (e.g. forgetting about the stream and
> having the GC get (or not?) rid of it).

GC should kill the zombies.
doesn't it?

GC is not deterministic, so it might never actually collect the
specific stream.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
(let ((a "(let ((a %c%s%c)) (format a 34 a 34))")) (format a 34 a 34))

---

From: Arseny Slobodjuck <[email-masked]>
Date: Thu Jun 5 16:59:10 2003

Hi,

> I'm opposed to such global removal. It's would be a Schemesque approach (i.e.
> "why do you need catch/throw/block/return/ignore-errors/handler-case when all is needed is call/cc?").

I'm "against" only run-program and run-shell-command on windows. I
wanted to improve them by making lowlevel supporting function, but
actually LAUNCH almost duplicates them. It should be improved however.

> In some situations (e.g. rexx-send-command) we used to just return a FOREIGN-POINTER object.
> Formerly, we used a simple bit-vector -- at least it was something else than an integer, and EQ-ness
> could be checked for, preventing fake arguments!
> What I mean here is it can be enough to agree to either document that the exact type is purposely
> not documented and to be considered opaque, even though it may happen to be an integer on some
> systems, or document all OS cases.
FOREIGN-POINTER will introduce dependency on FFI!

> o I consider execxyz() a good thing, because it avoids tcl-style madness (the quoting problem, also known to linguists) to some extent. BTW, did you ever use find -o or -exec on UNIX?
execxyz gives no way to redirect streams on windows, that's why we use
CreateProcess (I think) and bearing with quoting.

-- 
Best regards,
 Arseny

---

From: Sam Steingold <[email-masked]>
Date: Thu Jun 5 19:19:02 2003

> * In message <[email-masked]>
> * On the subject of "Re[3]: [clisp-list] Separate Unix error stream keyword for ext:ru n-shell-command"
> * Sent on Fri, 6 Jun 2003 11:02:46 +1000
> * Honorable Arseny Slobodjuck <[email-masked]> writes:
>

nope. FOREIGN-POINTER depends on FOREIGN which is defined on UNIX,
AMIGA, and WIN32 (via DIR_KEY).

> CreateProcess (I think) and bearing with quoting.

I see.  this boils down to the effective absence of dup2() on woe32.
could you please find out how cygwin implements dup2?

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Lisp: Serious empowerment.

---

From: Sam Steingold <[email-masked]>
Date: Fri Jun 6 17:48:04 2003

> [*] Please write an essay on the pros and cons of open() etc. using
> small integers vs. fopen() using FILE pointers. I'd be happy to
> discuss such design issues with everybody.

the original decision, I think, was predicated on avoiding libc (amiga?).

this is now gone, but using fopen()/FILE would result in libc doing
buffering, in addition to CLISP own buffering, thus complicating
unbuffered streams and introducing an unnecessary overhead for buffered
streams.

on a more mundane level, I see no way to duplicate FILE*.
we do need dup2 on several occasions.

I don't think it is worth our time to rewrite stream.d to use FILE.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Just because you're paranoid doesn't mean they AREN'T after you.