Subject: [clisp-list] Separate Unix error stream keyword for ext:run-shell-command

From: Adam Warner <[email-masked]>
Date: Tue Jun 3 18:24:02 2003

Hi all,

In August last year a missing error stream keyword for
ext:run-shell-command was discussed and Marco Baringer said he'd be
"happy to take a whack at" implementing it. Did this ever get
implemented in the CVS version of CLISP?

Regardless, I'm so looking forward to CLISP version 2.31.

Thanks,
Adam

PS: Sam Steingold approved of implementing this
(23 Aug 2002 11:02:17 -0400):

> * In message <[email-masked]>
> * On the subject of "Re: [clisp-list] How to suppress / redirect standard error?"
> * Sent on 23 Aug 2002 16:29:41 +0200
> * Honorable Marco Baringer <[email-masked]> writes:
>
> 1) wouldn't it be nice to have an :ERROR-STREAM keyword arg ala :INPUT
>    and :OUTPUT? (Sam, if you don't have objections i'd be happy to
>    take a whack at this.)

go ahead!

> 2) Why can't one specify a stream as the arg to :OUTPUT or :INPUT?

a missing feature?

thanks!

---

From: Sam Steingold <[email-masked]>
Date: Wed Jun 4 06:08:15 2003

> * In message <[email-masked]>
> * On the subject of "[clisp-list] Separate Unix error stream keyword for ext:run-shell-command"
> * Sent on 04 Jun 2003 13:22:50 +1200
> * Honorable Adam Warner <[email-masked]> writes:
>

Arseny is working on this - see EXT::LAUNCH (internal!)

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
I want Tamagochi! -- What for?  Your pet hamster is still alive!

---

From: Arseny Slobodjuck <[email-masked]>
Date: Wed Jun 4 06:47:17 2003

> Arseny is working on this - see EXT::LAUNCH (internal!)

I'm slighltly confused about making it user-level. It is very similar
to run-program and run-shell-command but it is more `direct' and does
slightly more. But old ones are available on OS/2 (I'm not sure
whether current clisp is still compileable on OS/2). Well, I'll
document it and make user level. This is short description for
discussion. BTW, Sam, did you changed default behavior to `don\'t wait'
intentionally ?

/* (LAUNCH executable [:arguments] [:wait] [:input] [:output] [:error])
   Launches a program.
   :arguments : a list of strings
   :wait - nullp/not nullp - whether to wait for process to finish
   :priority : on windows : HIGH/LOW/NORMAL on UNIX : fixnum - see nice(2)
   :input, :output, :error - i/o/e streams for process. basically file-streams
     or terminal-streams. see stream_lend_handle() in stream.d for full list
     of supported streams
     returns: exit code (zero when (nullp wait))  */

-- 
Best regards,
 Arseny

---

From: Adam Warner <[email-masked]>
Date: Wed Jun 4 07:53:36 2003

>      returns: exit code (zero when (nullp wait))  */

Looks great. I'll be able to eliminate all the unnecessary but verbose
messages currently clutering up my Apache logs (e.g. every time ls
"fails" by not finding a particular wildcard match--Note that I do know
about other file location options).

Thanks,
Adam

---

From: Sam Steingold <[email-masked]>
Date: Wed Jun 4 07:57:05 2003

> * In message <[email-masked]>
> * On the subject of "Re[2]: [clisp-list] Separate Unix error stream keyword for ext:run-shell-command"
> * Sent on Thu, 5 Jun 2003 00:51:29 +1000
> * Honorable Arseny Slobodjuck <[email-masked]> writes:
>
> whether current clisp is still compileable on OS/2).

OS/2 has been officially dead and unsupported (by IBM) for years.

> Well, I'll document it and make user level.

Please don't.
We already have a whole bunch of built-in functions that do
similar things: SHELL, MAKE-PIPE-INPUT-STREAM, MAKE-PIPE-OUTPUT-STREAM,
MAKE-PIPE-IO-STREAM.
That's more than enough.
I want to have just one single built in.
Let us make LAUNCH user-level only when it has absorbed all the
functionality of the 4 other commands, and implement them as calling
LAUNCH and deprecate them.

In particular, it means that it needs an :INDIRECT argument to indicate
that a shell should be called.  Note that win32 calls CreateProcess()
which, I think, calls a shell(?), while unix uses execvp() which does
not...

It will also need to accept :STREAM as the :INPUT/:OUTPUT/:ERROR
argument to mean to create - and return - a new stream.

> BTW, Sam, did you changed default behavior to `don\'t
> wait' intentionally ?

no, sorry.  I will fix that.

>      returns: exit code (zero when (nullp wait))  */

it returns NIL if wait is NIL.
(should probably return the PID, INPUT, OUTPUT, ERROR?)

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Please wait, MS Windows are preparing the blue screen of death.

---

From: Douglas Philips <[email-masked]>
Date: Wed Jun 4 08:21:04 2003

On Wednesday, Jun 4, 2003, at 10:53 US/Eastern, Sam Steingold wrote:
> t returns NIL if wait is NIL.

This has all been interesting to follow.
I'm wondering, did I miss a message or five about how one might be able 
to wait for a subprocess started without waiting. Sometimes I need to 
run a subprocess :wait nil and yet later be able to get its exit 
status... Esp. if I want to start a lot of them and not have zombies 
floating around... ;-)

Just a thought...

<D\'gou

---

From: Pascal Bourguignon <[email-masked]>
Date: Wed Jun 4 08:58:04 2003

> status... Esp. if I want to start a lot of them and not have zombies 

Use linux:wait or linux:waitpid (with the no hang option).

This  notion of  waiting on  a  processus is  strongly unix-related  I
think.  So using a POSIX, or in the lack of it, a unix or linux API is
acceptable.

-- 
__Pascal_Bourguignon__                   http://www.informatimago.com/
----------------------------------------------------------------------
Do not adjust your mind, there is a fault in reality.

---

From: Douglas Philips <[email-masked]>
Date: Wed Jun 4 09:03:23 2003

On Wednesday, Jun 4, 2003, at 11:55 US/Eastern, Pascal Bourguignon 
> think.  So using a POSIX, or in the lack of it, a unix or linux API is

Agreed.

Unfortunately last I tried to build the CVS head on Mac OS/X it didn't 
build.
I'm running 2.29 (under Jaguar 10.2.5) and don't have that function. 
Sigh.
I didn't have time to track down the build failure before. Maybe this 
weekend I'll try again, but maybe not.

-Doug

---

From: Sam Steingold <[email-masked]>
Date: Wed Jun 4 09:33:08 2003

> * In message <m3y90hzv8l.fsf@totally-fudged-out-message-id>
> * On the subject of "Re: Re[2]: [clisp-list] Separate Unix error stream keyword for ext:run-shell-command"
> * Sent on Wed, 4 Jun 2003 11:18:52 -0400
> * Honorable Douglas Philips <[email-masked]> writes:
>
> I'm wondering, did I miss a message or five about how one might be
> able to wait for a subprocess started without waiting. Sometimes I
> need to run a subprocess :wait nil and yet later be able to get its
> exit status... Esp. if I want to start a lot of them and not have
> zombies floating around... ;-)

good point.

Arseny, could you please add a function EXT:WAIT, based on
WaitForSingleObject() on woe32 and waitpid() on Unix
<http://www.opengroup.org/onlinepubs/007904975/functions/wait.html>
[I do not have access to woe32 docs, so I have to pass this to you :-)
with the following signature:

(WAIT pid &key :hang-p)

returning 1 or 2 values:
if the process exited, then
        value1 = :EXIT
        value2 = status
if the process was signalled, then
        value1 = :SIGNAL
        value2 = signal number
if the process was stopped, then
        value1 = :STOP
        value2 = signal number
if the process was continued,
        value1 = :CONTINUE

of course we will need to accomodate the differences between woe32 and
unix.

I suggest that we use this function in (LAUNCH ... :WAIT NIL)

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Bill Gates is great, as long as `bill' is a verb.

---

From: Arseny Slobodjuck <[email-masked]>
Date: Wed Jun 4 15:27:09 2003

Hello Sam,

Thursday, June 05, 2003, 2:31:47 AM, you wrote:

> Arseny, could you please add a function EXT:WAIT

> of course we will need to accomodate the differences between woe32 and

Give me a handle and I... I believe we need a lisp type 'process'
being returned from LAUNCH and also WAIT, KILL, EXITCODE, whatever for
it. But I have never created lisp types from scratch.

As about run-program/run-shell-command: is the ability of creating
pipe-streams so critical ? How widely it is used ? It's break all the
simplicity of the function interface.

-- 
Best regards,
 Arseny

---

From: Sam Steingold <[email-masked]>
Date: Wed Jun 4 15:45:24 2003

Hello Arseny,

> * In message <[email-masked]>
> * On the subject of "Re[4]: Separate Unix error stream keyword for ext:run-shell-command"
> * Sent on Thu, 5 Jun 2003 09:31:08 +1000

why isn't process handle / PID enough?

> simplicity of the function interface.

pipes are absolutely crucial.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
A man paints with his brains and not with his hands.

---

From: Kaz Kylheku <[email-masked]>
Date: Wed Jun 4 15:55:12 2003

> pipes are absolutely crucial.

Without pipes, you can't do any Unix systems programming with CLISP;
the kind of stuff that is done with awk, perl, etc.

Pipes are used Meta-CVS, for instance, to read the output of ``cvs log'' 
on a file, which contains all kinds of useful information that can be
parsed out.  The ``mcvs merge'' command critically depends on this; it
reads the list of tags to figure out what has been merged last and how
to tag the module to delimit the present merge.

---

From: Arseny Slobodjuck <[email-masked]>
Date: Wed Jun 4 16:48:09 2003

Hello Sam,

> why isn't process handle / PID enough?
You mean passing (uint)HANDLE as fixnum to lisp ?
Not abstract enough to my taste... No typecheck... Not extendable.
No guarantee we will not want to store additional info some day. Users
would do (loop for i from 0 to 10000 do (kill i)).

Possibly I don't realize the overhead of a new type. Is it too big ?

> pipes are absolutely crucial.

Why not to extend make-pipe-*-stream instead (trying to keep the
code reused) ? Another way I like even more - to make unopened
make-pipe-*-stream and pass it as stream to launch - how do you like
it ?

-- 
Best regards,
 Arseny

---

From: Sam Steingold <[email-masked]>
Date: Wed Jun 4 18:03:02 2003

> * In message <[email-masked]>
> * On the subject of "Re: [clisp-list] Re: Re[4]: Separate Unix error stream keyword for ext:run-shell-command"
> * Sent on Thu, 5 Jun 2003 10:53:29 +1000

yeah...

> No guarantee we will not want to store additional info some day.

like what?  a process is a just a process ID.
all other information can be accessed via a system call.

> Users would do (loop for i from 0 to 10000 do (kill i)).

why not? :-)
they will be able to write /etc/rc.d/* in CLISP!

if you are worried about "safety and security",
then don't run CLISP as root.

> Possibly I don't realize the overhead of a new type. Is it too big ?

you need to add a bunch of objects in constobj.d, modify io.d,
predtype.d, and a zillion other places.

If we do decide to add such a new type, we should have a generic
OS-HANDLE type which will point to an FD or PID or whatever else.

> Why not to extend make-pipe-*-stream instead (trying to keep the code
> reused) ?

we already have create_input_pipe() and friends, they should be called
by LAUNCH when necessary.
Except that they use shell and LAUNCH should not use shell, unless
given the :INDIRECT T argument.

> Another way I like even more - to make unopened make-pipe-*-stream and
> pass it as stream to launch - how do you like it ?

I don't think I like this.
how is
(launch ... :input (make-pipe-input-stream))
better than
(launch ... :input :stream)
?

if the only thing you can do with the return value of
MAKE-PIPE-INPUT-STREAM is pass it to LAUNCH, then this is not a good
idea, IMO.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Stupidity, like virtue, is its own reward.

---

From: Adam Warner <[email-masked]>
Date: Wed Jun 4 23:31:02 2003

>      returns: exit code (zero when (nullp wait))  */

Thanks for this info. I've now built and installed an i386 Debian
package of the CVS version of CLISP and can confirm that this works!

But so far I've only been able to grok the syntax for file streams.
Here's an example:

(with-open-file
    (stream "file.txt" :direction :output :if-exists :supersede)
  (ext::launch "ls" :arguments '("-1" "does-not-exist")
               :error stream :wait t :priority 0))

The file "does-not-exist" does not exist and the error output is written
to "file.txt" instead of appearing in the teminal.

Is there any way to currently avoid output to a file without outputting
to a terminal? As the documentation says only file streams and terminal
streams are supported it explains why this is not possible:

[31]> (setf a (make-string-output-stream))
#<output string-output-stream>
[32]> (stream-element-type a)
character
[33]> (ext::launch "ls" :arguments '("-1" "does-not-exist") :error a
:wait t :priority 0))

*** - ext::launch: argument #<output string-output-stream> does not
contain a valid OS stream handle
The following restarts are available:
USE-VALUE      :R1      You may input a value to be used instead.

Many thanks,
Adam

---

From: Sam Steingold <[email-masked]>
Date: Thu Jun 5 07:19:07 2003

> * In message <[email-masked]>
> * Sent on 05 Jun 2003 18:29:54 +1200

this is the intended behavior.

> Is there any way to currently avoid output to a file without
> outputting to a terminal?

:ERROR NIL is not supported yet, so, I guess, you will need to do

(with-open-file (null "/dev/null" :direction :output :if-exists :append)
  (launch ... :error null))

how should :ERROR NIL be implemented?
should we open /dev/null and remember the FD?
what is the standard way?

> As the documentation says only file streams and terminal

1. what is not clear in the above?
   please remember that neither Arseny nor myself (nor Bruno, for that
   matter) is a native English speaker, so you should be doubly careful
   with your English so that we will understand you.  thanks.

2. how should (launch ... :output (make-string-output-stream)) be
   implemented?  with a (named?) pipe from which a new CLISP thread
   will read and copy into the string stream?

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
The paperless office will become a reality soon after the paperless toilet.

---

From: Adam Warner <[email-masked]>
Date: Thu Jun 5 09:42:02 2003

> this is the intended behavior.

Yes. I was only describing a working example.

>   (launch ... :error null))

Thanks for the tip.

> what is the standard way?

I don't know what the standard way is.

> 1. what is not clear in the above?

Your language is very clear thanks. You appear to have confirmed that
currently the only way to grab the stream contents is by writing to the
filesystem and then reading the file back in.

>    will read and copy into the string stream?

My preferred mental model would copy however it is implemented at the
command line:

$ command > output
            ^ named stream

One would create a symbol name for the stream and then use READ-CHAR,
READ-BYTE, READ-LINE, etc. on the symbol name to read from the stream
(depending upon whether the stream is created as a binary or character
stream).

Say I knew the output stream was encoded as UTF-8. I would name a UTF-8
string stream and then pass that name to launch. launch would copy the
relevant output stream into the string stream to be read by the user.

Better syntax might be (for character streams):

(ext:launch ... :output (stream-name (ext:make-encoding :charset
'charset:utf-8 :line-terminator :unix))

And something like this for binary streams:

(ext:launch ... :output (stream-name '(unsigned-byte 8))

Regards,
Adam

---

From: Arseny Slobodjuck <[email-masked]>
Date: Thu Jun 5 17:00:05 2003

Hello Adam,

Thursday, June 05, 2003, 4:29:54 PM, you wrote:

> to a terminal?

Currently there is not, but I'll try to make it.

-- 
Best regards,
 Arseny

---

From: Arseny Slobodjuck <[email-masked]>
Date: Thu Jun 5 17:04:05 2003

Hello Sam,

> all other information can be accessed via a system call.
For perfect oses it is, but there's always various quirks.

> MAKE-PIPE-INPUT-STREAM is pass it to LAUNCH, then this is not a good

No! It's just the value return method. LAUNCH initializes (second half
of initialization) pipe-stream, then the user works with it.

(setf stream (make-pipe-input-stream))
(launch ... :output stream) ;initializes stream
(read stream)

As I undestand, pipe-input-stream should be supplies as :output
argument since the program outputs to it and lisp side inputs.

What I want (user) to avoid - to parse output value lists depending on
parameters supplied. I.e. if :input was :create then we get 2 values.
First value is pid if only :wait was nil. If :wait wasn't nil...
BTW, how pipes will work with :wait t.

-- 
Best regards,
 Arseny

---

From: Sam Steingold <[email-masked]>
Date: Thu Jun 5 19:30:05 2003

> * In message <[email-masked]>
> * On the subject of "Re[2]: [clisp-list] Re: Re[4]: Separate Unix error stream keyword for ext:run-shell-command"
> * Sent on Fri, 6 Jun 2003 10:37:31 +1000

what if the user tries to read from stream before launch inits it?!

(launch ... :output (:stream :element-type ...))

is so much cleaner!

> First value is pid if only :wait was nil. If :wait wasn't nil...

this is inevitable.
if :WAIT was T, then the values are the same as those of EXT:WAIT
(specified in <http://article.gmane.org/gmane.lisp.clisp.general/7089>)
if :WAIT was NIL, the values are PID, INPUT stream (or NIL), OUTPUT (or
NIL) and ERROR (or NIL)

> BTW, how pipes will work with :wait t.

they won't.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Linux - find out what you've been missing while you've been rebooting Windows.

---

From: Arseny Slobodjuck <[email-masked]>
Date: Fri Jun 6 07:24:17 2003

Hello Sam,

> what if the user tries to read from stream before launch inits it?!

As usual - error. It's easy to detect this situation.

> is so much cleaner!

So we need one stream init parser in launch, other in
make-pipe-*-stream - code duplication.

> this is inevitable.
Why ? I do suggest the method.

> if :WAIT was NIL, the values are PID, INPUT stream (or NIL), OUTPUT (or

I'm sure other users heat this as much as I. How many other lisp functions
change their return value count depending on input parameters ?

-- 
Best regards,
 Arseny

---

From: Sam Steingold <[email-masked]>
Date: Fri Jun 6 09:53:07 2003

> * In message <[email-masked]>
> * On the subject of "Re[4]: [clisp-list] Re: Re[4]: Separate Unix error stream keyword for ext:run-shell-command"
> * Sent on Sat, 7 Jun 2003 01:28:23 +1000

no, make-pipe-*-stream will be implemented in lisp, not C, and will
just pass their stream arguments to launch.

> Why ? I do suggest the method.

the half-baked pipe streams are out - I see no reason to create an
object which can only be only use in one single way - as an argument to
LAUNCH.   If you insist, you can consider the list (:STREAM
:ELEMENT-TYPE ... :EXTERNAL-FORMAT ... :BUFFERED ...) to be this kind
of stream.

I prefer the lispy/functional style of returning values to the C style
of modifying the arguments.

Unless you get Bruno to override me, I will resist that to death.

> I'm sure other users heat this as much as I. How many other lisp
> functions change their return value count depending on input
> parameters ?

when :WAIT is :T, we can return two additional NIL values just to make
sure that we always return 4 values. :-)

Consider FIND-SYMBOL: when the symbol is not found, it returns NIL, NIL,
even though the first value is sufficient (just check that the string is
not "NIL").

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat9 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
The only guy who got all his work done by Friday was Robinson Crusoe.

---

From: Douglas Philips <[email-masked]>
Date: Sat Jun 7 01:38:03 2003

On Friday, Jun 6, 2003, at 11:28 US/Eastern, Arseny Slobodjuck wrote:

>> (specified in 
>> <http://article.gmane.org/gmane.lisp.clisp.general/7089>)
> functions

Changing the subject (line, literally!). I don't think this is problem. 
Once you know that up to n values can be returned, you need some kind 
of multiple-value-bind sort of expression to receive them. Since 
returning fewer values than are being received means the "missing" ones 
are set to NIL, and since you have to look at the first one(s) to see 
if later ones are valid anyways, I don't think this is a big deal. A 
bigger deal is making sure that the first value returned is useful even 
when the others are discarded (because the expression returns into a 
single value receiving place).

Just my buck two-fifty, ;-)
    <D\'gou