some problems with scope of fds in process substitution

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

some problems with scope of fds in process substitution

Stephane Chazelas-8
FYI,

as seen at https://unix.stackexchange.com/a/408171, there are
still a few "problems" with process substitution, where some fds
are closed where they probably shouldn't:

> Note that even with the latest (4.4.12 as of writing) version, bash still has a few bugs here like:
>
> $ bash -c 'eval cat <(echo test)'
> test # OK but:
> $ bash -c 'eval "echo foo;cat" <(echo test)'
> foo
> cat: /dev/fd/63: No such file or directory
> $ bash -c 'eval f=<(echo test) "; cat \$f"'
> cat: /dev/fd/63: No such file or directory
>
> and some still triggered by pipes like:
>
> $ cat file
> echo "$1"
> cat "$1"
> $ bash -c 'source ./file <(echo test)'
> /dev/fd/63
> test  # OK but:
> $ cat file2
> echo "$1" | wc -c
> cat "$1"
> $ bash -c 'source ./file2 <(echo test)'
> 11
> cat: /dev/fd/63: No such file or directory

Also, there's a lot of problems reported at
unix.stackexchange.com at least that are caused by bash not
waiting for the processes started by process substitutions,
especially the >(...) form.

It would be more useful IMO if bash waited for them like zsh
does (under some conditions) or ksh93 can be told do.

More details at:

https://unix.stackexchange.com/a/403788

--
Stephane

Reply | Threaded
Open this post in threaded view
|

Re: some problems with scope of fds in process substitution

Chet Ramey
On 12/1/17 2:00 PM, Stephane Chazelas wrote:

> Also, there's a lot of problems reported at
> unix.stackexchange.com at least that are caused by bash not
> waiting for the processes started by process substitutions,
> especially the >(...) form.

Bash always reaps these processes. Do you mean waiting for them
to terminate before executing the next command?


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    [hidden email]    http://cnswww.cns.cwru.edu/~chet/

Reply | Threaded
Open this post in threaded view
|

Re: some problems with scope of fds in process substitution

Stephane Chazelas-8
2017-12-03 17:31:00 -0500, Chet Ramey:
> On 12/1/17 2:00 PM, Stephane Chazelas wrote:
>
> > Also, there's a lot of problems reported at
> > unix.stackexchange.com at least that are caused by bash not
> > waiting for the processes started by process substitutions,
> > especially the >(...) form.
>
> Bash always reaps these processes. Do you mean waiting for them
> to terminate before executing the next command?
[...]

Hi Chet,

yes, that's what I meant as in:

$ rm -f a; bash -c 'echo test > >(cat > a); cat a'
$

In:

cmd1 <(cmd2)
cmd3

It's usually OK not to wait for cmd2, same as

cmd2 | cmd1
cmd3

where not all shells wait for cmd2 (before running cmd3).

as cmd1 will generally wait for cmd2 by virtue of it waiting on
eof on the pipe but in:

cmd1 >(cmd2)

It's more cmd2 that would be waiting for cmd1. So when cmd1
returns, often, cmd2 has not finished yet, and if cmd3 needs
something produced by cmd2, that's where we run into problems.

See the link I gave (https://unix.stackexchange.com/a/403788)
for more details including the situation with other shells that
implement process substitution (ksh, zsh, rc, es).

Cheers
Stephane

Reply | Threaded
Open this post in threaded view
|

Re: some problems with scope of fds in process substitution

Chet Ramey
On 12/3/17 6:07 PM, Stephane Chazelas wrote:

> 2017-12-03 17:31:00 -0500, Chet Ramey:
>> On 12/1/17 2:00 PM, Stephane Chazelas wrote:
>>
>>> Also, there's a lot of problems reported at
>>> unix.stackexchange.com at least that are caused by bash not
>>> waiting for the processes started by process substitutions,
>>> especially the >(...) form.
>>
>> Bash always reaps these processes. Do you mean waiting for them
>> to terminate before executing the next command?
> [...]
>
> Hi Chet,
>
> yes, that's what I meant as in:
>
> $ rm -f a; bash -c 'echo test > >(cat > a); cat a'
> $

Bash-4.4 allows you to wait for the last process substitution, since the
pid appears in $!, like ksh93. There's still no option to wait for more
than one, though I imagine I could make that work as part of `wait'
without arguments.

$ cat ~/x20a
rm -f a;
echo test > >(cat > a)
echo waiting for $!
wait $!
echo $?
cat a
$ ../bash-4.4-patched.rhe6/bash ~/x20a
waiting for 2762
0
test
$ ../bash-4.4-patched.rhe6/bash --version
GNU bash, version 4.4.12(2)-release (x86_64-unknown-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    [hidden email]    http://cnswww.cns.cwru.edu/~chet/

Reply | Threaded
Open this post in threaded view
|

Re: some problems with scope of fds in process substitution

Stephane Chazelas-8
2017-12-04 08:46:24 -0500, Chet Ramey:
[...]
> Bash-4.4 allows you to wait for the last process substitution, since the
> pid appears in $!, like ksh93.

Thanks,

I hadn't noticed it had changed in 4.4

One major differnce with ksh93 though is that it won't work with

cmd | tee >(cmd2)

unless you enable lastpipe.

In:

cmd | tee >(cmd2) | cmd3

you won't get access to cmd2's pid in $! either in ksh93, but
those are usually OK as cmd2's stdout also goes to the pipe to
cmd3, so "waited for" by virtue of cmd3 waiting for it.

In any case, one can always work around it without lastpipe by doing:

cmd | (tee >(cmd2); ret=$?; wait "$!" || exit "$ret")



> There's still no option to wait for more
> than one, though I imagine I could make that work as part of `wait'
> without arguments.
[...]

Yes, that would be useful and align with ksh93.

That could however break some work flows like

exec > >(tee logfile) # not supported by ksh93

cmd1 & cmd2 &
wait


Or:

{
  cmd1 & cmd2 &
  wait
} > >(tee logfile)

--
Stephane

Reply | Threaded
Open this post in threaded view
|

Re: some problems with scope of fds in process substitution

Chet Ramey
On 12/4/17 10:19 AM, Stephane Chazelas wrote:

> One major differnce with ksh93 though is that it won't work with
>
> cmd | tee >(cmd2)
>
> unless you enable lastpipe.

Of course not: that's the whole point of lastpipe, and the major
difference there is broader than $! and process substitutions.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    [hidden email]    http://tiswww.cwru.edu/~chet/