Why “process substitution” is a late feature in Unix shells

Fun fact, in bash you lose the exit codes of process subs. It doesn&#x27;t even wait() on those processes in many cases.<p>So there is no way to abort a bash script if something like &lt;(sort nonexistent) fails.<p>OSH lets you opt into stricter behavior:<p><pre><code> $ osh -c set -e shopt -s oil:basic diff &lt;(sort left) &lt;(sort nonexistent) echo &quot;should not get here&quot;&#x27; &#x27; sort: cannot read: nonexistent: No such file or directory diff &lt;(sort left) &lt;(sort nonexistent) ^~ [ -c flag ]:1: fatal: Exiting with status 2 (command in PID 29359) </code></pre> In contrast, bash will just keep going and ignore failure.<p>You can also get all the exit codes with @_process_sub_status, which is analogous to PIPESTATUS.<p>(I should probably write a blog post about this: <a href="http:&#x2F;&#x2F;www.oilshell.org&#x2F;blog&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.oilshell.org&#x2F;blog&#x2F;</a>)

A fun trick is using the `paste` command with process substitution to combine lines of output from different commands.<p><pre><code> .&#x2F;cmd1</code></pre> outputs:<p><pre><code> a b c .&#x2F;cmd2</code></pre> outputs:<p><pre><code> 1 2 3 paste &lt;(.&#x2F;cmd1) &lt;(.&#x2F;cmd2)</code></pre> outputs:<p><pre><code> a 1 b 2 c 3 </code></pre> You can then use xargs to use each line as arguments for another command:<p><pre><code> paste &lt;(.&#x2F;cmd1) &lt;(.&#x2F;cmd2) | xargs -L1 .&#x2F;cmd3</code></pre> calls:<p><pre><code> .&#x2F;cmd3 a 1 .&#x2F;cmd3 b 2 .&#x2F;cmd3 c 3</code></pre>

I love zsh’s take on process substitution:<p><pre><code> cat =(echo foo) </code></pre> It’s almost identical to &lt;(echo foo), but crucially it drains the output before performing the substitution. It’s like &lt;(... | sponge), but somehow much more reliable.<p>I’ve used it in a few situations where process substation failed, though being old and feeble I can’t remember why it failed. But for example you can copy the file safely, knowing that it will contain the full output of the shell pipeline, unlike process substitution.<p>I don’t even know the name of =(...). Hmm. Sponge substitution?

I quite like how fish does process substitution.<p><pre><code> foo (bar | psub) </code></pre> Instead of syntax it&#x27;s just a plain function.<p>docs: <a href="https:&#x2F;&#x2F;fishshell.com&#x2F;docs&#x2F;current&#x2F;cmds&#x2F;psub.html" rel="nofollow">https:&#x2F;&#x2F;fishshell.com&#x2F;docs&#x2F;current&#x2F;cmds&#x2F;psub.html</a> source of psub.fish: <a href="https:&#x2F;&#x2F;github.com&#x2F;fish-shell&#x2F;fish-shell&#x2F;blob&#x2F;master&#x2F;share&#x2F;functions&#x2F;psub.fish" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;fish-shell&#x2F;fish-shell&#x2F;blob&#x2F;master&#x2F;share&#x2F;f...</a>

The usability of unixy shells generally falls down a cliff when you need to deal with more than just one input one output. The awkwardness in trying to shoehorn process substitution is just one of the examples of that.

Great read, but worth noting from the end of the article that &quot;late feature&quot; here means it was added in the early &#x27;90s. The late addition to Unix that surprises me is ssh. It was only invented in the late &#x27;90s. Before that everyone used unencrypted remote shells like telnet.<p>Encryption in general was in a pretty bad state in the &#x27;90s: original http was unencrypted and early mobile phone standards that are still in use have very weak encryption.

For whatever reason I can never remember the syntax of &lt;(command) and end up &quot;rediscovering&quot; it every year. It&#x27;s seldom used but when it&#x27;s needed it&#x27;s rather elegant.<p>Another somewhat related useful bash feature is using this form:<p><pre><code> wc &lt;&lt;&lt; &#x27;a b c&#x27; </code></pre> instead of:<p><pre><code> echo &#x27;a b c&#x27; | wc</code></pre>

Didn&#x27;t even know about process substitution, had been using fifo&#x27;s to achieve this!

The claim is false; process substitution can be cobbed together with named fifos,* and those are &quot;ancient&quot;.<p>Only problem that those are temporary objects that have to be created in the file system, and cleaned up.<p>However, temporary objects (files, no fifos) are also used in here doc implementations.<p>Process substitution is a late feature simply because the creativity juice in Unix (tm) dried up some time before the middle 1990&#x27;s, leaving the FOSS reimplementations of Unix to carry the development torch.<p>Those projects had to balance among other goals like quality&#x2F;robustness and compatibility.<p>(If we look at the quality of the FOSS tools compared to the Unix originals, we could also remark that &quot;quality and robustness was late in coming to Unix&quot;. But we equivocate on Unix, because GNU stands for GNU is Not Unix!)<p>Features appearing in FOSS utilities like GNU Bash take time to make into Unix (tm).<p>Process substitution is not yet in the standard, therefore it is not in in fact in Unix (tm).<p>Shell scripting is a conservative activity. The language isn&#x27;t very good and so improving it is like kicking a dead horse in some ways; the most important matter in any new shell release is that old scripts keep working. (Like configuration scripts for the build systems of nicer languages).<p>---<p>* See GNU Bash manual: <a href="https:&#x2F;&#x2F;www.gnu.org&#x2F;savannah-checkouts&#x2F;gnu&#x2F;bash&#x2F;manual&#x2F;bash.html#Process-Substitution" rel="nofollow">https:&#x2F;&#x2F;www.gnu.org&#x2F;savannah-checkouts&#x2F;gnu&#x2F;bash&#x2F;manual&#x2F;bash....</a>: <i>&quot; Process substitution is supported on systems that support named pipes (FIFOs) or the &#x2F;dev&#x2F;fd method of naming open files. &quot;</i>

In practice I end up caching the output often. I have used process substitution but the iteration process feels more useful to me if I&#x27;ve slowly built up data and I can inspect the internal pieces each time and reuse them in different ways.<p>But I can see if it&#x27;s relatively fast. I like it. I just don&#x27;t end up using it often.

0. Process substitution is a potential DoS vector as it could take up all of RAM and&#x2F;or disk space.<p>1. Also, not all commands are compatible with it, especially if they need rewinding or reopening. diff has issues with using it for both arguments often. It&#x27;s likely the use of memory mapped files, but I could be wrong.<p>2. Shells ought to implement a flag for process substitution to allow temporary files to reside on disk for the lifetime of the command line. This way, it can operate on extremely large files.

An unfortunate thing is that process substitution does not work in git bash on Windows. (at least it was the case last time I tested; googling around I found a random comment in random github repo saying it&#x27;s been fixed in 2.25 but I don&#x27;t have a laptop handy to test it now).

This is a great explanation! I wonder many times why we had to play with obscure xargs instead of being able to pipe a command output to an argument.

Which is the best she&#x27;ll, fish or zsh?&#x2F;&#x2F;