How do Unix pipes work?

This article promotes bad practices for dealing with SIGPIPE.<p>1. Closing stderr in Python is not a good idea because that’ll swallow any other errors that occur at exit. Redirecting stdout to devnull is really just a way to prevent the flushed output from going to the now-closed stdout and triggering another SIGPIPE. That’s more preferable than closing stderr and losing error output at exit.<p>2. Ignoring SIGPIPE is a terrible idea for a process that should do stream processing. Try making a yes clone and ignoring SIGPIPE - your process will likely run forever trying to shove “y” into a closed pipe. There’s a reason SIGPIPE was invented! Very few programs bother to check the return value from write&#x2F;printf&#x2F;etc.

<p><pre><code> If we cat this file, it will be printed to the terminal. &gt; cat brothers_karamazov.txt ... many lines of text! ***FINIS*** It takes a noticeable amount of time to finish. </code></pre> The amount of time it takes for cat(1) to read and output the file is almost certainly insignificant. The time the author is noticing is probably related to how long it takes for his <i>console</i> to process the text.

This article only shows basic usage of pipes (this is what they mean by &quot;how pipes works&quot;), but doesn&#x27;t explain at all &quot;how pipe works&quot; (as in: how are they implemented).

I had a nice surprise and learning experience, when I discovered that the output of<p><pre><code> (echo red; echo green 1&gt;&amp;2) | echo blue </code></pre> is indeterministic:<p><a href="http:&#x2F;&#x2F;www.gibney.de&#x2F;the_output_of_linux_pipes_can_be_indeter" rel="nofollow">http:&#x2F;&#x2F;www.gibney.de&#x2F;the_output_of_linux_pipes_can_be_indete...</a><p>As it turns out, this short line and its behavior nicely demonstrate a bunch of aspects that happen under the hood when you use a pipe.

See Section 1.2 this &amp; 1.3 of the MIT Unix teaching OS for a great intro to FDs and pipes: <a href="https:&#x2F;&#x2F;pdos.csail.mit.edu&#x2F;6.828&#x2F;2019&#x2F;xv6&#x2F;book-riscv-rev0.pdf" rel="nofollow">https:&#x2F;&#x2F;pdos.csail.mit.edu&#x2F;6.828&#x2F;2019&#x2F;xv6&#x2F;book-riscv-rev0.pd...</a>

Here&#x27;s something that you should remember about using pipes and fork(2) in Python 3: By default, O_CLOEXEC is passed to the pipe(2) system from the CPython runtime.<p>This means, that reading the read end of the side in the parent process after you forked will not work. Thefore you should explicitly change fctl flags and remove os.O_CLOEXEC:<p><pre><code> fcntl.fcntl(readfd, fcntl.F_SETFL, fcntl.fcntl(readfd, fcntl.F_GETFL) &amp; ~os.O_CLOEXEC)</code></pre>

My own rule of thumb of whether or not to ignore SIGPIPE is simple:<p>* If you only deal with file descriptors provided to you (stdin, stdout, stderr) as well as some files that you open (including special files like FIFOs), do not ignore SIGPIPE.<p>* If you deal with sophisticated file descriptors (socket(2) and pipe(2) count as sophisticated), you&#x27;d better ignore SIGPIPE, but also make sure to check for EPIPE in every single write.<p>In my view, SIGPIPE is a kludge so that programs that are too lazy to check for errors from write(2) (and fwrite(3) and related friends) will not waste resources. But if you are dealing with sophisticated file descriptors, there is a lot more happening than just open&#x2F;read&#x2F;write and a lot more error cases you must handle, and at that point the incremental cost of handling EPIPE isn&#x27;t a significant addition.

My favorite sentence from Brian Kernighan&#x27;s latest book &quot;UNIX A History and a Memoir&quot;: Pipes are the quintessential Unix invention, an elegant and efficient way to use temporary connections of programs .. so I&#x27;ll read this article :-)

Another point where you have to ignore SIGPIPE is concurrent code that handles multiple fds (say, like a web server). In this case you <i>have to</i> ignore the signal and process EPIPE correctly, because the signal is not associated with a particular fd so you cannot tell which one of them failed.