TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

The Beauty of Unix Pipelines

649 点作者 0x4FFC8F将近 5 年前

43 条评论

gorgoiler将近 5 年前
Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix:<p>(1) everything is <i>text</i><p>(2) everything (ish) is a <i>file</i><p>(3) including <i>pipes</i> and <i>fds</i><p>(4) every piece of software is accessible as a file, invoked at the <i>command line</i><p>(5) ...with local <i>arguments</i><p>(6) ...and persistent globals in the <i>environment</i><p>A lot of understanding comes once you know what execve does, though such knowledge is of course not necessary. It just helps.<p>Unix is seriously uncool with young people at the moment. I intend to turn that around and articles like this offer good material.
评论 #23421901 未加载
评论 #23421769 未加载
评论 #23423566 未加载
评论 #23422358 未加载
评论 #23422415 未加载
评论 #23423650 未加载
评论 #23421980 未加载
评论 #23422423 未加载
评论 #23422995 未加载
评论 #23423158 未加载
评论 #23422680 未加载
评论 #23429251 未加载
评论 #23424512 未加载
评论 #23425584 未加载
评论 #23426521 未加载
评论 #23421707 未加载
评论 #23460350 未加载
评论 #23422989 未加载
cowmix将近 5 年前
I use pipelines as much as the next guy but every time I see post praise how awesome they are, I&#x27;m reminded of the Unix Hater&#x27;s Handbook. Their take on pipelines is pretty spot on too.<p><a href="http:&#x2F;&#x2F;web.mit.edu&#x2F;~simsong&#x2F;www&#x2F;ugh.pdf" rel="nofollow">http:&#x2F;&#x2F;web.mit.edu&#x2F;~simsong&#x2F;www&#x2F;ugh.pdf</a>
评论 #23422030 未加载
评论 #23421913 未加载
评论 #23422127 未加载
评论 #23422967 未加载
评论 #23423585 未加载
评论 #23421698 未加载
评论 #23421814 未加载
评论 #23421876 未加载
评论 #23423257 未加载
评论 #23421364 未加载
atombender将近 5 年前
Pipes are a great idea, but are severely hampered by the many edge cases around escaping, quoting, and, my pet peeve, error handling. By default, in modern shells, this will actually succeed with no error:<p><pre><code> $ alias fail=exit 1 $ find &#x2F; | fail | wc -l; echo $? 0 0 </code></pre> You can turn on the &quot;pipefail&quot; option to remedy this:<p><pre><code> $ set -o pipefail $ find &#x2F; | fail | wc -l; echo $? 0 1 </code></pre> Most scripts don&#x27;t, because the option makes everything much stricter, and requires more error handling.<p>Of course, a lot of scripts also forget to enable the similarly strict &quot;errexit&quot; (-e) and &quot;nounset&quot; options (-u), which are also important in modern scripting.<p>There&#x27;s another error that hardly anyone bothers to handle correctly:<p><pre><code> x=$(find &#x2F; | fail | wc -l) </code></pre> This sets x to &quot;&quot; because the command failed. The only way to test if this succeeded is to check $?, or use an if statement around it:<p><pre><code> if ! x=$(find &#x2F; | fail | wc -l); then echo &quot;Fail!&quot; &gt;&amp;2 exit 1 fi </code></pre> I don&#x27;t think I&#x27;ve seen a script ever bother do this.<p>Of course, if you also want the error message from the command. If you want that, you have to start using name pipes or temporary files, with the attendant cleanup. Shell scripting is suddenly much more complicated, and the resulting scripts become much less fun to write.<p>And that&#x27;s why shell scripts are so brittle.
评论 #23424034 未加载
评论 #23424492 未加载
geophile将近 5 年前
I love pipelines. I don&#x27;t know the elaborate sublanguages of find, awk, and others, to exploit them adequately. I also love Python, and would rather use Python than those sublanguages.<p>I&#x27;m developing a shell based on these ideas: <a href="https:&#x2F;&#x2F;github.com&#x2F;geophile&#x2F;marcel" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;geophile&#x2F;marcel</a>.
评论 #23422708 未加载
评论 #23421672 未加载
评论 #23421331 未加载
ketanmaheshwari将近 5 年前
Unix pipelines are cool and I am all for it. In recent times however, I see that sometimes they are taken too far without realizing that each stage in the pipeline is a process and a debugging overhead in case something goes wrong.<p>A case in point is this pipeline that I came across in the wild:<p>TOKEN=$(kubectl describe secret -n kube-system $(kubectl get secrets -n kube-system | grep default | cut -f1 -d &#x27; &#x27;) | grep -E &#x27;^token&#x27; | cut -f2 -d&#x27;:&#x27; | tr -d &#x27;\t&#x27; | tr -d &quot; &quot;)<p>In this case, perhaps awk would have absorbed 3 to 4 stages.
评论 #23421811 未加载
评论 #23421556 未加载
评论 #23421624 未加载
评论 #23421729 未加载
评论 #23424467 未加载
评论 #23422069 未加载
russellbeattie将近 5 年前
I think there&#x27;s an interesting inflection point between piping different utilities together to get something done, and just whipping up a script to do the same thing instead.<p>First I&#x27;ll use the command line to, say, grab a file from a URL, parse, sort and format it. If I find myself doing the same commands a lot, I&#x27;ll make a .sh file and pop the commands in there.<p>But then there&#x27;s that next step, which is where Bash in particular falls down: Branching and loops or any real logic. I&#x27;ve tried it enough times to know it&#x27;s not worth it. So at this point, I load up a text editor and write a NodeJS script which does the same thing (Used to be Perl, or Python). If I need more functionality than what&#x27;s in the standard library, I&#x27;ll make a folder and do an npm init -y and npm install a few packages for what I need.<p>This is not as elegant as pipes, but I have more fine grained control over the data, and the end result is a folder I can zip and send to someone else in case they want to use the same script.<p>There is a way to make a NodeJS script listen to STDIO and act like another Unix utility, but I never do that. Once I&#x27;m in a scripting environment, I might as well just put it all in there so it&#x27;s in one place.
评论 #23422945 未加载
评论 #23425496 未加载
nojito将近 5 年前
Debugging Linux pipelines is not a fun experience.<p>This is one clear area where Powershell with its object model has got it right.
评论 #23424194 未加载
评论 #23424451 未加载
antipaul将近 5 年前
A similar philosophy has made the &quot;tidyverse&quot; a much-loved extension of the statistical language R.<p>Compare the following 2 equivalent snippets. Which one seems more understandable?<p><pre><code> iris_data %&gt;% names() %&gt;% tolower() %&gt;% gsub(&quot;.&quot;, &quot;_&quot;, ., fixed=TRUE) %&gt;% paste0(&quot;(&quot;, ., &quot;)&quot;) </code></pre> or:<p><pre><code> paste0(&quot;(&quot;, gsub(&quot;.&quot;, &quot;_&quot;, tolower(names(iris_data)), fixed=TRUE), &quot;)&quot;)</code></pre>
评论 #23425266 未加载
CalmStorm将近 5 年前
Unix pipelines are indeed beautiful, especially when you consider its similarity to Haskell&#x27;s monadic I&#x2F;O: <a href="http:&#x2F;&#x2F;okmij.org&#x2F;ftp&#x2F;Computation&#x2F;monadic-shell.html" rel="nofollow">http:&#x2F;&#x2F;okmij.org&#x2F;ftp&#x2F;Computation&#x2F;monadic-shell.html</a><p>Unix pipelines actually helped me make sense of Haskell&#x27;s monad.
评论 #23421961 未加载
评论 #23422768 未加载
nsajko将近 5 年前
Surprised there is no mention of Doug McIlroy: <a href="https:&#x2F;&#x2F;wiki.c2.com&#x2F;?DougMcIlroy" rel="nofollow">https:&#x2F;&#x2F;wiki.c2.com&#x2F;?DougMcIlroy</a>
lexpar将近 5 年前
Interesting comment in the header of this site.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;prithugoswami&#x2F;personal-website&#x2F;blob&#x2F;master&#x2F;layouts&#x2F;partials&#x2F;head.html" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;prithugoswami&#x2F;personal-website&#x2F;blob&#x2F;maste...</a>
评论 #23424112 未加载
rzmnzm将近 5 年前
Powershell is the ultimate expression of the Unix pipeline imo.<p>Passing objects through the pipeline and being able to access this data without awk&#x2F;sed incantations is a blessing for me.<p>I think anyone who appreciates shell pipelines and python can grok the advantages of the approach taken by Powershell, in a large way it is directly built upon existing an Unix heritage.<p>I&#x27;m not so good at explaining why, but for anyone curious please have a look at the Monad manifesto by Jeffrey Snover<p><a href="https:&#x2F;&#x2F;devblogs.microsoft.com&#x2F;powershell&#x2F;monad-manifesto-the-origin-of-windows-powershell&#x2F;" rel="nofollow">https:&#x2F;&#x2F;devblogs.microsoft.com&#x2F;powershell&#x2F;monad-manifesto-th...</a><p>You may not agree with the implementation, but the ideas being it, I think, are worth considering.
akavel将近 5 年前
I think it will be <i>on topic</i> if I let myself take this occasion to once again plug in a short public service announcement of an open-source tool I built, that helps interactively build Unix&#x2F;Linux pipelines, dubbed &quot;The Ultimate Plumber&quot;:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;akavel&#x2F;up&#x2F;" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;akavel&#x2F;up&#x2F;</a><p>I&#x27;ve also recently seen it being described in shorter words as a &quot;sticky REPL for shell&quot;. Hope you like it, and it makes your life easier!
评论 #23429112 未加载
tekert将近 5 年前
To my understanding, this is the same pattern where every &quot;object&quot; outputs the same data type for other &quot;objects&quot; to consume. This pattern can have a text or gui representation that is really powerful in its own nature if you think about it, its why automation agents with their events consumption&#x2F;emition are so powerful, its why the web itself shifts towards this pattern (json as comunication of data, code as object), The thing is, this will always be a higher level of abstraction, i think that a gui of this pattern should exist as a default method in most operating systems, its would solve a lot learning problems like learning all the names and options of objects, i would be the perfect default gui tool, actually, sites like zapier, or tools like huginn already do this pattern, i always wondered why this pattern expands so slowly being so useful.
tarkin2将近 5 年前
I wish there were gui pipes. Pipes are wonderful but they’re neither interactive nor continuous.<p>Pipes that loop, outputting a declarative gui text format, and listen for events from the gui, would be marvellous.<p>I can’t think how to do that without sockets and a bash loop. And that seems to create the kind of complexity that pipes manage to avoid.
评论 #23426366 未加载
benjaminoakes将近 5 年前
I find gron to be much more Unix-y than jq. It &quot;explodes&quot; JSON into single lines for use with grep, sed, etc and can recombine back into JSON as well.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;TomNomNom&#x2F;gron" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;TomNomNom&#x2F;gron</a>
评论 #23427442 未加载
parliament32将近 5 年前
Related: Pipelines can be 235x faster than a Hadoop cluster <a href="https:&#x2F;&#x2F;adamdrake.com&#x2F;command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html" rel="nofollow">https:&#x2F;&#x2F;adamdrake.com&#x2F;command-line-tools-can-be-235x-faster-...</a>
theshadowmonkey将近 5 年前
Pipes are like one of the best experiences you’ll have whatever you were doing. I was debugging a remote server logging millions of Logs a day and was aggregating a little on the server. Then all it required was wget, jq, sed and awk. And I had a powerful log analyzer than splunk or any other similar solution on a developer Mac. Which you think is awesome when you’re paying a fortune to use Splunk. And for getting some insights quick, Unix pipes are a godsend.
sedatk将近 5 年前
It&#x27;s ironic that the article ends with Python code. You could have done everything in Python in the first place and it would have probably been much more readable.
评论 #23425212 未加载
评论 #23424114 未加载
enriquto将近 5 年前
If you eval this simple pipeline<p><pre><code> file -b `echo $PATH:|sed &#x27;s&#x2F;:&#x2F;\&#x2F;* &#x2F;g&#x27;`|cut -d\ -f-2|sort|uniq -c|sort -n </code></pre> it prints a histogram of the types of all the programs on your path (e.g., whether they are shell, python, perl scripts or executable binaries). How can you ever write such a cute thing in e.g., python or, god forbid, java?
评论 #23426412 未加载
评论 #23423434 未加载
stevefan1999将近 5 年前
I think it is terrible, as if everything is a text you can&#x27;t precisely describe some data structure, e.g. circular list.
praveen9920将近 5 年前
I recently wrote a golang code for fetching rss feed and displaying gist based on requirement.<p>Looking at this code, I am tempted to reimplement this using pipes but saner mind took over and said &quot;don&#x27;t fix something that is not broken&quot;<p>I probably would be still do it and get some benchmark numbers to compare both.
hi41将近 5 年前
Thank you! I did not know you get to a &quot;sql like group by&quot; using uniq -c. That&#x27;s so cool! I think I used to pipe it to awk and count using an array and then display but your method is far better than mine.
评论 #23423141 未加载
Silamoth将近 5 年前
That was a great article! Pipes can definitely be very powerful. I will say, though, that I often find myself reading pages of documentation in order to actually get anything with Unix and its many commands.
miclill将近 5 年前
Great write up. One thing I would add is how pipes do buffering &#x2F; apply backpressure. To my understanding this is the &quot;magic&quot; that makes pipes fast and failsafe(r).
not2b将近 5 年前
I&#x27;ve been using pipes for decades to get my work done, but it was cool to learn about jq as I have much less experience with JSON. It&#x27;s a very cool program. Thanks.
sigjuice将近 5 年前
Minor nitpick: It is Unix &quot;pipes&quot; (not pipelines).
评论 #23426516 未加载
kazinator将近 5 年前
&gt; <i>If you append &#x2F;export to the url you get the list in a .csv format.</i><p>Text filtering approach narrowly rescued by website feature.<p>Phew, that was close!
stormdennis将近 5 年前
The video from 1982 is brilliant great explanations from an era before certain knowledge was assumed as generally known
mehrdadn将近 5 年前
What kills me about pipelines is when I pipe into xargs and then suddenly can&#x27;t kill things properly with Ctrl+C. Often I have to jump through hoops to parse arguments into arrays and avoid xargs just for this. (This has to do with stdin being redirected. I don&#x27;t recall if there&#x27;s anything particular about xargs here, but that&#x27;s where it usually comes up.)
jakubnarebski将近 5 年前
The first example in the article is what `git shortlog -n` does; no need for the pipeline.
JadeNB将近 5 年前
What does it mean to say that the video shows &quot;Kernighan being a complete chad&quot;?
评论 #23421007 未加载
评论 #23421275 未加载
评论 #23427649 未加载
评论 #23421255 未加载
0xsnowcrash将近 5 年前
Aka The Ugliness of Powershell. Would be fun to see equivalents to these in Powershell.
nickthemagicman将近 5 年前
I have problems understanding what commands I can pipe. Some work some don&#x27;t.
评论 #23421338 未加载
评论 #23421785 未加载
评论 #23422240 未加载
unnouinceput将近 5 年前
Cygwin and you can do exactly this on Windows too. For me Cygwin is a bless.
hbarka将近 5 年前
abinitio.com was borne from these principles.
Dilu8将近 5 年前
Good article. Thanks for information
staycoolboy将近 5 年前
The first time I saw pipes in action I was coming from Apple and PC DOS land. It blew my mind. The re-usability of so many &#x2F;bin tools, and being able to stuff my own into that pattern was amazing.<p>If you like pipes and multimedia, checkout gstreamer, it has taken the custom pipeline example to real-time.
known将近 5 年前
The capacity of pipe buffer is important for doing serious producer|consumer work;
billfor将近 5 年前
I think systemd can do all of that.
niko0221将近 5 年前
Esta genial Laexplicacion. Se aprende mucho aqui Soy nuevo y este articulo es bastante genial.
fogetti将近 5 年前
I am sorry for playing the devil&#x27;s advocate. I also think that pipes are extremely useful and a very strong paradigm, and I use them daily in my work. Also it is not an accident that it is fundamental and integral part of powershell too.<p>But is this really HN top page worthy? I have seen this horse beaten to death for decades now. These kind of articles have been around since the very beginning of the internet.<p>Am I missing something newsworthy which makes this article different from the hundreds of thousands of similar articles?
评论 #23423514 未加载
adamnemecek将近 5 年前
Unix pipes are the a 1970&#x27;s construct, the same way bell bottom pants are. It&#x27;s a construct that doesn&#x27;t take into account the problems and scale of today&#x27;s computing. Unicode? Hope your pipes process it fine. Video buffers? High perf? Fuggetaboutit. Piping the output of ls to idk what? Nice, I&#x27;ll put it on the fridge.
评论 #23426504 未加载
评论 #23424203 未加载