TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Parallel Shells with Xargs

61 pointsby oedmarapover 4 years ago

5 comments

saagarjhaover 4 years ago
You can actually do this with GNU Paral…oh, wait, I forgot that this isn't Stack Overflow and I'm not Ole. But seriously, xargs is pretty powerful by itself; I've seen a number of awkward pipelines using heavyweight tools when xargs would have sufficed.
评论 #26188972 未加载
philsnowover 4 years ago
I bristle when I have to switch from a shell for loop to piping jobs to xargs. For loops are like infix notation (the operator&#x2F;operation kind of goes in the middle) and xargs is postfix (the op goes on the end).<p><pre><code> for file in *pdf *azw3; do calibre-convert $file $file.epub; done </code></pre> vs<p><pre><code> find . -name \*pdf -o -name \*azw3 -print0 | xargs -I {} calibre-convert {} {}.epub </code></pre> Something about the former is more natural for my brain to produce. If I want to parallelize, for shell loops my muscle memory very quickly converts the above to<p><pre><code> for file in *pdf *azw3; do calibre-convert $file $file.epub &amp; done; wait </code></pre> ... but if I have a million files my machine is going to have a bad day, whereas the xargs version is easy to limit to N parallel invocations.<p>I&#x27;d really like a shell that lets me define my own sugared versions of shell grammar. I imagine writing a &#x27;forp&#x27; that would let me do<p><pre><code> forp $(nproc) file in ...; do ...; done </code></pre> and the shell would run up to $(nproc) invocations in parallel. Anybody know of such a shell? (Don&#x27;t say eshell.)
评论 #26188038 未加载
评论 #26190654 未加载
gjvcover 4 years ago
More often than not, you&#x27;ll get a better result (often more safely) by writing a small thing which generates the complete command lines for all the tasks which need running. Then once you are happy the right thing will be done, feed the list of commands to GNU parallel or something else which does the same thing.<p>Not only does this approach provide confidence before pulling the trigger, but the list of commands may be retained forever for reference &#x2F; example. It&#x27;s also clearer and doesn&#x27;t get the reader mystified by &quot;shell wizardry&quot;, as some people are wont to be.
LukeShuover 4 years ago
The phrase &quot;POSIX.2&quot; is archaic. If POSIX is IEEE Std 1003, then &quot;POSIX.1&quot; is IEEE Std 1003.1, &quot;POSIX.2&quot; is IEEE Std 1003.2, and so on.<p>In the first revisions, 1003.1 specified &quot;Core Services&quot; (mostly the C language stuff), and 1003.2 specified &quot;Shell and Utilities&quot;. But when the Austin Group formed in 1997, they decided to roll the shell and utilities in to 1003.1. There no longer is a .2; it&#x27;s been deprecated by 1003.1-2001 and all revisions since then.
评论 #26188714 未加载
dhashover 4 years ago
Kind of embarrassing, but I never actually learned bash well enough to use it even after a decade of programming. When there was only one pair of feet in the sand, that’s where xargs carried me. Everything from web scrapers, to kubernetes admin, I’ve &#x2F;only&#x2F; used xargs to do all of the variable splicing and parallelism that I need.