This was quite interesting to look through!<p>Perl 5.8.0 is over 20 years old (<a href="https://dev.perl.org/perl5/news/2002/07/18/580ann/" rel="nofollow">https://dev.perl.org/perl5/news/2002/07/18/580ann/</a>) while centOS 3.9 was released in 2007! At the same time it seems not-that-old and ancient.<p>My personal anecdote with gnu parallel was running into it while working in academia. It worked well and saved me some time, but I felt that it was unreasonable of a tool to ask for a citation to parallelise a script - it seemed that matplotlib, jupyter and co would need one as well.
On the other hand, I decided to not use it, because I also feel that authors can ask for whatever they want.
Wait what: `parallel` is a Perl script!? [1]<p>I would have thought it's black magic with assembler optimisations for MIPS and special considerations for HP-UX...<p>This is such a lovely and interesting writeup, it's wonderful that people take their time to share so generously!<p>[1] : an 11k loc petal script, you can read along here: <a href="https://github.com/gitGNU/gnu_parallel/blob/master/src/parallel">https://github.com/gitGNU/gnu_parallel/blob/master/src/paral...</a>
I found GNU parallel useful when I wanted to queue up transcoding of flac files to mp3 on my Raspberry Pi. A few ffmpeg flags plus a list of files meant I could easily just saturate one job per core with a one-line bash command.
parallel is a tool I've reached for many times; the citation bit it prints is odd - it seems to assume that the general use case is research/academic - but easily squelched.<p>A sample use case would be having a file that has words in it, one per line, and you want to run a program that operates on each word (device name, dollar amount, whatever). Sure, you can use a loop, but if the words and actions are independent, parallel is one way to spin up N copies of your program and pass it a single word from the file. Can get around Python's GIL without having to use multiprocessing or threads (as a more concrete example).<p>Didn't realise that it busy waits, but I'm typically running it on a not very busy server with tens of cores.
Before GNU Parallel I used to use Ruby's workers and job queue to keep ${N} cores busy with work. It sorta worked like GNU parallel but was quite basic. I've since switched to using GNU Parallel. Stable code I don't have to write doesn't have to be maintained... not to mention it has more features than I normally supported.
I couldn’t make heads or tails of what this would be useful for from the OP (maybe it’s something I should already have known), but this from the official site was pretty helpful: <a href="https://www.gnu.org/software/parallel/parallel_cheat.pdf" rel="nofollow">https://www.gnu.org/software/parallel/parallel_cheat.pdf</a>
I once replaced a 10 machine Hadoop cluster job with a python script and parallel on my laptop because I didn't want to wait for hours for it to finish.<p>The i7 on my laptop with quite a few CPUS/threads and a few optimisations got the job finished in 10 minutes.<p>(I later put the Hadoop use on my resume, not the GNU parallel. That's the joke of modern job hunting. There is no interested in what you did, just buzzwords and leetcode. Luckily there are still a few places that value real work or I'd be too old to get a job. :) )
If anyone needs a pretty basic alternative with Windows support, there's Rush:<p><a href="https://github.com/shenwei356/rush">https://github.com/shenwei356/rush</a><p>I use it pretty extensively with ffmpeg, imagemagick and the like.<p>I'd been using the mmstick/parallel for a while, but it moved to RedoxOS repos and then stopped being updated, while still having some issues not ironed out.<p><a href="https://github.com/shenwei356/rush">https://github.com/shenwei356/rush</a>
Parallel is a fun tool. I use it as a sort of simple slurm to distribute work over many VMs to process tens to hundreds of TBs of data. Sometimes across 2400+ cores.
I've never been sure if it's too much of a hack, but I've used GNU parallel in Docker containers as a quick and easy way of getting multiple processes running for web applications.<p>And with the `--halt now,done=1` option (that I think is relatively recent?) it means that if any of the parallel processes exit, parallel would exit itself, the whole container will shut down, and external orchestration would start another one if needed.
I wrote down a small usage example here: <a href="https://savannah.gnu.org/forum/forum.php?forum_id=9197" rel="nofollow">https://savannah.gnu.org/forum/forum.php?forum_id=9197</a><p>No need for massive distributed clusters when you have a simple perl oneliner
I recently used parallel to write a 1TB data file for testing using all cores<p><pre><code> seq 0 10000 | parallel dd if=/dev/urandom of=/mnt/foo/input bs=10M count=10 seek={}0</code></pre>