Maybe some controversial advice: Go ahead, fall in these pits.<p>I write my fair share of shell scripts and I've hit practically every one of these snags in the past. However, for the majority of tasks I perform with bash, I genuinely don't care if I support spaces in filenames, or if I throw away a little efficiency with a few extra sub-shells, or if I can't test numbers vs strings or have a weird notion of booleans.<p>Your scripts are going to have bugs. The important question is: What happens when they fail?<p>Are your scripts idempotent? Are they audit-able? Interruptible? Do you have backups before performing destructive operations? How do you verify that they did the right job?<p>For example, if your shell scripts operate only on files under version control, you can simply run a diff before committing. Rather than spent a bunch of time tracking down a word expansion bug, you can simply rename that one file that failed to not include a space in its name.
The Unix shell may be a highly powerful interactive programming environment, but it's sure hard to think of anything that comes anywhere close to <i>sucking</i> as badly. With the shell and the standard Unix commands, some things that are hard in other languages are easy, and most of the things that are easy in other languages are hard to impossible... I'd love to see a clean slate replacement for the shell that still feels Unix-like and retains most of its existing benefits.<p>(I suspect PowerShell would be a good environment to take design cues from or even port, but I've never used it so I can't say for sure.)
I have a mirror of this site. wooledge.org runs off of greycat's (#bash on freenode) home DSL connection:<p><a href="http://bash.cumulonim.biz/BashPitfalls.html" rel="nofollow">http://bash.cumulonim.biz/BashPitfalls.html</a>
Apparently a mod changed the URL (to relieve a not so powerful host). The original URL (and thus the one you should be bookmarking/remember) was <a href="http://mywiki.wooledge.org/BashPitfalls" rel="nofollow">http://mywiki.wooledge.org/BashPitfalls</a><p><a href="http://bash.cumulonim.biz/BashPitfalls.html" rel="nofollow">http://bash.cumulonim.biz/BashPitfalls.html</a> is a mirror, see mlacitation's comment.
So why do we put up with classic command line tools in general that are so full of horrible, counterintuitive pitfalls? Is it just tradition? Backwards compatibility?<p>The "Unix should be hard" crew has gotten a lot quieter in the last ten years with the rise of Ubuntu and other relatively user-friendly distros, but I feel like there's still an underlying current of elitism there; people are proud of mastering these bizarre, arcane methods, and they're offended that someone else might be able to accomplish just as much without doing half as much work.
Here’s my personal favorite shell pitfall, which was the last drop to make me start recommending _against_ using shell except for very very narrow niche use cases: <a href="https://plus.google.com/+MichaelStapelberg/posts/YLarC7WPVQB" rel="nofollow">https://plus.google.com/+MichaelStapelberg/posts/YLarC7WPVQB</a>
Would someone recommend an automatic bash style checker, such as a 'linter'? Perhaps something along the lines of Chef's Food Critic? <a href="http://acrmp.github.io/foodcritic/" rel="nofollow">http://acrmp.github.io/foodcritic/</a>
Ah yes, the ultimate reference from Freenode #bash, after you learn from Greycat's wiki you won't simply Google/DDG "bash tutorial" again and just head straight here.
I find it sad and amusing that we're writing a ton of mission-critical code in this language that has an incredible number of obscure quirks. Yes, most of these pitfalls are directly connected to the semantics of Unix, but I wish someone made a concerted effort to get rid of them in an otherwise evolutionary way.
Bash is for most things, one of the easier languages I have dealt with, being even beyond python, etc. That is though, for shell type scripting.<p>There's many problems with it, but the only one I've run into that keeps it from being more useful is that there are no multi dimensional arrays built-in. There are super hacky ways I have seen them implemented, but by default it's something I basically never am able to turn to when scripting in bash and have to turn to other languages, even when the particular task I was working with would be mostly simpler in bash.<p>That said, there are associative arrays in bash these days.
The following does not work on files with spaces according to the article:<p><pre><code> for i in $(ls *.mp3); do
some command $i
done
</code></pre>
So does that mean, that "for" will do something per <i>word</i> of the output of $, rather than per <i>line</i> of output of it?<p>What to do if I want to do something for every line? What for example if I really want the output of ls, find (or any other command you can put int he $()) and loop through that line per line, even if some output has spaces?<p>Thanks.
I know someone will sooner or later propose that we ban spaces and special characters in names. Let me just put my two cents forward.<p>We should absolutely ban special characters from names. Specifically, all whitespace, the colon, semicolon, forward slash, backward slash, question mark, star, ampersand, and whatever else I'm missing that will confuse the shell. Also files cannot start with a dash.<p>However, people should be able to name files with these characters. So I propose that these characters in filenames be percent-encoded like they would be in a URL. Specifically, the algorithm should be<p>1. Take the file name and encode it as UTF-8. Enforce some sort of normalization.<p>2. Substitute each problematic byte with equivalent percent-encoded form. This does not touch bytes over 0x80 - they are assumed non-problematic.<p>3. Write the file in the file system under that name.<p>4. When displaying files, run the algorithm in reverse.<p>In the general case files like "01 - Don't Eat the Yellow Snow.mp3" would simply become 01%20-%20Don't%20Eat%20the%20Yellow%20Snow.mp3 in the filesystem and cause absolutely no further problems. To make it completely backwards-compatible we should also add the following rule: If a filename includes a problematic byte or a percent-encoded byte higher than 0x80, then it is assumed to be raw and will not undergo percent decoding.<p>Basically, I propose that every program which receives free text input for a file name percent-encode the filenames before writing them to the filesystem and decode them for display. Everything else remains unchanged.<p>Why this will not work:<p>Requiring programmers to keep track of two filenames instead of just one is rather a lot of work. File APIs will have to take both encoded and non-encoded forms and encode the non-encoded form, creating problems when people inadvertently use the wrong function with a name, either double-encoding it or not encoding it and leading to "this file does not exist" errors.<p>It will be possible to create two files with different names on disk which are nonetheless shown with the same name to the user.<p>Why it is ugly:<p>We're taping over a deficiency of an ancient language by inflicting pain on programmers.<p>Double-encoded filenames? MADNESS.<p>Why I like it:<p>I'll be able to have ?, * and : in filenames in windows.<p>My shell scripts will be much simpler.<p>What do you guys think?
<p><pre><code> for arg
</code></pre>
instead of<p><pre><code> for arg in "$@"
</code></pre>
is gold. That is going straight to the pool room.