Bashing the Bash – Replacing Shell Scripts with Python (2017)

I was <i>hoping</i> someone finally had a library or something to make python good at the things shell does well (mostly, running stuff and piping output around). If only.<p>This really just reads like someone who doesn&#x27;t know shell and does know python, but thinks the problem is the tool. They continually describe shell with phrases like &quot;obscure and difficult-to-predict&quot; while talking through things that are either obvious with experience or from context, and then write something <i>different</i> in python (i.e. something you <i>could</i> write in BASH that does something different) that&#x27;s not less obscure but is only using <i>python</i> obscurity.<p>&gt; An example of a shell obscurity is the way the current working directory is set. The cd command is clear enough, but in the presence of sub-shells using (), can make it difficult to discern a stack of nested shell invocations and how the working directory changes when the sub-shells exit.<p>Er... yeah? So you expect PWD to be a global and it was actually a local?<p>Or more explicitly:<p>&gt; One of the shell’s ickier features is that variables tend to be global. There are some exceptions and caveats, however, that lead to shell scripts that are broken or behave inconsistently.<p>That&#x27;s not a shell feature, that&#x27;s just how people frequently write it - you can make functions and declare your variables local if you want. I can easily write a python script with all my variables at the top, too.

I think this article is supposed to commend Python as a better choice than Bash? I think the author forgot to make a convincing argument for that, though.<p>&gt; The point of bash-bashing is to reduce use of the shell.<p>That&#x27;s a tautology. I get it, we&#x27;re supposed to reduce the use of bash ... but why?<p>&gt; Without much real work, it’s easy to replace shell scripts with Python code.<p>But you&#x27;re writing the same thing again? There needs to be more reason than this.<p>&gt; The revised code is easier to read and maintain, runs a little faster, and can have a proper unit test suite.<p>Three claims. Any data to support it?<p>Some of the examples of &quot;bash is bad&quot; are also not convincing to me. Here&#x27;s an example:<p>&gt; An example of a shell obscurity is the way the current working directory is set. The cd command is clear enough, but in the presence of sub-shells using (), can make it difficult to discern a stack of nested shell invocations and how the working directory changes when the sub-shells exit.<p>So...why do you have &quot;a stack of nested shell invocations&quot; that all need to be working directory aware? And how would this be any better in Python? (It could be better in either Python or Bash but either one still requires the developer to avoid simple mistakes.)<p>I don&#x27;t know. I don&#x27;t get it. I feel like the author had the unfortunate experience of knowing more Python than Bash and inheriting someone else&#x27;s (who also didn&#x27;t have much Bash experience) crappy Bash code.

I have some bad news for you, from a long, long, long time CS&#x2F;EE&#x2F;Sysadmin person:<p>I can probably run my bash script on every linux I ever touched.<p>I can probably have that python code break on half of the linux boxen I use. This one does not have module X installed. This one is too old, this one is too new, this python is not holding its mouth just at the right angle to work.<p>Yeah, bash bash all you want. I have pulled out scripts decades old and run them. I can not say the same for other things I have written.

I have ksh scripts I wrote over 20 years ago on xenix, which still work today on bash or ksh. Meanwhile I have python apps that didn&#x27;t make it a year.<p>Any particular version of Python is a saner choice, if you could pick one and freeze it retroactively for the last 20 years and for at least the next 20. But that is not how Python works. Python breaks every 11 minutes.<p>I would not write anything in Python that I cared the slightest bit about longevity or portability.<p>Maybe if we made a subset of python, locked down the syntax and feature set, discouraged any use of plugins or libraries in any fancy way that relies on any kind of repository or package manager system, and gave it a new name to distinguish it from normal python, <i>maybe</i> that would be an ok replacement for any of the shells.<p>But that is not happeing and don&#x27;t even try to pretend like it could.

I myself kind of went full circle. I started mostly python, then started to really embrace bash, mainly because I joined a company where everyone else used it a ton, got pretty decent at it. Even wrote basically a primitive container orchestrator in Bash because my PM didn&#x27;t want to use anything &quot;newfangled&quot;. Then I joined a startup at basically the ground floor, and it&#x27;s almost entirely pure python, with some Makefiles to automate common operations. All that to say, I have a lot of experience with both paradigms.<p>And frankly...bash kinda sucks as a programming language and environment. It&#x27;s a footgun factory. Even when I was most fluent, I constantly had to deal with nested quotes, escaping, stringly types, empty variables, implicit behavior.<p>Python&#x27;s biggest weak point is definitely its packaging and dependency ecosystem... but it at least has one. Bash doesn&#x27;t even have modules. And it&#x27;s gotten MUCH better over the years, with pyproject.toml, poetry, actual version resolvers, pipx makes managing venvs for cli tools way easier.<p>&quot;but bash is everywhere&quot; says many folks. yeah, and? There was a point in time where it wasn&#x27;t. Python is seeing way more market penetration.<p>Xonsh (python based shell) and plumbum (python library which makes pipe-operating and subprocessing easier) are both great.

I&#x27;m fully in favor of replacing shell scripts with Python3 scripts wherever possible. In fact that&#x27;s part of my job. That being said, the author of this article is using confusing &#x2F; wrong terminology to discuss bash.<p>This article says that, &quot;The [bash] shell isn&#x27;t a complete programming language&quot;. While I agree that bash lacks any real data types besides strings, bash is a Turing-complete programming language[1], so it&#x27;s theoretically possible to write _any_ program in bash that can be written in Python. It might be a hell of a lot uglier and lack things like imports, modules, etc., but it can be done.<p>For example, there is an implementation of an HTTP daemon written purely in bash[2]. Once again-- terrible idea, but great execution.<p>[1]: <a href="https:&#x2F;&#x2F;en.wikibooks.org&#x2F;wiki&#x2F;Bash_Shell_Scripting#A_few_notes_on_terminology" rel="nofollow">https:&#x2F;&#x2F;en.wikibooks.org&#x2F;wiki&#x2F;Bash_Shell_Scripting#A_few_not...</a><p>[2]: <a href="https:&#x2F;&#x2F;github.com&#x2F;avleen&#x2F;bashttpd" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;avleen&#x2F;bashttpd</a><p>edit: added newline to separate references

As much as I appreciate Python, I much rather use Perl as a replacement for complex shell scripts. Perl 5 is ubiquitous, it has been stable for the best part of the last 20 years and it&#x27;s unbeatable for all those tasks that involve heavy text processing.<p>Python is good for complex, structured applications, but shell scripts are not that. Shell scripts are glue, and there&#x27;s arguably no better glue than Perl.

The one thing about Python that makes it so much more compelling over Bash is that thing you discover you need once your script starts getting long and repetitive: functions, with arguments, and return values. Python does this much better than Bash. It’s the one thing that makes me no longer use Bash. Oh and exceptions. Exceptions!<p>The TWO things about Python that make it more compelling are functions and exceptions. You stand a chance of handling errors in Python. In bash there’s very little else you can do, when encountering an exceptional situation that is an error, except stop. The nice thing about all these functions is you can put them in modules as well. Modules! Of course! A namespaced way of laying out non trivial amounts of code!<p>Ok so real functions, exceptions, modules. Three things that makes Python my default tool of choice. And libraries. Bash doesn’t have pip. Sure, bash’s “pip” is the commands it can run aka &#x2F;usr&#x2F;bin, but if that’s your interface then it’s hardly as flexible as say gitlab.py or requests.py.<p>So: apart from the functions, exceptions, modules, libraries. And speed. SPEED! And code formatting. And a debugger. And stack traces. …what has Python ever given us that makes it a no brainer replacement for bash scripts?<p><i>“Brought Unicode”</i><p><i>”Unicode! Oh shut up!”</i><p>— <i>After MP</i>

In my experience, bash makes the simple things simple and the hard things (like complex pipelines, validating arguments, or “pure bash”) really hard. YMMV.<p>Python makes everything medium difficulty which can be a big win depending on the size of automation required.

I decided a while ago to write all my future scripts in Python whenever possible instead of sh or bash due to the overall better syntax and power of Python.<p>Typer is one of the better packages for running Python scripts from CLI. Fabric &amp; Invoke are pretty good too.<p>To run any bash commands in Python I have a Python script that creates a temporary bash file with the code to execute, makes it executable, runs it with the commands in the built-in subprocess package, then deletes the file. This makes any complex bash commands with piping and whatnot runnable straight from Python.<p>This way I don&#x27;t have to learn all the different bash-based Python commands from pathlib and whatnot that throw unexpected errors, and I can run them using pure bash syntax from Python. But then you also get all the benefits of Python&#x27;s looping syntax, classes etc.

This feels like:<p>&quot;Moving furniture by throwing it in the back of a pickup truck is obviously bad. Here&#x27;s how to do it with a forklift and 18-wheeler instead, because this is definitely better.&quot;

Beware of Python devs writing Python shell scripts. They&#x27;re used to using all the brand new (backwards incompatible) language features as soon as they come out. The vast majority have no consideration for a 10 year old PC running a 10 year old OS. The python3.x they write will not run.<p>Bash, surprisingly, gets backwards incompatible language features too. But the vast majority of bash developers are writing for <i>all</i> computers, not just computers with software from the last 3 years.<p>Python language could work if you stuck to an old target. But python culture is a problem and makes this very unlikely. What python is, and what python devs deal with, changes constantly. That doesn&#x27;t work for shell scripts where stability is king.

I wrote this to simplify just that for my own tools; the subprocess module that comes with the batteries has a very general interface, i think that it is a bit complex for a quick script.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;MoserMichael&#x2F;subb" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;MoserMichael&#x2F;subb</a><p><a href="https:&#x2F;&#x2F;pypi.org&#x2F;project&#x2F;subb&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pypi.org&#x2F;project&#x2F;subb&#x2F;</a><p>Python doesn&#x27;t have the problem of shell scripting language, it doesn&#x27;t get impractical, as the program is getting more complex. In bash you have arrays, and even maps, but these aren&#x27;t pretty. Also the shell scripting language is being evaluated by an parse tree&#x2F;AST interpreter, that&#x27;s significantly slower than even python, in it&#x27;s byte code interpreted form.<p>My objective was to get an abstraction, for a one line process run and extraction of the result, similar to what we had in Perl5 with the system library function. <a href="https:&#x2F;&#x2F;perldoc.perl.org&#x2F;functions&#x2F;system" rel="nofollow">https:&#x2F;&#x2F;perldoc.perl.org&#x2F;functions&#x2F;system</a><p>Also the shell is impractical, when it comes to slightly more complex programs. There is a limit on what you can do with pipes. Maybe that&#x27;s the reason why perl is that flexible, as they tried to bridge both realms: Perl had to be useful as a replacement for the quick shell like script, and to be useful as a general purpose programming language.

I didn&#x27;t even read the article. The reason is that it depends what you need to do, from there you should decide what&#x27;s the best tool to solve something. Recently I&#x27;ve started to automate all the package installations and configurations for some of the machines I have. I can&#x27;t even imagine doing that in Python despite I&#x27;ve been using Python for a long time. I don&#x27;t know that much about bash, but I got everything I needed and expected from bash scripts for the task I needed to do.<p>Yes, there are moments when you try to solve something in bash that you think is trivial in other languages but with bash is painful. Few days ago tried to test if a value is present in a bash array using a function. Couldn&#x27;t get it working after trying few things. Then I realized I could perform a very similar (and good enough) check just seeing if a directory and a file do exist in my relative path to the script I was running: `if [-d DIR ] &amp;&amp; [ -f FILE ]; then ...`.<p>So I needed to change my mindset a bit. Another example of where bash is really good is for tiny small tasks: I wrote a small script to increase&#x2F;decrease volume that is coupled in i3 and calls the `mixer` command in FreeBSD. Works perfectly and never touched once it was working. Another one: I get an acoustic alert when my battery drops down below certain percentage. And a couple more: set a random background every time I log in to i3, download a page for offline reading.<p>Again, doing any of these things (or similar ones) with something different than bash (or maybe your favorite shell lang) seems like picking up the wrong tool for the job. There was a time when I cursed shell scripting a lot and that was because I couldn&#x27;t see when to use shell scripting.<p>EDIT: Paragraph spacing.

OK, I&#x27;ll bite... The author would complain about my python3 in the same way and conclude it would be better to write it in my Bash.<p>Without questioning the &quot;why&quot; of the script, sans my customary morning coffee, and without having tested the code below.<p>Maybe I&#x27;d do something like this.<p>For the small price of function invocation overhead, the nice benefit of doing it this way is that one can `source` the functions into one&#x27;s Bash shell and use each one as a standalone Unix tool, complete with tabtab completion, pipelines etc.<p><pre><code> #!&#x2F;usr&#x2F;bin&#x2F;env bash stop_app() { pkill ${1:?Fail. App name required.} } today() { date +%Y%m%d } analyse() { local analyser=${1:?Fail. Analyser script name required.} local out_dir=$(printf &quot;results_%s&quot; $(today)) while yaml_file do local out_file=&quot;${out_dir}&#x2F;summary_$(basename yaml_file).txt&quot; python3 ${analyser} ${yaml_file} &gt; ${out_file} printf &quot;%s\n&quot; ${out_file} done } exec_analytics() { local analyser=${1:?Fail. Analyser script name required.} local source_dir=${2:-&quot;~&#x2F;Documents&#x2F;ExtFin-EFS&#x2F;smoke&#x2F;&quot;} find ${source_dir} -type f -name *.yaml | sort -r | analyse ${analyser} | tail -1 } update_current() { local latest_outfile=${1:?Fail. Provide latest output file.} ln -sf ${latest_outfile} &quot;current.txt&quot; } </code></pre> And maybe one can invoke it like...<p><pre><code> stop_app &quot;whatever_app&quot; &amp;&amp; ( trap &quot;rm -f &#x2F;tmp&#x2F;module_design_analytics_outfile&quot; 0 HUP TERM PIPE INT if exec_analytics module_design_analytics.py | tail -1 &gt; &#x2F;tmp&#x2F;module_design_analytics_outfile then update_current $(printf &#x2F;tmp&#x2F;module_design_analytics_outfile) echo &quot;Done&quot; else echo &quot;Oops. Something went wrong.&quot; fi trap - 0 HUP TERM PIPE INT )</code></pre>

(2017)<p>Previous discussion: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=14998213" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=14998213</a>

Showing the code as images instead of text and not even with a fixed width font for code ... and I am suppose to trust that person? :)

Places that I&#x27;ve implemented shell&#x2F;utility scripts in Python tend to do so because their other application code was also in Python and Python has much more testing support than bash scripts will ever have.<p>That&#x27;s why I&#x27;ve done it before, to write tools like custom log rotating &#x2F; clean-up stuff; another project that comes to mind was a data pipeline that used some Python regex stuff to clean up non-printable characters and ugly stuff from text files before importing data. Both of these projects had lots of tests to verify that the code did what we wanted it to do.

I love to write Makefile recipes that wrap bash into small make commands. My make boilerplate generates help from comments in the make file. I get completions for free. I want all make vars and some internal make recipes to not complete, so I prepend the name with an underscore. Make is great because it handles errors and dependency trees.

Im dealing with some python3 crap today on my main laptop.<p>Primarily, because of a library I compiled and installed, I&#x27;m getting other nasty side effects form it. Primarily that protonvpn-cli isnt now running becasue of some broken library.<p>I&#x27;ve never had bash break like that. Quirks, sure. But never this sort of brokenness.

For folks that like the idea of porting over shell scripts to python: <a href="https:&#x2F;&#x2F;plumbum.readthedocs.io&#x2F;en&#x2F;latest&#x2F;" rel="nofollow">https:&#x2F;&#x2F;plumbum.readthedocs.io&#x2F;en&#x2F;latest&#x2F;</a>

I guess, if you <i>really</i> want a &quot;coding-lang&quot; rather than &quot;just&quot; bash. I&#x27;d give Golang a try, just because of the static-binaries.<p>No environment issues or missing and clashing libraries.<p>PS. I&#x27;d still go for bash 99&#x2F;100.

Yo - I&#x27;m sure there are advantages, but &quot;that python ecosystem&quot; !<p>Bash is usually everywhere even on &quot;new installations&quot;. I&#x27;d hate to fight package compatibility, virtual environments etc..<p>YMMV

I agree with the general thesis that Bash is unsuited for production work of any complexity, but the coverage of pipelines is incomplete. The only two examples I have found (the examples are images, so I may have missed something) may be implemented in Python by calling sorted(), which is not the case in general. I feel it would be more persuasive if it had at least one example of setting up and running a pipeline.

Depends on what you’re doing.<p>I wouldn’t want to worry about python environment when dealing with OS post-install scripting for instance.

I think some of the examples of Python being better are places you just shouldn&#x27;t use Bash.<p>Like testing (Bats sucks).<p>Like string manipulation (use other utils or even languages callable from Bash).<p>Like complicated logic. Just don&#x27;t do that in Bash.<p>Bash is for (glorified) one-liners. And for I-can&#x27;t-believe-you-did-that-in-Bash-ers.

I&#x27;ve spent many years becoming quite fluent in bash. It&#x27;s not trivial. But all objections here are surmountable if not outright features. Python is the right tool for many jobs. But so is bash, for someone willing to truly learn it.

&gt; Without much real work, it’s easy to replace shell scripts with Python code. The revised code is easier to read and maintain, runs a little faster, and can have a proper unit test suite.<p>And give a wonderful syntax error.

I once worked for a company that had a product which had a fairly complicated and time-consuming build process. Someone had written a distributed build process using the traditional CMD.exe shell scripting language. It was running multiple processes in parallel, and they communicated by writing and reading text files. Later they added PowerShell scripts. All in all, there were tens of thousands of lines of PS and BAT scripts. It was quite a nightmare.<p>The team was supposed to maintain this mess. We decided to create a compiler that translated .BAT scripts to C#. The idea was that the C# code would be easier to understand and modify. I left before it was complete, so I&#x27;m not sure how that worked out. But last I heard, they were pleased that some parts of it now ran thousands of times faster.

having a hard time being sold here.<p>mkdir -p would create the directory without error. checking if a directory exists is trivial before creating it. Python throws an error if the path given to os.mkdir exists too.<p>This article is way too long to digest, so i should stop now.<p>Perhaps the argument should be, rewrite hacky things in bash as programs and build them into your apps?

That&#x27;s basically what I did with this project.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;Mylab6&#x2F;PiBluetoothMidSetup" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Mylab6&#x2F;PiBluetoothMidSetup</a><p>Of course I could of done this in bash, but Python is just so much cleaner.

More Lillipythonian flame baiting. Summoning the great dang.

TCL would be a better choice, more so back then.

Rock The Bash-Bah!<p>(Sorry, Oingo Boingo)

The only real alternatives to bash are shit-tier languages like python and perl.<p>I will never use a programming language which has significant white space, specially if I may have to view and edit the scripts in a remote terminal with vi.<p>I am also not too interested in using a programming language which looks the same before and after encryption: <a href="https:&#x2F;&#x2F;www.goodreads.com&#x2F;quotes&#x2F;tag&#x2F;437174-perl-the-only-language-that-looks-the-same-before?utf8=%E2%9C%93&amp;id=" rel="nofollow">https:&#x2F;&#x2F;www.goodreads.com&#x2F;quotes&#x2F;tag&#x2F;437174-perl-the-only-la...</a>