Do-nothing scripting: the key to gradual automation

We use this at my place, though in bash, and we call it executable documentation. Stuff like:<p>NAME=&quot;$1&quot;<p># Create directory for the code<p>mkdir &quot;$NAME&quot;<p># Checkout the source<p>cd $NAME &amp;&amp; git something<p># and so on<p>Sometimes with an echo &quot;You need to read and understand this before just using it&quot;; exit 1 thrown in somewhere for good measure.<p>These files will have no logic, and make very little use of variables. They will also have a .txt filename to ensure that people understand, that first of all, this is a set of instructions for how to do something, which just so happens to be a valid bash script.

This idea is great. You could even argue that these scripts actually do something: they allow storing and updating the <i>state</i> of a process. By having these scripts run in a shell rather than in your head, the current state is stored in a machine, not in an unreliable human brain.

Very nice idea, Dan! This reminds me of the power of checklists that has been discussed here a few times: Simple but highly effective.<p>It allows for a pretty natural transition:<p>- (Markdown) Manual<p>- Do-nothing-script (printing out the manual in steps)<p>- Partial automation<p>- Full automation

I have a script in our automation setup called &quot;be-the-change&quot; which takes either markdown or a wiki url as a parameter. It will output the markdown or wiki content, wait for the user to confirm they&#x27;ve completed the steps and then print:<p>Doing manual steps sucks, wouldn&#x27;t it be great if this was automated for the next person? Be the change you want to see in the world.

I love things like this. Mindbendingly simple ideas that rely purely on overturning preconceived notions about what things are &quot;worth&quot; a certain amount of effort. There&#x27;s a tautological, cart-before-horse paradox in many cases.

I&#x27;d go farther... you can have steps that <i>literally</i> &quot;do nothing&quot;. They can be comments, or simple echo statements. They&#x27;re placeholders to remember to write the code for that step later. This is particularly important when you need to deal with integration that isn&#x27;t necessarily easy - handling auth credential safely, making sure it works in your pipeline, etc.<p>But all this gets back to something I used to say a lot, and the more refined version I say now. I used to say &quot;Have humans do what humans do well; have computers do what computers do well&quot;. The key part of that was the idea that computers can&#x27;t do what humans do. Now, I say &quot;Anything a computer can do well, a human cannot do well&quot;. Which includes those pesky twenty step manual processes.

I get it, but this script looks a lot like a checklist to me. Seems like a checklist is a good place to start, no?<p>If you want more &quot;interest&quot; during the process, make it a point to refine the documentation for your process (clarify a step here, split one command into two steps). Then you have a very clear spec by the time the process needs to be automated.

This seems like a surprisingly powerful and should-be-obvious kind of idea... I think I&#x27;ll start making use of it!

We do a similar thing for tests that are hard to automate. Instead of having the test protocol in a document it’s in a script that prompts the user to do things. The observations are then recorded and stored.

They are <i>do-nothing scripts</i> but it isn&#x27;t equivalent to say that <i>the scripts do nothing</i>. That is a very important distinction to me.<p>These scripts organize tasks into flows in a way that makes it possible to split work by logical checkpoints. The tasks between these checkpoints are dependent on one another in the overall flow, but have independent implementations that can be delegated to software or done by humans.

It seems to me that all you want is to document a manual process. Having them documented in dump scripts is no different than having them documented as a guide in your knowledge base .e.g in Confluence or as product requirements.<p>I can see the usage in an isolated team of a few people that will write these scripts for their own benefit.<p>I am opposed in general with having scarce information around. Better have everything a single place, whether that&#x27;s a guide on how to provision a user or is how to add a new endpoint to your API or write a new end to end test.

An additional benefit of this approach is that your checklists now get code reviews, git hooks, history, etc. for free.<p>I mention this because I worked at a company where checklists were typically simply added to a wiki and required separate processes for things like peer review.

This is a really great idea, and appealing in its simplicity -- I think we might try to implement this in my company for some of our tasks which are really very manual, but for which automation is a daunting task.

Suggested upgrade: Have such scripts ask for a unique name&#x2F;&quot;instance identifier&quot; when run so they can log to simple text files in a well-known directory and potentially pick up where they left off. For when you invariably use these for long-running tasks and have 7 of them in-flight, and your power goes out or Windows forces a reboot.

I like the checklist nature of this. I approached this problem from another side, which involved a very powerful REPL, but lacked the structure this shows: <a href="https:&#x2F;&#x2F;orestis.gr&#x2F;clojure-unlocking-incremental-automation&#x2F;" rel="nofollow">https:&#x2F;&#x2F;orestis.gr&#x2F;clojure-unlocking-incremental-automation&#x2F;</a>

That&#x27;s actually a runbook. But it&#x27;s a pretty good idea to make your runbook into code! This is a nice, gradual way to begin automating. You automate one step at a time, easiest tasks first, until eventually all of it is automated. The more your steps are reusable, the easier it is to automate other tasks.<p>An example is approving new users to access a service. Say your list of users is a YAML file. Someone edits the file, fires off a PR, waits for someone to approve it, merges it, and applies the config. You might think it&#x27;s so easy to do by hand, automating it would be a waste. But if you actually add up the time it takes for humans to do those steps, plus the lead-time from creation to completion, plus the occasional typo, wrong username, or Norway bug, it can take from hours to days to actually complete the task. By automating steps one at a time, you gradually move toward the entire process being a single script that takes a username, and completes the task in about 30 seconds. This is a small win, but added up over all your operations, reduces hundreds of hours of toil <i>and</i> reduces human error. If every engineer automates one step of one runbook per week, over a year you&#x27;ll have hundreds of reusable automation steps.<p>To the OP, I&#x27;d add the following: 1) use the least-complex way to accomplish the task. If you don&#x27;t <i>need</i> to write Python, don&#x27;t. Bourne shell is your friend. Re-use existing tools and automated features. 2) Add documentation to the code that can be auto-generated, so you can use the script as long-lived static documentation. Documenting somewhere else will lead to stale docs. 3) Run all your runbooks from a Docker container. 4) Plan to run these runbooks-in-containers from a CI&#x2F;CD system using paramaterized builds. 5) Organize your reusable automation steps by technology, but have product-specific runbooks that call the automated features. 6) In a big org, a monorepo of automation features is an efficient way to expose automated features to other teams and ramp up on shared automation quicker.

I do the same sorts of things for my scripts.<p>I start with my &quot;template&quot; script and fill it out.<p>totally over-engineered for most tasks, but then as the tasks mature they have a firm foundation.<p>very simplified example:<p><pre><code> #!&#x2F;usr&#x2F;bin&#x2F;python import sys,os,argparse parser = argparse.ArgumentParser(argument_default=None) parser.add_argument(&#x27;-d&#x27;, &#x27;--debug&#x27;, action=&#x27;store_true&#x27;, help=&#x27;debug flag&#x27;) parser.add_argument(&#x27;files&#x27;, nargs=&#x27;*&#x27;, default=[], help=&#x27;files to process&#x27;) arg = parser.parse_args() def die(errmsg,rc=1): print(errmsg, file=sys.stderr) sys.exit(rc) if not arg.files: die(&#x27;filename required&#x27;)</code></pre>

I can&#x27;t help looking at this and working out how to build a tool that can generate a Do Nothing script straight from your histfile, making the business of archiving the commands you just executed as easy as possible too...

This is a checklist. Checklists are incredible, and I quite like this version.

I did something very similar just recently. I have a calendar event to backup all personal data every 3 months, based on a checklist. I&#x27;d always wanted to automate it but every time I tried, it just seemed such a daunting task that I never ended up doing it.<p>Last time, I figured I&#x27;d start going at it gradually. So what I did was I turned the checklist into a bash script, telling me what to do at each step. I also implemented the ones that are easy to automate. The plan is to knock out two or three items each time I do the backup, until it&#x27;s fully automated.

That&#x27;s not “do-nothing” scripting, it&#x27;s orchestration scripting for a process where the steps are executed by a human.<p>It&#x27;s basically the minimal version of a standalone workflow automation; it&#x27;s got crappy state storage and is really only good for a single human execution unit per script (or perhaps multiple swapped sequentially rather than in parallel), but it&#x27;s an instance of a fairly well-known class of do-something software.

Interesting idea! I&#x27;ll wrote some boilerplate that I think is a bit nicer:<p><a href="https:&#x2F;&#x2F;gist.github.com&#x2F;mbarkhau&#x2F;60fc4bbe505914369ebd2fec1ab6d21b" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;mbarkhau&#x2F;60fc4bbe505914369ebd2fec1ab...</a>

Nice idea! I&#x27;m actually in the process of automating a task bit by bit. Unfortunately it&#x27;s with the intention of moving it into a CI pipeline, so the &quot;success&quot; there is rather binary, but I will most definitely adopt this process in future.

In situations where using full-blown configuration management has too much overhead I often do something like this by just copying the commands I run into a script to begin with. It allows me to easily generalize a sequence of actions into a do-nothing template and doubles as documentation.<p>I&#x27;m very much against documenting concrete steps in a wiki if it is something that can be scripted. Such documentation becomes something that you must maintain, and faulty documentation is worse than no documentation.<p>I think that wiki documentation should be used to summarize workflows and provide context, but if you&#x27;re writing step-by-step instructions, often you can just write a script with exactly the same amount of effort.

Why do I think that writing a script in Expect (<a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Expect" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Expect</a>) would take as long as doing this and then you would be done?

Anyone interested in this idea should check out <a href="https:&#x2F;&#x2F;github.com&#x2F;braintree&#x2F;runbook" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;braintree&#x2F;runbook</a>

This is a good idea. I recently wrote down some ideas about &quot;How to Grow a Program,&quot; in a symbolic programming environment: <a href="https:&#x2F;&#x2F;github.com&#x2F;theronic&#x2F;hoist&#x2F;" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;theronic&#x2F;hoist&#x2F;</a><p>I call it Hoist - from &quot;hoisting up&quot; concrete, hard-coded values into abstract symbols for reuse. It&#x27;s complete vaporware at this point, just a place to collect my thoughts for a future essay.

Great idea! From personal experience I can say that there are some things ops teams never get around to automate, either because there is always something more important to do, or because it would be too expensive to have a developer spend time automating them. This is a simple, quick solution that solves the problem of accidentally skipping a subtask, thus screwing something up and feeling stupid for doing so.

This feels like a useful concept.<p>I stumbled into doing roughly the same in my macOS system bootstrap script. I found it from the other side, though--usually only after failing to find a good way to automate a step, or realizing automation would be vastly more complex&#x2F;fragile than just prompting myself to do something manually at the right time.<p>Probably a more useful concept when it&#x27;s conscious from the getgo. :)

if you are working at a small shop or startup - then whatever makes you happy, even these classes are OK.<p>if you are working at enterprise - then these processes gotta be implemented according to ITIL in a specialized IT Service Management system - like ServiceNow, which allows building process flows like directed acyclical graphs and automation of necessary steps

What is the value of this over a checklist?

raw_input is python 2.x. Dont use it.

Seems basically to be a checklist.<p>However, I don&#x27;t think it&#x27;s a good idea. Checklists are not bad, but they are to be executed by humans.<p>I think trying to automate an existing human process directly is a mistake. Human processes often have features that only humans can do, such as pattern recognition or adapting to small difference. It&#x27;s often easier to create a computerized (automated) process from scratch than to try to account for all the edge cases that humans might have to deal with (and do without problems).<p>(The opposite is also true, processes that computers have no issues with can make trouble for humans. For example, humans have imperfect memory. So if the process asks humans to keep track something for extended periods, it can be difficult.)

When I can (often):<p>1. Write SQL to automate<p>2. Put SQL with output in Metabase<p>3. Have it manually checked alongside day-to-day operation<p>4. Iterate 1-3 (you often will get a lot of feedback)<p>5. Replace the manual work, now that the SQL it checked on a lot of cases in production

Implementation question:<p>Whats a good way to generalize such a python script so that windows and nix users can run it?

Are there any good slack plugins out there that can do this kind of workflow&#x2F; state management?

Combined with the concept of literate programming this would also help capture the &quot;why&quot;s of the slog, resulting in inline documentation. Jupyter and org-mode for the win!

Checklists as Code

It&#x27;s a checklist

This just seems like checklists with extra steps.

I do this!

I know this isn’t the point of the article but the code shown there is the archetypal case where writing classes is an anti-pattern, as argued in “Stop Writing Classes” [1]. All the classes in that code can — should! — be functions. There’s no downside, and it’s less code (and otherwise identical).<p>[1] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=3717715" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=3717715</a>

Is anyone else bothered by how they were handling ssh keys in the example?<p>The whole point of public keys is that the private key never has to leave the box it’s generated on. Once you go sharing them you might as well just use a random passphrase.