A lesson in dockerizing shell scripts

Are &quot;Random shell scripts from the internet&quot; categorically worse than &quot;random docker images from the internet&quot;?<p>With the shell script, you can literally read it in an editor to make sure it isn&#x27;t doing anything that weird. A single pass through shellcheck would likely tell you if it&#x27;s doing anything that is too weird&#x2F;wrong in terms of structure.<p>Auditing a docker container is way more difficult&#x2F;complex.<p>&quot;Dockerize all the things&quot;, especially in cases when the prereqs aren&#x27;t too weird, seems like it wastes space, and also is harder to maintain - if any of the included components has a security patch, it&#x27;s rebuild the container time...

I can appreciate the work to shrink the image, but copying the various standardized CLI tools and related library files into the image versus installing them with APK can introduce _many_ compatibility challenges down the road as new base Alpine versions are released which can be difficult to detect if they don&#x27;t immediately generate total build errors. Using static binary versions of the various CLI tools would be a better approach here, which inevitably means larger base binaries to begin with, again ballooning the docker image size... all for a minimal gain of 14MB overall is not worth it for a production build unless you&#x27;re working in the most minimal of minimal embedded OS environments, which the inclusion of FZF -and- findutils would already seem to negate since there is so much duplication in functionality between the two tools already.<p>Overall this approach results in an image so fragile I would never use the resulting product in a high-priority production environment or even just my local dev environment as I want to code in it, not have to fix numerous compatibility issues in my tools all over 14MB of space.

<p><pre><code> [...] COPY --from=ugit-ops &#x2F;usr&#x2F;bin&#x2F;tr &#x2F;usr&#x2F;bin&#x2F;tr COPY --from=ugit-ops &#x2F;bin&#x2F;bash &#x2F;bin&#x2F; COPY --from=ugit-ops &#x2F;bin&#x2F;sh &#x2F;bin&#x2F; # copy lib files COPY --from=ugit-ops &#x2F;usr&#x2F;lib&#x2F;libncursesw.so.6 &#x2F;usr&#x2F;lib&#x2F; COPY --from=ugit-ops &#x2F;usr&#x2F;lib&#x2F;libncursesw.so.6.4 &#x2F;usr&#x2F;lib&#x2F; COPY --from=ugit-ops &#x2F;usr&#x2F;lib&#x2F;libpcre* &#x2F;usr&#x2F;lib&#x2F; COPY --from=ugit-ops &#x2F;usr&#x2F;lib&#x2F;libreadline* &#x2F;usr&#x2F;lib&#x2F; [...] </code></pre> For me, insane sh*t like this proves that those who do not learn from distribution and package management infrastructure engineering history are condemned to reinvent it, poorly.

Dive is a great tool for debugging this. I like image reduction work just because it gives me a chance to play with Dive: <a href="https:&#x2F;&#x2F;github.com&#x2F;wagoodman&#x2F;dive">https:&#x2F;&#x2F;github.com&#x2F;wagoodman&#x2F;dive</a><p>One easy low hanging fruit I see a LOT for ballooning image sizes is people including the kitchen sink SDK&#x2F;CLI for their cloud provider (like AWS or GCP), when they really only need 1&#x2F;100 of that. The full versions of both of these tools are several hundred mb each

&gt; In the Alpine ecosystem, it is generally not advised to pin minimum versions of packages.<p>I think it would be more accurate to say, in the Alpine ecosystem, it is generally not advised to pin versions of packages <i>at all</i>. Actually, this is not so much a recommendation as it is a statement of impossibility: You can&#x27;t pin package versions (without your Docker builds starting to fail in a week or two), period. In other words: Don&#x27;t use Alpine if you want reproducible (easily cacheable) Docker builds.<p>I had to learn this the hard way:<p>- There is no way to pin the apk package sources (&quot;cache&quot;), like you can on Debian (snapshot.debian.org) and Ubuntu (snapshot.ubuntu.com). The package cache tarball that apk downloads will disappear from pkgs.alpinelinux.org again in a few weeks.<p>- Even if you managed to pin the sources (e.g. by committing the tarball to git as opposed to pinning its URL), or if you decided to pin the package versions individually, package versions that are up-to-date today will likely disappear from pkgs.alpinelinux.org in a few weeks.<p>- Many images that build upon Alpine (e.g. nginx) don&#x27;t pin the base image&#x27;s patch version, so you get another source of entropy in your builds from that alone.<p>Personally, I&#x27;m very excited about snapshot images like <a href="https:&#x2F;&#x2F;hub.docker.com&#x2F;r&#x2F;debian&#x2F;snapshot" rel="nofollow">https:&#x2F;&#x2F;hub.docker.com&#x2F;r&#x2F;debian&#x2F;snapshot</a> where all package versions and the package sources are pinned. All I, as the downstream consumer, will have to do in order to stay up-to-date (and patch upstream vulnerabilities) is bump the snapshot date string on a regular basis.<p>Unfortunately, the images don&#x27;t seem quite ready for consumption yet (they are only published once a month) but see the discussion on <a href="https:&#x2F;&#x2F;github.com&#x2F;docker-library&#x2F;official-images&#x2F;issues&#x2F;16044">https:&#x2F;&#x2F;github.com&#x2F;docker-library&#x2F;official-images&#x2F;issues&#x2F;160...</a> for a promising step in this direction.

Thanks for sharing this. I like what the author did, they pursued a goal and kept working at it, until they found a balancing point.<p>I think my experience in similar pursuits would have led me to stop very early on - 31.4 MB is already pretty good, to be fair. Looking at the amount of potential maintenance required in the future, for example if the original ugit tool starts to need more dependencies which then have to be wrangled and inspected, makes me think that the size I didn&#x27;t reduce is worth the tradeoff. Since the dependencies can be managed with package managers, without having to think too much, and as the author says, Linux is pretty awesome about these things already.

I love reducing Docker images to their smallest forms. It&#x27;s great for security (minimizes the bill of materials and makes it easier to update at-risk libraries and such), makes developers really think about what their application absolutely needs to do what it needs to do (again, great for security), and greatly improves startup performance (because they are smaller).<p>We can definitely go smaller than 20MB and six layers.<p>Here&#x27;s a solution that compresses everything into a single 8.7MB layer using tar and an intermediate staging stage: <a href="https:&#x2F;&#x2F;gist.github.com&#x2F;carlosonunez&#x2F;b6af15062661bf9dfcb86880a94fd629" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;carlosonunez&#x2F;b6af15062661bf9dfcb8688...</a><p>Remember, every layer needs to be pulled individually and Docker will only pull a handful of layers at a time. Having everything in a single layer takes advantage of TCP scaling windows to receive the file as quickly as the pipe can send it (and you can receive it) and requires only one TCP session handshakes instead of _n_ of them. This is important when working within low-bandwidth or flappy networks.<p>That said, in a real-world scenario where I care about readability and maintainability, I&#x27;d either write this in Go with gzip-tar compression in the middle (single statically-compiled binaries for the win!) or I&#x27;d just use Busybox (~5MB base image) and copy what&#x27;s missing into it since that base image ships with libc.

How would I use this? Say I just made a bad commit in my terminal. How would I run this container to fix it? The container doesn&#x27;t have my working directory does it? Or is that the idea, to mount a volume with the working for or something?<p>In that case, maybe it could be helpful, but to make it convenient, don&#x27;t I need a script that stays in my main system and invokes the docker run command for me?<p>So if you do that and just give me a one liner install command to copy paste then I guess this actually makes sense. A small docker container could eliminate a lot of potential gotchas with trying to install dependencies in arbitrary environments.<p>Except it&#x27;s a bash script. I guess it would make more sense to get rid of the dependency on fzf or something nonstandard. Then they can just install your bash script.<p>For cases where you have more dependencies that really can&#x27;t be eliminated then this would make more sense to me.<p>Why does it need fzf? Is it intended to run the container interactively?

ugh, i would hate to maintain this dockerfile. i actually dont mind a 34MB docker image vs a 17MB image like this

&gt; or maybe ends up sponsoring...<p>Sponsorship for a 500 line shell script. Wow!

While I likely would not have made the same tradeoffs, I do relate to the desire to get the image as small as reasonably possible and commend the efforts. Going to &quot;FROM scratch&quot; is likely going to get you one of the best results possible before you start patching the application and switching out components.<p>I find it mildly ironic, however, that bundling the dependencies of a shell script is - in some ways - the exact opposite of saving space, even if it is likely to make running your script more convenient.<p>Unfortunately, I don&#x27;t have a great alternative to offer. The obvious approach is to either let the users handle dependencies (which you can also do with ugit) or write package definitions for every major distribution. And if I were the author, I wouldn&#x27;t want to do that for a small side project either.

Whats wrong with make or dare I even suggest a package manager like conda? I get having a half dozen dependencies can be specified in tools like docker but its just another way to do the same old task thats been solved a dozen ways for decades. We are sharing a shell script here. Seems crazy to me to run an entire redundant file system to share a couple hundred line bash script. Plus now users need docker skills as well as command line skills to install and run this tooling. There are corners of the command line user&#x2F;programmer world that have thankfully not been polluted by docker yet so its not nearly as widespread a tool as setting up environments for bash scripts using some older ways.

This is neat :)<p>I love going and making containers smaller and faster to build.<p>I don&#x27;t know if it&#x27;s useful for alpine, but adding a --mount=type=cache argument to the RUN command that `apk add`s might shave a few seconds off rebuilds. Probably not worth it, in your case, unless you&#x27;re invalidating the cached layer often (adding or removing deps, intentionally building without layer caching to ensure you have the latest packages).<p>Hadolint is another tool worth checking out if you like spending time messing with Dockerfiles: <a href="https:&#x2F;&#x2F;github.com&#x2F;hadolint&#x2F;hadolint">https:&#x2F;&#x2F;github.com&#x2F;hadolint&#x2F;hadolint</a>

I didn&#x27;t see it in the final tree listing, but I would expect the fzf.tar.gz to linger around after extraction as it was never removed. If that is so, should help squeeze a few more bytes out of the final image.

Dockerizing a shell script????<p>Unless your tool is converted to a service how would anyone ever use this? Do you expect them to run their project inside of your container?<p>This is very bizarre.

One more factor in this is that most developers already have the alpine:latest layer downloaded through some other image.<p>In the best case, that first layer will be reused. Meaning that creating a different base layer may actually increase the size in the end, even if the image in isolation may appear smaller!

&gt; Yeah, I know, I know. REWRITE IT IN GO&#x2F;RUST&#x2F;MAGICLANG. The script is now more than 500+ lines of bash.<p>These screeds get more and more random.<p>The standard advice was always to just not let a program in Bash get beyond X lines. Then move to a real programming language. Like Python (est. 1991).

I&#x27;ve been writing containers for 10+ years and this last few years I&#x27;ve started using supervisord as pid 1 that manages multiple processes inside the container for various things that CAN&#x27;T function as disparate microservices in the event that one fails&#x2F;updated&#x2F;etc a lot more.<p>And man I love it. It&#x27;s totally against the 12 microservice laws and shoudl NOT be done in most cases, but when it comes to troubleshooting- I can exec into a container anywhere restart services because supervisord sits there monitoring for the service (say mysql) to exit and will immediately restart it. And because supervisor is pid1 as long as that never dies your container doesn&#x27;t die. You get the benefit of the containerization and servers without the pain of both, like having to re-image&#x2F;snapshot a server once you&#x27;ve thoroughly broken it enough vs restarting a container. I can sit there for hours editing .conf files trying to get something to work without ever touching my dockerfile&#x2F;scripts or restarting a container.<p>I don&#x27;t have to make some changes, update the entrypoint&#x2F;dockerfile, push build out, get new image, deploy image, exec in..<p>I can sit there and restart mysql, postgres, redis, zookeeper, as much as I want until I figure out what I need done in one go and then update my scripts&#x2F;dockerfiles THEN prepare the actual production infra where it is split into microservices for reliability and scaling, etc.<p>I&#x27;ve written a ton of these for our QA teams so they can hop into one container and test&#x2F;break&#x2F;qa&#x2F;upgrade&#x2F;downgrade everything super quick. Doesn&#x27;t give you FULL e2e but it&#x27;s not we&#x27;d stop doing what tests we already do now.<p>I mention this because it was something I did once a long long time ago but completely forgot something that you could do until I recently went that route and it really does have some useful scenarios.<p><a href="https:&#x2F;&#x2F;gdevillele.github.io&#x2F;engine&#x2F;admin&#x2F;using_supervisord&#x2F;" rel="nofollow">https:&#x2F;&#x2F;gdevillele.github.io&#x2F;engine&#x2F;admin&#x2F;using_supervisord&#x2F;</a><p>I&#x27;m also really tired of super tiny containers that are absolute nightmares to troubleshoot when you need to. I work on prod infra so I need to get something online immediately when a fire is happening and having to call debug containers or manually install packages to troubleshoot things is such a show stopper. I know they&#x27;re &quot;attack vectors&quot; but I have a vetted list of aliases, bash profiles and troubleshooting tools like jq mtr etc that are installed in every non-scratch container. My containers are all standardized and have the exact same tools, logs, paths, etc. so that everyone hopping into one knows what they can do.<p>If you&#x27;re migrating your architecture to ARM64 those containers spin up SO fast that the extra 150-200mb of packages to have a sane system to work on when you have a fire burning under you is worth it. For some scale the cross datacenter&#x2F;cluster&#x2F;region image replication would be problematic but you SHOULD have a container caching proxy in front of EVERY cluster anyway. Or at least at the datacenter&#x2F;rack. It could be a container ON your clusters with it&#x27;s storage volume a singular CEPH cluster, etc.

Does anyone here have experience using Nix to build minimal Docker images? How well does it work, and how does it compare to the author&#x27;s approach of manually copying shared libraries into a scratch image?

I have been able to run ksh93 in an nspawn container under systemd in a tiny fraction of what is presented here.<p>I did this by tracking the output of the ldd command and moving only needed libraries into the container.<p>Why is docker so big?

Whenever I think there can&#x27;t be any worse of a &quot;use case&quot; to dockerize something, someone comes along and proves me wrong...<p>For the last goddamn time: Docker is not a package manager!

Or just write a clean specification and get a docker image close to optimal, and if it’s not, you can prove cryptographically if by some chance you beat the defaults:<p><a href="https:&#x2F;&#x2F;xeiaso.net&#x2F;blog&#x2F;i-was-wrong-about-nix-2020-02-10&#x2F;" rel="nofollow">https:&#x2F;&#x2F;xeiaso.net&#x2F;blog&#x2F;i-was-wrong-about-nix-2020-02-10&#x2F;</a><p>I’ve got plenty of gripes with nixlang, but being worse than Dockerfile-lang isn’t one of them.

do all those COPY lines result in intermediate layers?<p>Could you maybe modify stage 2 to:<p><pre><code> FROM scratch as stage2 COPY ... COPY ... ... </code></pre> and finally at the end:<p><pre><code> FROM SCRATCH COPY --from=stage2 &#x2F; &#x2F;</code></pre>

How does removing the shebang save two megabytes? Seems like a lot. Is it the env binary?

&gt; <i>The use of env is considered a good practice when writing shell scripts, used to tell the OS which shell interpreter to use to run the script</i><p>When using a shebang line, the reason for &#x27;env&#x27; is actually something different.<p>You can just leave out &#x27;env&#x27; and do a shebang with &#x27;bash&#x27; directly like this:<p><pre><code> #! &#x2F;usr&#x2F;bin&#x2F;bash </code></pre> But the problem with that is portability. On different systems, the correct path may be &#x2F;bin&#x2F;bash or &#x2F;usr&#x2F;bin&#x2F;bash. Or more unusual places like &#x2F;usr&#x2F;local&#x2F;bin&#x2F;bash. On old Solaris systems that came with ksh, bash might be somewhere under &#x2F;opt with all the other optional software.<p>But &#x27;env&#x27; is at &#x2F;usr&#x2F;bin&#x2F;env on most systems, and it will search $PATH to find bash for you, wherever it is.<p>If you&#x27;re defining a Docker container, presumably you know exactly where bash is going to be, so you can just put that path on the shebang line.<p>TLDR: You don&#x27;t have to have a shebang, but you can have a shebang at no cost because <i>your</i> shebang doesn&#x27;t need an env.

When FirecrackerOS?!<p>Fly.io, deliver us.