Don't use ENV variables for secret data (2017)

So the author offers two alternatives:<p>1. Using docker-secret inside of a Docker swarm<p>2. Using Keywhiz [1], a Java server together with a FUSE client.<p>This seems overkill for a lot of cases. If environment variables are such a security problem, why not just use a config file (not checked into the source code repository) with proper permissions set?<p>[1] <a href="https:&#x2F;&#x2F;developer.squareup.com&#x2F;blog&#x2F;protecting-infrastructure-secrets-with-keywhiz&#x2F;" rel="nofollow">https:&#x2F;&#x2F;developer.squareup.com&#x2F;blog&#x2F;protecting-infrastructur...</a>

I don&#x27;t agree at all. The reasons in the article all seem like &quot;envs are bad because if you make a mistake you can expose them&quot;. This is not exclusive to envs, it applies to all secrets, independent of the medium used to make it available to the process using it.<p>In my experience, if you prevent using envs for secrets (as docker swarm does) all you get is a disgruntled programmer reading the contents of a secret file to an env in the entrypoint.

Great :-(<p>Had to read to the end of the description of why environment variables are bad to discover that it is effectively an advertisement for Docker. I don&#x27;t use Docker so the article told me pretty much nothing that wasn&#x27;t fairly obvious already, although it is a valuable reminder.

I am progressively adopting Hashicorp Vault as the secrets manager of choice. It can be used by a variety of different scenarios -- directly into the application using AppRole, with Terraform using it&#x27;s secrets provider, by developers, during vault authentication when they get their secrets and access regenerated.<p>This way I am not bound to docker swarm, or keywhiz, or god forbid AWS Secrets Manager.<p>As of now, I am still exposing secrets with Env Vars, but the next step is to use Vault directly. Vault has been pretty reliable so far. It is using AWS KMS for managing the master key and a scalable DynamoDB table for high availability backend.

Also this pattern makes path traversal vulnerabilities (a thing not uncommon in web frameworks) have the potential to allow for privilege escalation on Linux via the &#x2F;proc&#x2F;self&#x2F;environ file.<p>I&#x27;ve been on a pentest where a recently disclosed path traversal bug in Rails was not patched in the environment I was testing and I thought I would get at least some credentials from at least one service, but every host used a dedicated API for secret retrieval and there was nothing sensitive exposed via any system.<p>Maybe your threat model doesn&#x27;t care, just adding a data point.

If you think that putting secrets in ENV is bad, you probably shouldn&#x27;t take the advice of running &#x27;<i>docker service create --secret=&quot;secure-secret&quot; redis:alpine</i>&#x27; either!<p>Putting secrets in a command line makes them just as visible, in fact more so, to other processes. It also makes the secrets available to anyone regardless of permissions, since everyone can monitor the currently running processes and their args (e.g. a &#x27;<i>ps waux</i>&#x27; or &#x27;<i>cat &#x2F;proc&#x2F;12345&#x2F;cmdline</i>&#x27;)

The proposed solution appears to be &quot;use Docker&quot;, which is a bit lame.

Every production product I’ve ever worked on, the entire team put database credentials in the environment variables.

We have recently started removing credentials in env vars and started using google secret manager (previously berglas) and its been amazing so far. AWS and azure should have the same. More challenging when you don’t have all the infrastructure and services in place (ie. when working on plain VPS or other systems without those tools)

For those in the comments suggesting to use another system to store credentials instead of ENV:<p>Do you need a password to access that system? Why not?

Are secrets specific to docker-swarm? If you&#x27;re using plain docker, I imagine this wouldn&#x27;t work?<p>k8s has its own secrets system built into deployments. I haven&#x27;t used it though. I&#x27;ve been at shops that use k8s+vault, and other places that uses marathon&#x2F;DCOS+consoul.<p>If you&#x27;re on AWS you can use pod2iam in a k8s cluster and then use the SSM parameter store to encrypt&#x2F;decrypt secrets based on pod roles. I&#x27;m sure Google Cloud must have similar services.<p>The most agnostic way would be to mount in a file or volume at runtime. It&#x27;s still accessible to the process, but just via the filesystem and not via environment variables. You still need to program with security in mind, but it&#x27;s less likely for inadvertent leaks; basic layers of security. From there you could use something that encrypts that mount at rest and decrypt it when you start the container.<p>&gt; Environment variables are passed down to child processes, which allows for unintended access.<p>Doesn&#x27;t this depend on how you create the new process? fork() would keep a copy of the env in both parent&#x2F;child processes and exec would keep the env because it replaces the current process. But if you start a process using something like the subprocessing module in Python, it would give you a fresh shell for that process, right?

The way I got around this was to store secrets in Google KMS encrypted files in Google Cloud Storage. The KMS key and encrypted files share the same name and can be accessed by that name programmatically. This secret storage method works really well for me and lets you easily access &amp; manage secrets across all environments. It&#x27;s so convenient, I sometimes even use this system as a simple key&#x2F;value store.

I came across a blogpost describing this workflow recently and I&#x27;m curious to hear HN opinions about it. Any pitfalls?<p><a href="https:&#x2F;&#x2F;matthewdowney.github.io&#x2F;encrypting-keys-in-clojure-applications.html" rel="nofollow">https:&#x2F;&#x2F;matthewdowney.github.io&#x2F;encrypting-keys-in-clojure-a...</a><p>1. Generate a new set of API keys.<p>2. Read my encrypted map of keys from disk, decrypt it with a passphrase, assoc in the new key &amp; secret, encrypt it again, and write it to disk.<p>3. At the entry point for my application, use (.readPassword (System&#x2F;console)) to securely read in the passphrase, and then use it to decrypt the key file and read it into a Clojure map.<p>4. Instead of passing the key map around (allowing it to potentially escape into a debug log, or be printed at the REPL if I do something dumb), the top level code of my application passes the credentials into a signer-factory for each api that closes over the credentials.<p><pre><code> ;; The factory is shaped something like this (defn request-signer-factory [{:keys [key secret]] (fn [request-to-sign] (sign-request request-to-sign key secret))) ;; Then an API endpoint looks like this (defn place-order! [signer {:keys [price qty side market post-only?]}] (let [request (comment &quot;Format the order data for the exchange&quot;) signed (singer request)] (do-http-request! signed))) </code></pre> I like this workflow more than others which are centered around only encrypting credentials inside of your Git repository, and decrypting them when you clone &#x2F; pull, because it means that not even on my development machine are keys just sitting around in plaintext.

From experience, it&#x27;s good to support multiple ways of configuring an app. Depending on your case, this could become hard or requires naming conventions.<p>This way you can move secret data to files if needed depending on the deployment choice.<p>My own rule of thumb is:<p>1) sensible default value<p>2) read from file<p>3) read from env<p>Examples:<p>- <a href="https:&#x2F;&#x2F;github.com&#x2F;spf13&#x2F;viper" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;spf13&#x2F;viper</a><p>- <a href="https:&#x2F;&#x2F;docs.spring.io&#x2F;spring-boot&#x2F;docs&#x2F;current&#x2F;reference&#x2F;html&#x2F;spring-boot-features.html#boot-features-external-config" rel="nofollow">https:&#x2F;&#x2F;docs.spring.io&#x2F;spring-boot&#x2F;docs&#x2F;current&#x2F;reference&#x2F;ht...</a>

And fix it by making it dependent on something docker specific?

I use and recommend Mozilla SOPS <a href="https:&#x2F;&#x2F;github.com&#x2F;mozilla&#x2F;sops" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;mozilla&#x2F;sops</a>

I&#x27;m surprised at the amount of responses here that hate this article.<p>It&#x27;s not without cause&#x2F;justification. Any stack tracking software (PagerDuty, Rollbar, NewRelic...) gets huge amounts of secrets pumped out to them regularly because of things like this.<p>And the author is not wrong, the environment doesn&#x27;t need the secret. The application does.<p>Sure, you may not like the proposed solutions, but there are plenty out there, he just named 2.

Maybe if all of your stuff is in docker does this article make sense. Otherwise it is just directly wrong. Use ENV variables they work.

So to mitigate the security risk of ENV variables I should run a JavaVM, a Java Server, and a FUSE client? No thanks.

I just did a write up about how we use a secrets manager to load our environments allowing an easy centralised management across multiple projects&#x2F;envs.<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=23822681" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=23822681</a>

If migrating your infrastructure to swarm is not feasible:<p>- make sure to sanitize the environment before spawning any child processes.<p>- Be sure to `set +x` (or your shell&#x27;s equivalent) in your CI process<p>- that your secrets never get interpolated into a string through your scripting language.

Oddly not talked about much:<p>In a lot of cases, if an attacker gets even limited access to the application environment, dumping the ENV variables is trivial as they were never intended to be secure.<p>`process.env`, `printenv` (including php attacks), `ENV`, etc

Is transferring them to memory in your startup routine, then doing unsetenv() a reasonable mitigation?<p>It seems like it addresses several of the listed concerns. It&#x27;s not perfect, of course, but perhaps better, and straightforward.

What&#x27;s a best practice that doesn&#x27;t use this Docker-specific feature ?

I&#x27;m curious why such opinion is unpopular. ENV is too global and easily exposed in accident. I think people just don&#x27;t want to think to leave 12factor (including me).

&gt; <i>At a previous job I helped solve this problem with a really elegant solution</i><p>That distinction might go to something packaged in a single .h and .c file, if it passes additional smell tests.

I mean, if the argument against it is: what happens if I run arbitrary code in your server? The lease of your problems is having secrets in your env...

There is secret manager is AWS and gcp.

env vars are not collected by default within containers. if OP&#x27;s alternative is to use docker for every service then the whole problem is nonexistent since containers cannot access the host env.

Rails already solved this with credentials files...

The advice is good. But why is ENV capitalized? The term is just &quot;environment variable&quot;. One heuristic I use for evaluating technical material is orthography: if you spell or capitalize or spell something improperly, it&#x27;s likely you&#x27;ll get a lot else wrong too.<p>As for secret storage: didn&#x27;t we solve this problem with keyrings? If I must put a secret in long-term plaintext storage, I might as well put it in a file, where I can see, access-control it, and audit it. Where&#x27;s the audit log for someone reading an environment variable value?