Environment Variables Considered Harmful for Your Secrets

This article is misguided. Environment variables can be more secure than files. Furthermore, in the presented case there&#x27;s no improvement in security by switching to a file.<p>To address my second claim first: file permissions work at the user or group level. ACLs &#x2F; MAC likewise. SELinux can be configured to assist in this case but it&#x27;s not as trivial as it appears at first glance, it would be easier to use environment variables.<p>In the example case of spawning imagemagik, it&#x27;s running as the same user and therefore has the same level of access to the properties file. That is, it can access the secrets without negotiating any authorisation to do so.<p>Depending on how imagemagik is launched and how the parent process handles the config file, it&#x27;s possible that imagemagik could inherit a file handle, already open to the file.<p>Now to address my first claim, if the parent process is following best practice then it will sanitise the environment before exec&#x27;ing imagemagik, that should mean launching the imagemagik process with only the environment it needs.<p>To give a concrete example, the postfix mail transfer agent is extraordinarily high quality software, its spawn process owns the responsibility of launching external processes, potentially sysadmin supplied &#x2F; external to postfix. This case would be very comparable to the web app invoking imagemagik.<p>We can see that it explicitly handles this case as I&#x27;ve suggested is best practice: <a href="https://github.com/vdukhovni/postfix/blob/master/postfix/src/spawn/spawn.c" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;vdukhovni&#x2F;postfix&#x2F;blob&#x2F;master&#x2F;postfix&#x2F;src...</a><p>EDIT: accidentally posted before finishing.<p>If the parent sanitised the child&#x27;s environment, then the only way for the child to access the data would be to read the parents memory. In practice this is quite easy - try the &quot;ps auxe&quot; command for a sample, however this access can much more easily be controlled by SELinux policy than can file access.<p>Any obfuscation technique applicable to a config file can similarly be applied to an environment variable.

My company has an internal bit of infrastructure that I think is a somewhat novel approach that allows us to never have any secrets stored unencrypted on disk. There&#x27;s a server (a set of servers, actually, for redundancy) called the secret server, and its only job is to run a daemon that owns all the secrets. When an app on another server is started up, it must be done from a shell (we use cap) which has an SSH agent forwarded to it. In order for the app to get its database passwords and various other secrets, it makes a request to the secret server (over a TLS-encrypted socket), which checks your SSH identity against an ACL (different identities can have access to different secrets) and does a signing challenge to verify the identity, and if all passes muster, it hands the secrets back. The app process keeps the secrets in memory and your cap shell disconnects, leaving the app unable to fetch any more secrets on your behalf.<p>The other kink is that the secret server itself reads the secrets from a symmetrically-encrypted file and when it boots, it doesn&#x27;t actually know how to decrypt it. There&#x27;s a master key for this that&#x27;s stored GPG encrypted so that a small number of people can retrieve it and use a client tool that sends the secret server an &quot;unlock&quot; command containing the master key. So any time a secret server reboots, someone with access needs to <i>gpg --decrypt mastersecret | secret_server_unlock_command someserver</i><p>There are some obvious drawbacks to this whole system (constraining pushes to require an SSH agent connection is a biggie and wouldn&#x27;t fly some places, and agent forwarding is not without its security implications) and some obvious problems it doesn&#x27;t solve (secrets are obviously still in RAM), but on the whole it works very well for distributing secrets to a large number of apps, and we have written tools that have basically completely eliminated any individual&#x27;s need to ever actually lay eyes on a secret (e.g. if you want to run any tool in the mysql family, there&#x27;s a tool that fetches the secret for you and spawns the tool you want with MYSQL_PWD temporarily set in the env, so you need not copy&#x2F;paste it or be tempted to stick it in a .my.cnf).

There&#x27;s a not-widely-publicized feature of Linux that allows programs to store secrets directly in the kernel: <a href="https://www.kernel.org/doc/Documentation/security/keys.txt" rel="nofollow">https:&#x2F;&#x2F;www.kernel.org&#x2F;doc&#x2F;Documentation&#x2F;security&#x2F;keys.txt</a> That has some advantages, including the guarantee that it can&#x27;t be swapped to disk. Kerberos can use it for secret storage, I haven&#x27;t seen it used elsewhere though.<p>It looks like process-private storage is one of its features.

Classic UNIX behavior was that environment variables were public (any user could see them with the right flags to &quot;ps&quot;) so it was well-known not to put anything secret there.<p>Most (all?) of the current brand of UNIX variants have locked this down quite a while ago, which is a good thing. There are still a few old boxes kicking around though so if you&#x27;re writing code that is meant to be widely deployed please don&#x27;t put stuff there. For example: <a href="https://github.com/keithw/mosh/issues/156" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;keithw&#x2F;mosh&#x2F;issues&#x2F;156</a><p>Even if you are sure that your code will only be running on modern machines I think this article gives good advise. Unless you purge the secret environment variables when you launch they&#x27;ll get sent to all of your subprocesses and it&#x27;s quite possible that one of them won&#x27;t consider their environment secret.

I&#x27;ve always been uncomfortable with the &quot;store config in the environment&quot; part of the 12 factor app thing, since it does imply storing things like database passwords and such, and the argument is that those <i>shouldn&#x27;t</i> be in files. But, filesystem permissions are reasonably flexible and are easy to reason about (unlike the potential visibility of ENV).<p>I also don&#x27;t really buy the arguments for ENV storage of even non-sensitive data. There&#x27;s just not really any good reason to do so; your config has to be put in place by <i>some</i> tool, even if it is in ENV; why not make your tool produce a file, with reasonable permissions in a well-defined default location? The 12 Factor App article seems to believe that config files live in your source tree, and are thus easily accidentally checked into revision control. That&#x27;s not where my config files live. My config files live in &#x2F;etc; or, if I want it to isolate a service to a user, I make a &#x2F;home&#x2F;user&#x2F;etc directory.<p>One could say, &quot;Don&#x27;t store passwords in the revision control system alongside your source.&quot; And that would be reasonable. But, there&#x27;s no reason to throw the baby out with the bath water.

Installing secrets on disk exposes them to potential leakage through backups. This is a major issue, since much less attention is typically paid to access management for backups than to production servers. Therefore I support the approach of providing secrets through the environment.<p>Once an application has been written to get its secrets from the environment, there is a question of how the secrets are obtained. They can be sourced from a file in an init script, but today we are seeing a lot of momentum towards containerized architecture, and the use of service discovery and configuration systems like etcd, zookeeper and consul.<p>However, secrets require much more attention to security concerns than the data that these tools are designed to handle. Least privilege, separation of duties, encryption, and comprehensive audit are all necessary when dealing with secrets material. To this end, we have written a dedicated product which provides management and access for secrets and other infrastructure resources (e.g. SSH, LDAP, HTTP web services). The deployment model is similar to the HA systems provided by etcd, consul, SaltStack, etc. It&#x27;s called Conjur (conjur.net).

Fundamentally, any secrets you store will have some mode of access - there&#x27;s a downside to each and every way of distributing them.<p>If you&#x27;re shelling out to commands you think might snarf credentials, the environment is easy for them to pick it out of, but if they&#x27;re running as the same user then they could probably read the secrets from the config file. If they aren&#x27;t running as the same user, you need a way of passing in the secrets - and we tend to come back to environment variables..<p>The good practice here is just to reset the environment when calling shell commands, as he notes. It&#x27;s not hard to do.

Ultimately, secrets need to live somewhere and need to be accessed as plain text. Just make sure that the access as small window is as possible, and try to obliterate it after use, if possible.<p>If one absolutely needs to centralize secrets (TLS&#x2F;SSL private keys, entropy sources, etc.) (at risk of SPOF or some HA setup), use some PSK style setup that delivers them directly, out-of-band (via separate NICs) or prioritized ahead of regular traffic. Keep it simple. Otherwise, prefer something like zookeeper with encrypted secrets (again PSK keying per box). Try to not deploy the same secret on every box, if possible. Also, try to avoid magic secrets if you can too (remove all local password hashes, use only auth keys).<p>If you&#x27;re uncomfortable with plaintext secrets, encrypt them (as end-to-end as possible) and require an out-of-band decryption key at the last possible moment.<p>It&#x27;s like having a secure document viewing system... ultimately, someone will need to browse just enough of the plaintext version, or it&#x27;s not a document viewing system.

Isn&#x27;t this what TPM was designed to avoid ?<p>Neither files nor env variables.<p>Most chipsets have a rather unused TPM function, and it should be possible to have developers and processes hook into that.<p>Perhaps using tmptool ? On master process startup ask user for passphrase, and use that to query the TPM stored values ?<p><a href="http://manpages.courier-mta.org/htmlman1/tpmtool.1.html" rel="nofollow">http:&#x2F;&#x2F;manpages.courier-mta.org&#x2F;htmlman1&#x2F;tpmtool.1.html</a>

Please don&#x27;t use environment variables to store secrets. There are to many angles - as stated by others - where this data may leak into files or processes.<p>I would propose to use just one folder like &#x2F;secret and put your config files in there. Exclude this folder from backup on all relevant hosts.<p>Then spend your time on security of your hosts, applications (OWASP) and monitoring &#x2F; alerting. Something that you have to do anyway.

Assuming you arn&#x27;t trying to go for top security, and just want a way to keep things safe from leaking due to errors and the such<p>why not just make use of the OS&#x27;s secrets store? for example, like how <a href="https://pypi.python.org/pypi/keyring" rel="nofollow">https:&#x2F;&#x2F;pypi.python.org&#x2F;pypi&#x2F;keyring</a> operates

we are in a similar situation and there is another approach I&#x27;d like to research. In order to have a distributed properties I was considering using something like consul[1] or etcd[2] which have some control access and load the required variables from upstart scripts<p>[1] <a href="https://www.consul.io/" rel="nofollow">https:&#x2F;&#x2F;www.consul.io&#x2F;</a><p>[2] <a href="https://github.com/coreos/etcd" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;coreos&#x2F;etcd</a>

You could still store your secret keys in ENV but encrypt them. Only your program has the method to decrypt them so in the case of an ImageMagick sub process it would access only your encrypted secret key with no knowledge to decrypt it.<p>Same thing while debugging : only the encrypted key is printed.

Ansible has a neat feature, called Ansible Vault, which lets you encrypt sensitive files. This in combination with dotenv-deployment works pretty well for our Rails Apps. The only thing I’m worried about is someone gaining unauthorised access to our serves and thus being able to read all the credentials stored inside the .env file especially the username &amp; password to our externally hosted db. Probably the only way to prevent this would be “to properly secure your server” and the use of an IDS? Anyone has any experience with someone hacking their servers and successfully preventing e.g. a db dump? In this particular case, how easy would it be to stop attackers in their tracks?

I typically store the env _name_ in the environment, and then use that in my apps to build a path to the file containing secrets (e.g. &#x2F;etc&#x2F;{mycompany}&#x2F;{environment}&#x2F;myapp.conf). The file is locked down by ACLs or permissions.

There&#x27;s a (not widely publicized) feature of Linux that enables secure key storage inside the kernel: <a href="https://www.kernel.org/doc/Documentation/security/keys.txt" rel="nofollow">https:&#x2F;&#x2F;www.kernel.org&#x2F;doc&#x2F;Documentation&#x2F;security&#x2F;keys.txt</a> Storing keys in the kernel has some advantages -- your key will never get inadvertently swapped to disk etc.<p>It&#x27;s been too long since I used it to remember the details, but I believe process-private keys are one of this API&#x27;s features.

I got bit by env vars a few years back, but due to performance issues in getenv() and ended up writing a whole bunch of PHP magic to ship config files safely and fast.<p>With pecl&#x2F;hidef, I can hook a bunch of text .ini files into the PHP engine and defines constants as each requests comes in.<p>Originally, it was written for a nearly static website, which was spending a ridiculous amount of time defining constants, which rolled over every 30 minutes.<p>Plus those .ini files were only readable to root, which reads it and then forks off the unprivileged processes.<p>But with the hidef.per_request_ini, I could hook that into the Apache2 vhost settings, so that the exact same code would operate with different constants across vhosts without changing any code between them.<p>Used two different RPMs to push those two bits so that I could push code &amp; critical config changes as a two-step process.<p>And with a bit of yum trickery (the yum-multiverse plugin), I could install two versions of code and one version of config, and other madness like that with the help of update-alternatives.<p>That served as an awesome A&#x2F;B testing setup, to innoculate a new release with a fraction of users hitting a different vhost for the micro-service.<p>I&#x27;m rambling now, but the whole point is that you need per-request&#x2F;per-user config overlay layers, for which env vars are horrible, slow and possibly printed everytime someone throws some diagnostics up.

Please consider the environment before printing this config?

We had to migrate our software from single tenant (per machine) to multiple tenant for our cloud offering, on a 11 year old code base.<p>We used Michael&#x27;s trick: environment variables pointing to config files works <i>unbelievably</i> well if you ever need to implement a multiple tenant cloud offering.<p>So apart from the security aspect, there&#x27;s the aspect that it is a more versatile design.

We&#x27;ve had good success with distributing our secrets using a GPG-encrypted file that we put in &#x2F;etc, not in the source code tree. We then use an ENV setting to point the app to the file. This gives us good flexibility (because one server can have multiple GPG files if we want, such as alpha&#x2F;beta&#x2F;gamma) and good encryption.

Clearing your environment variables after reading them, and only passing the ENVs required to perform the new task are pretty basic security measures. This was pretty common practice in the 90&#x27;s, and then I was hoping that would be one of the lessons out of ShellShock.

Ok I&#x27;m confused by &#x27;environment variable&#x27; vs files. How does one set an environment variable without putting it in a file on the particular server. Or by &#x27;file&#x27; in this article (and the 12 factor one) do they mean a file that in source control?

<i>1. It&#x27;s easy to grab the whole environment and print it out (can be useful for debugging) or send it as part of an error report for instance.</i><p>If you have software in your deployment that will send &quot;error reports&quot; to untrusted third parties then you have bigger problems than your shell environment.<p><i>2. The whole environment is passed down to child processes</i><p>If you don&#x27;t trust your child processes then you have bigger problems than your shell environment.<p><i>3. External developers are not necessarily aware that your environment contains secret keys.</i><p>And?<p>I&#x27;m not sure what you mean by &quot;external developer&quot; and what you expect them to do with your environment. E-Mail it out when an error occurs?<p>If you tolerate that kind of developer on your project then you.. oh well, see above.

why not just split it up with an OTP, that you store in the code (or a file), then the other half in the environmental variable. combine in code (or include the file). seems like that would work. (you need both parts.)<p>I think this article is a response to people&#x27;s practice of keeping API keys as an environmental variable so as to keep them off of the filesystem (or at least what git sees and checks in) so that they don&#x27;t accidentally publish them, as happened in that article article where some gem he was using to respect .gitignore didn&#x27;t work for some reason.<p>would this work as a solution?

ENV propagation to unwanted targets is a legit point.<p>Our most common crash report scenario is the airbrake gem sending crash reports from our rails app to errbit. I can confirm airbrake gem does not post any sensible env data.<p>Of course, this is only a good news for that specific case, others apps may transfer environment, and we can&#x27;t just wonder for each app installed &quot;what will it ever send ?&quot;.

I wrote a library to handle mixed configuration values by using asymmetric RSA encryption [1].<p>[1]: <a href="https://github.com/jacobgreenleaf/greybox" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;jacobgreenleaf&#x2F;greybox</a>

How do you suppose I do this in AWS where I have several auto-balanced servers? It kind of forces you to put it in environment variables.

while i agree that storing api keys in the code repository is not the best idea, i am curious about the suggestion of moving it into chef configs.<p>wouldnt that, in turn, also be stored in a code repository, likely accessible in the same way as the main coe repo? then, this feels like a non-solution to me.

While I could simply tell you to blank out ENV vars once you&#x27;ve internalized them, I will instead write an infinitely long essay on how they are &quot;considered harmful&quot; that contributes absolutely nothing back to society.

I created a quick gem (which you shouldn&#x27;t install) that demonstrates having some untrusted code in your app which will post all of your environmental variables to a 3rd party server: <a href="https://github.com/tibbon/env_danger" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;tibbon&#x2F;env_danger</a><p>Now, of course no one would install and run this... but I could imagine someone accidentally typing the name of a Gem wrong, someone accepting a bad PR (a sub-dependancy perhaps even doing so?), etc and somehow something untrusted getting in there. Yes, that means you have other problems, but it isn&#x27;t outside the realm of possibility that accidental access like this is had.<p>Just because it <i>shouldn&#x27;t</i> happen, doesn&#x27;t mean it will never happen.