I'm evaluating solutions for secrets management in relation to a distributed microservice architecture and am curious to hear what everyone else out there does. Some options I've considered:<p>- Git-crypt and deploying secrets along with binaries<p>- Hashicorp Vault<p>- Square Keywhiz<p>- AWS KMS<p>- Lyft Confidant<p>- Roll your own<p>All seem to have pros and cons depending on use cases and how mission critical the service you are offering is.<p>So what do you do to solve this problem in your world?
You should check out Daniel Somerfield's talk at OWASP, "Turtles All the Way Down: Storing Secrets in the Cloud and the Data Center"<p><a href="https://youtu.be/OUSvv2maMYI" rel="nofollow">https://youtu.be/OUSvv2maMYI</a>
We use Kubernetes, which includes its own secrets API:<p><a href="http://kubernetes.io/v1.1/docs/user-guide/secrets.html" rel="nofollow">http://kubernetes.io/v1.1/docs/user-guide/secrets.html</a><p>I can't remember which issue this was on, but it seemed like there was some discussion on their GitHub project about making pluggable secrets backends (HashiCorp's Vault was mentioned).<p>Kubernetes' secrets API is still very basic, but I think the fundamental concept is very sound and has a great foundation to continue building on.
We have started work on exposing Hashicorp Vault secrets via FUSE and Docker volumes. The expectation is your containers will just mount secrets via a mount like /secret in the container.<p>The project is brand new and we'd love to hear your feedback: <a href="https://github.com/asteris-llc/vaultfs" rel="nofollow">https://github.com/asteris-llc/vaultfs</a>
We run a PHP hosting platform and that tickled us as well. We were especially upset with the common sense that storing secrets in ENV vars is a good idea — in PHP those vars are easily exposed. See our blog post: <a href="http://blog.fortrabbit.com/how-to-keep-a-secret" rel="nofollow">http://blog.fortrabbit.com/how-to-keep-a-secret</a> — here we suggested:<p>1. create a secret key, store it with the code of your App
2. store the encrypted credentials in env vars<p>Later on we even launched our own solution for our clients, an app_secrets.yml file, which can be edited via Dashboard. <a href="http://help.fortrabbit.com/secrets" rel="nofollow">http://help.fortrabbit.com/secrets</a><p>The nice thing is, that this file is partly managed by the platform for it's own credentials and partly by the user.<p>That has been running for a while now. The adaption rate is low until now. It turned out that not everything will fit into that ONE fault. Blackfire.io and NewRelic run as PHP extensions, thus the API-keys are stored with the extension setting.<p>We have also discussed to implement an some open source "Secret as a Service" but came to the conclusion that this can too easily turn into to be a SPOF.<p>I am amazed that this topic is getting discussed again and I have learned about many new concepts here.
I wrote some of my thoughts on the topic, and the primary motivation behind SOPS [1] (which uses PGP and KMS): <a href="https://jve.linuxwall.info/blog/index.php?post/2015/10/01/Introducing-Sops%3A-an-editor-of-encrypted-file-that-uses-AWS-KMS-and-PGP" rel="nofollow">https://jve.linuxwall.info/blog/index.php?post/2015/10/01/In...</a><p>The initial trust problem boils down to trusting the API that controls the provisioning of your infrastructure. Failing that, you have to ask a human to manually authorize new nodes to retrieve secrets (that's how puppet approves new agent certs).<p>[1] <a href="https://github.com/mozilla/sops" rel="nofollow">https://github.com/mozilla/sops</a>
We have an open source solution called Cryptex[0] to handle this. It's better explained by this blog post[1] that gives the thinking and configuration necessary for most scenarios.<p>[0]: <a href="https://github.com/TechnologyAdvice/Cryptex" rel="nofollow">https://github.com/TechnologyAdvice/Cryptex</a>
[1]: <a href="http://technologyadvice.github.io/lock-up-your-customer-accounts-give-away-the-key/" rel="nofollow">http://technologyadvice.github.io/lock-up-your-customer-acco...</a>
There's also blackbox by StackExchange <a href="https://github.com/StackExchange/blackbox" rel="nofollow">https://github.com/StackExchange/blackbox</a>
For AWS users, KMS's GenerateDataKey is a simple way to store secrets locally in a way that reuses your IAM policies. You can also use grants and EncryptionContext to restrict the ability to decrypt secrets in a very fine-grained manner. As a bonus, all decrypts are logged in CloudTrail. The KMS docs are awful but if you're on AWS then it is worth checking out!
Conjur (<a href="https://www.conjur.net/" rel="nofollow">https://www.conjur.net/</a>) has been working well the users of our PaaS. It's a self hosted commercial product.
Is there any detailed opinion on good and bad approaches to this problem? It is clearly a hot topic.<p>I'm sure each product listed conforms to one from a small set of design patterns. Has any credible analysis of these designs been published? Are competing offerings likely to evolve toward a stable converged solution? Or is there something in this problem that remains fundamentally unsolved?<p>I appreciate it's turtles all the way down, but I'm wondering if anyone has proven the merits of some approaches over others, or components within the approaches at least.
We use <a href="https://github.com/meltwater/secretary" rel="nofollow">https://github.com/meltwater/secretary</a> . The key differences with Secretary is that plaintext secrets are never stored to disk or otherwise made visible outside the container.
We ended up creating <a href="https://github.com/meltwater/secretary" rel="nofollow">https://github.com/meltwater/secretary</a> to allow storing encrypted secrets in config files checked into Git by devteams. The encrypted secrets are passed as env vars through the continuous delivery pipeline, Mesos/Marathon and into containers. They're then decrypted and injected into the app environment at runtime, safely inside the container.<p>At startup the container reaches out to the Secretary daemon that holds the master keys, using public key cryptos to authenticate itself. The Secretary deamon uses Marathon to authenticate containers (checking their public keys stored in env vars) and validate that they're authorized for the specific secret in question (checking that the encrypted secret is indeed part of the containers env vars).<p>Meaning that Marathon is the single source of trust of which container can access what secrets. The problem then becomes controlling who and how changes are made to the Git repo containing the CD config, which is something Github does well with roles, status/deployment API and pull requests.<p>We had a similar problem as some describe with the distribution of the initial secret (i.e. Vault token) and one time Vault tokens being cumbersome in dynamic scaling envs. We didn't want the cleartext token ending up in config files nor in the <a href="https://github.com/meltwater/lighter" rel="nofollow">https://github.com/meltwater/lighter</a> config we use to drive our continuous delivery pipelines that go into Mesos/Marathon. We also had some other aspects like<p>* Wanting to keep secrets, app config and code versions promoted together throughout or deployment pipelines. Seeing secrets as another type of app config we wanted to track all config and versions for an app in the same way, in the same place to avoid mismatches or deployment dependencies.<p>* Wanting to enable our very independent devteams to easily manage secrets for their services, same was as they manage the app config, versions and rollout of their services. And delegate management of what service is authorized for what secrets to devteams (with both automated checks for unencrypted secrets, and some gentle manual coaching post-commit)<p>* Versioning and rolling upgrades for secrets? E.g. how to roll out a new secret in a Marathon rolling upgrade? Creating and managing versioned keys in Vault seemed somewhat cumbersome.<p>Perhaps something like that could be used to solve your initial secret distribution problem or even handle the secrets themselves until Vault has solved the initial secret problem..?
Azure Key Vault! Disclosure: am dev in Azure, although not on this specific product.<p><a href="https://azure.microsoft.com/en-us/services/key-vault/" rel="nofollow">https://azure.microsoft.com/en-us/services/key-vault/</a>
Because we are already using Consul we went with Vault. We are using it in a POC type setup now (only for a few services/scripts) but so far it's been pretty easy to work with. The API is fairly easy to use outside of the fact that there is no search function (or wasn't last time I checked). The documentation could be better but since it's a public project and I haven't submitted anything I'm not going to bash that :)
The fact that it's a single binary is another thing we liked. just drop it out somewhere and run it.
I'm pretty happy with this solution from Strongauth.<p><a href="http://keyappliance.strongauth.com/" rel="nofollow">http://keyappliance.strongauth.com/</a><p>You can secure the root for it with TPM or HSM.
Another option for your list, which you'll have to evaluate for your use case: <a href="https://wiki.openstack.org/wiki/Barbican" rel="nofollow">https://wiki.openstack.org/wiki/Barbican</a><p>I've been evaluating most of these same options for my use case, but haven't made any decisions yet.
I used Marathon and Mesos and rolled my own pub/priv encryption for our developers JSON (encrypted the ENV parameters POSTed to Marathon): <a href="https://github.com/malnick/mantle" rel="nofollow">https://github.com/malnick/mantle</a>
You can encrypt and deploy secrets using Distelli.<p><a href="https://www.distelli.com/docs/user-guides/securing-your-applications" rel="nofollow">https://www.distelli.com/docs/user-guides/securing-your-appl...</a><p>Disclaimer: I'm the founder at Distelli
it's a huge pain point for us. We're a .NET shop rolling our own that mimics/overlays app.config and web.config patterns for both dev and production usage. Our concern is less on how do you get the secrets to the box (though that's obviously important) and more on how do you keep an attacker who has started penetrating your infrastructure from gaining control of the infrastructure that holds your secrets.
I'd choose the one that I am the most comfortable with and is less obtrusive to the rest of my stack.<p>Whatever you choose, make sure you are comfortable with it, it's easy to deploy and work with.