What if your Pods need to trust self-signed certificates?

Don’t bake trust stores into container images. At work we have a single place where trusted certs (commercial CAs plus our company root CA) are configured, and then those are mounted in the appropriate format (big PEM file for &#x2F;etc&#x2F;ssl, trust store for Java apps, etc) as a volume in deployments in all clusters.<p>This means that trust store updates are not dependent on the OS vendor, they’re consistent between native&#x2F;Java&#x2F;etc, and updates don’t require rebuilding container images.

Plug (but it&#x27;s open source and free - and mentioned in the article!): We&#x27;ve been trying to address this in Kubernetes with trust-manager. [1] Trust bundles need to be a runtime concern and they need to support trusting both the old a new version of a cert to safely allow for rotation. It&#x27;s pretty simple but it seems to work well!<p>trust-manager also supports pulling in the Mozilla trust bundle which most Linux distros (and therefore most containers) use!<p>Handling trust of private [2] certificates is done poorly generally across many orgs and platforms, not just Kubernetes. There are lots of ways of shooting yourself in the foot - particularly when it comes to rotating CA certificates. I think there&#x27;s a lot of space here for new solutions here!<p>[1] <a href="https:&#x2F;&#x2F;cert-manager.io&#x2F;docs&#x2F;projects&#x2F;trust-manager&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;cert-manager.io&#x2F;docs&#x2F;projects&#x2F;trust-manager&#x2F;</a><p>[2] I try to avoid &quot;self-signed&quot; in this use case because its literal meaning is that the certificate signs itself using its own key, which is what root certificates do. The Let&#x27;s Encrypt ISRG X1 root certificate is self-signed but it&#x27;s definitely not what I&#x27;d call a &#x27;private CA&#x27;; see <a href="https:&#x2F;&#x2F;letsencrypt.org&#x2F;certificates&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;letsencrypt.org&#x2F;certificates&#x2F;</a>

That means you designed your architecture incorrectly; get an internal CA and automate it, then pod can just have long term root CA embedded in it.

I&#x27;ve been working on a different approach to try to make it as easy as possible to use certificates from public CAs for private networks. I think it&#x27;s really hard to get all your trust stores updated with private CA certs; there&#x27;s just too many of them. I&#x27;ve encountered devices that weren&#x27;t properly configured at every company I&#x27;ve worked at. Any time you&#x27;ve got a BYOD policy, a variety of mobile devices, or engineers are spinning up their own VMs (which don&#x27;t copy the trust store of the host) you&#x27;re going to see certificate warning messages. Back when every computer ran Windows and was chained to the desk it was easier for IT departments to update trust stores, but the world has changed and it&#x27;s a losing battle.<p>I think the better approach is to just use certificates from public CAs on private networks. There&#x27;s some big issues to address: how does the public CA verify internal servers which don&#x27;t permit external connections, is leaking internal subdomain names a security issue? But I think these are solvable.<p>getlocalcert.net[1] is my project to simplify the process. Currently you can register a subdomain and use the ACME DNS-01 protocol via my site to issue certificates from from providers like Let&#x27;s Encrypt. All you need to issue a certificate is a getlocalcert API key and the ability to connect out-bound to the Let&#x27;s Encrypt API and the getlocalcert API. It supports a couple ACME clients and it should support cert-manager, mentioned in the article, but I haven&#x27;t had a chance to test and properly document it[2]. I&#x27;m not doing anything that other subdomain registrars or DNS providers can&#x27;t do, but I&#x27;m trying to address the public-CA-on-private-server niche as best as possible. Longer term I&#x27;m looking to add bring-your-own-domain support and other improvements.<p>[1] <a href="https:&#x2F;&#x2F;www.getlocalcert.net&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.getlocalcert.net&#x2F;</a><p>[2] <a href="https:&#x2F;&#x2F;docs.getlocalcert.net&#x2F;acme-clients&#x2F;cert-manager&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;docs.getlocalcert.net&#x2F;acme-clients&#x2F;cert-manager&#x2F;</a>

You can also do sidecar proxy pattern where the the sidecar handles TLS termination (incoming or outgoing). That&#x27;s basically service mesh light.<p>You can also use MutatingWebhooks to inject initContainers or side cars to achieve that pattern.<p>MutatingWebhooks increase the runtime complexity since you&#x27;re jamming more config in but they can handle vendor&#x2F;3rd party software.<p>Depending on the CNI, insecure TLS flags might not actually be that bad. There are some CNIs floating around that can handle encryption. In addition, software defined networks like AWS might not offer encryption but they do guarantee messages are authenticated so you can trust the source IP

In the age of cheap certs, it&#x27;s not clear to me why we have these enterprises with self-signed certificates. It definitely is a thing, but it seems to be tech debt that we should fix.