Static IPs for Serverless Containers

Hi! This is a blog post sharing some low-level Linux networking we&#x27;re doing at Modal with WireGuard.<p>As a serverless platform we hit a bit of a tricky tradeoff: we run multi-tenant user workloads on machines around the world, and each serverless function is an autoscaling container pool. How do you let users give their functions static IPs, but also decouple them from compute resource flexibility?<p>We needed a high-availability VPN proxy for containers and didn&#x27;t find one, so we built our own on top of WireGuard and open-sourced it at <a href="https:&#x2F;&#x2F;github.com&#x2F;modal-labs&#x2F;vprox">https:&#x2F;&#x2F;github.com&#x2F;modal-labs&#x2F;vprox</a><p>Let us know if you have thoughts! I&#x27;m relatively new to low-level container networking, and we (me + my coworkers Luis and Jeffrey + others) have enjoyed working on this.

this is a really neat writeup! the design choice to make each &quot;exit node&quot; control the local wireguard connections instead of a global control plane is pretty neat.<p>an unfinished project I worked on (<a href="https:&#x2F;&#x2F;github.com&#x2F;redpwn&#x2F;rvpn">https:&#x2F;&#x2F;github.com&#x2F;redpwn&#x2F;rvpn</a>) was a bit more ambitious with a global control plane and I quickly learned supporting multiple clients especially anything networking related is a tarpit. the focus on linux &#x2F; aws specifically here and the results achievable from it are nice to see.<p>networking is challenging and this was a nice deep dive into some networking internals, thanks for sharing the details :)

Thanks for sharing. This new feature is neat! It might sound a bit out there, but here&#x27;s a thought: could you enable assigning unique IP addresses to different serverless instances? For certain use cases, like web scraping, it&#x27;s helpful to simulate requests coming from multiple locations instead of just one. I think allowing requests to originate from a pool of IP addresses would be doable given this proxy model.

So much work seems to go into working around the limitations of IPv4 instead of towards a fully IPv6 capable world.

Static IPs for allowlists need to die already. Its 2024, come on, surely we can do better than this

This is just what I needed. Chefs kiss.

&gt; Modal has an isolated container runtime that lets us share each host’s CPU and memory between workloads.<p>Looks like Modal hosts workloads in Containers, not VMs. How do you enforce secure isolation with this design? A single kernel vulnerability could lead to remote execution on the host, impacting all workloads . Am I missing anything?

Why is it important to have a static outbound ip address?

Do you block certain ports?

It&#x27;s going to take years for orgs to adopt IPv6 and mTLS+JWT&#x2F;OIDC.<p>Even longer for QUIC&#x2F;H3.

I guess my first question is, why is this built on IPv4 rather than IPv6...

side question: what do you use to make the diagrams?

Couldn&#x27;t a NAT instance in-front of containers accomplish this as well (assuming only needed for outbound traffic)? The open source project fck-nat[1] looks amazing for this purpose.<p>[1] <a href="https:&#x2F;&#x2F;fck-nat.dev&#x2F;stable&#x2F;" rel="nofollow">https:&#x2F;&#x2F;fck-nat.dev&#x2F;stable&#x2F;</a>