One does not simply go from a flat network to overlays. Overlays are slow, difficult, cause really odd failures and are often hilariously immature. They are the experimental graph database of the network world.<p>Just have a segregated network, and let the VPC/dhcp do all the hard stuff.<p>Have your hosts on the default VLAN(or Interface if your cloudy), with its own subnet (Subnets should only exist in one VLAN.) Then if you are in cloud land, have a second network adaptor on a different subnet. If you are running real steel, then you can use a bonded network adaptor with multiple VLANs on the same interface. (The need for a VLAN in a VPC isn't that critical because there are other tools to impose network segregation.)<p>Then use macvtap, or macvlan(or which ever thing that gives each container a macaddress) to give each container its own IP. This means that your container is visible on that entire subnet, either inside the host or without.<p>There is no need to faff with routing, it comes for free with your VPC/network or similar. Each container automatically has a hostname, IP, route. It will also be fast. As a bonus it call cane be created at the start using cloudformation or TF.<p>You can have multiple adaptors on a host, so you can separate different classes of container.<p>Look, the more networking that you can offload to the actual network the better.<p>If you are ever re-creating DHCP/routing/DNS in your project, you need to take a step back and think hard about how you got there.<p>70% of the networking modes in k8s are batshit insane. a large amount are basically attempts at vendor lock in, or worse someone's experiment thats got out of hand. I know networking has always been really poor in docker land, but there are ways to beat the stupid out of it.<p>The golden rule is this:<p>Always. Avoid. Network. Overlays.