I know this invites some eye-rolling, but can somebody explain to me why the k8s people insist on ignoring IPv6 and the possibilities of large address fields?<p>Down the bottom, which is where 'things we probably will never do' is when IPv6 comes in the door.<p>Azure (for instance) is a fully IPv6 enabled fabric. Microsoft "get" IPv6. They are all over it. They understand it, its baked into the DNA. So how come K8s people just kind of think "yea.. nah.. not right now"?<p>Because proxy Ipv6 at the edge is really sucky. We should be using native IPv6, preserve e2e under whatever routing model we need for reliability, and gateway the V4 through proxies in the longer term.<p>(serious Q btw)
Not directly related, but can someone recommend a beginners resource to understand Kubernetes networking? There are some good ones out there that explain basic Kubernetes concepts like pods, replicas etc. But networking seems to be a more complicated topic, and most intro guides skip over it.
For those wondering what's the difference between macvlan and ipvlan, the main ipvlan paper [0] summarizes its raison d'être:<p>> This is especially problematic where the connected next-hop e.g. switch is expecting frames from a specific mac from a specific port.<p>e.g.: if the host is attached to a managed switch with a strict security policy, macvlan would not work.<p>[0] <a href="https://www.netdevconf.org/0.1/sessions/28.html" rel="nofollow">https://www.netdevconf.org/0.1/sessions/28.html</a>
Just wanted to give a shout out to kube-router[1], a really fantastic solution if you want to use BGP, that will soon support not needing bgp by implementing a featureset similar to flannel's hostgw support. They are really good about addressing things in the open on their github[2]. BGP is, by definition, "web scale" as it runs most routing for the internet. Lower latency and much higher throughput than any sort of overlay network.<p>[1] <a href="https://www.kube-router.io" rel="nofollow">https://www.kube-router.io</a><p>[2] <a href="https://github.com/cloudnativelabs/kube-router" rel="nofollow">https://github.com/cloudnativelabs/kube-router</a>
It all about trade offs. We've built a CNI for k8s and have looked into all of the techniques described. It seems that Lyft's design is a direct reflection of their requirements.<p>To the extent your requirement match theirs, this could be a good alternative. The most significant in my mind is that it's meant to be used in conjunction with Envoy. Envoy itself has its own set of design tradeoffs as well.<p>For example, Lyft currently uses 'service-assigned EC2 instances'. Not hard to see how this starting point would influence the design. The Envoy/Istio model of proxy per pod also reflects this kind of workload partitioning. Obviously, a design for a small number of pods (each with their own proxy) per instance is going to be very different from one that needs to handle 100 pods (and their IPs), or more, per instance.<p>Another is that k8s network policy can't be applied since the 'Kubernetes Services see connections from a node’s source IP instead of the Pod’s source IP'. But I don't think this CNI is intended to work with any other network policy API enforcement mechanism. Romana (the project I work on) and the other CNI providers that use iptables to enforce network policy rely on seeing the pod's source IP.<p>Again, this might be fine if you're running Envoy. On the other hand, L3 filtering on the host might be important.<p>Also, this design requires that 'CNI plugins communicate with AWS networking APIs to provision network resources for Pods'. This may or may not be something you want your instances to do.<p>FWIW, Romana lets you build clusters larger than 50 nodes without an overlay or more 'exotic networking techniques' or 'massive' complexity. It does it via simple route aggregation, completely standard networking.
The author states:<p>>"Unfortunately, AWS’s VPC product has a default maximum of 50 non-propagated routes per route table, which can be increased up to a hard limit of 100 routes at the cost of potentially reducing network performance."<p>Could someone explain why increasing from 50 to 100 non-propagated routes in a VPC results in network performance degradation?
IIUC ENIs are limited to 2 per host on small instances, 15 per host on larger ones. Doesn't this approach limit the number of Pods per host? I'm already running about 20 pods per host, and I don't more containers per host is atypical.
How does it compare to AWS' own CNI plugin? <a href="https://github.com/aws/amazon-vpc-cni-k8s" rel="nofollow">https://github.com/aws/amazon-vpc-cni-k8s</a>