TL;DR: Fastly needs full routing tables on all CDN nodes in order to determine which is the best transit path to push out content through. In order to save money, they used a programmable Arista switch instead of a traditional router. Their solution is to reflect BGP routes via the switch to the nodes and and fake direct connectivity between the nodes and the transit provides, so that nodes can directly push out content to whichever transit provider they determine is best on a per packet basis.<p>Please correct me if I'm wrong.<p>Maybe I'm just obtuse, but I found the blog post confusing about what it really is about and long winded, taking a very long time to come to the point. There is a severe lack of context in the beginning, it would have tremendously benefited from the what and the why of what they are trying to do.
Interesting approach. So Fastly is offloading the full routing table to their carriers' router(s). That's because routers that can hold full BGP tables are expensive to purchase and maintain. But to retain some form of control, they're terminating eBGP at the switch and using iBGP to disseminate (inject) the providers' route (next hop).<p>I feel (I don't have direct experience with this setup) like they're just offloading some compute power (therefore cost) to the hosts. So the cost is automatically spread out across their relatively massive edge nodes. A line showing a router plus support costing $100,000 looks bad in expenditures vs showing a server plus integrated routing showing $2500.<p>I'm curious about how this impacts Varnish considering how table look ups can be bus-expensive during odd route changes/flaps (storms). %sys must go through the roof as a result.
Also note Spotify published their own stuff also on Arista hardware<p><a href="https://labs.spotify.com/2016/01/26/sdn-internet-router-part-1/" rel="nofollow">https://labs.spotify.com/2016/01/26/sdn-internet-router-part...</a><p>Podcast where David Barroso talks about it:<p><a href="http://blog.ipspace.net/2015/01/sdn-router-spotify-on-software-gone-wild.html" rel="nofollow">http://blog.ipspace.net/2015/01/sdn-router-spotify-on-softwa...</a><p>Arista blog post:<p><a href="https://eos.arista.com/spotifys-sdn-internet-router/" rel="nofollow">https://eos.arista.com/spotifys-sdn-internet-router/</a><p>Disclaimer I work at Arista
Great read, love the "hey what does it need?" approach rather than the "how is this done?" approach. Tut Systems had bought one of the first "hotel internet" companies back in the 90's which used a similar approach by subverting the ARP protocol, when you connected any thing you tried to ARP for it would respond "Yup, that's me! Send me your packets" and you would end up at the "Give us your credit card" signup.<p>The nice thing is that at this level networking is really simple. And if you can get access to the internals of switches to craft behaviors at that level, it is a pretty good way to go.
That's pretty impressive! As a side note, a writeup on BGP security: <a href="https://security.stackexchange.com/questions/56069/what-security-mechanisms-are-used-in-bgp-and-why-do-they-fail" rel="nofollow">https://security.stackexchange.com/questions/56069/what-secu...</a>
If you like this, you'll like the ONS youtube channel.[1] In particular, the keynotes by Vahdat are pretty amazing.<p>[1] <a href="https://www.youtube.com/channel/UCHo2uqQqpmE_Cg5b4qiUpUg" rel="nofollow">https://www.youtube.com/channel/UCHo2uqQqpmE_Cg5b4qiUpUg</a>
Amazon, Google, etc. etc. all these companies building their own custom network devices, and so much of it coming back to both "It does too much, most of which we don't need" and "We want to do our own thing at that layer".<p>As James Hamilton noted at the AWS Re:Invent convention, not only is there they overhead and development expense of these unneeded components, just the sheer complexity of the application running is inevitably leading to bugs and unexpected behaviour. By simplifying the device to just do the few things you actually need it to do, you end up more performant, and more reliable.<p>I wonder if the entrenched network appliance providers will wake up?
They'll want to be looking at using MPLS and EPE techniques now that there's support for it on their Arista platforms. This L2 technique is arcane, going to be painful to scale and reapply generally to other areas.