Google and Microsoft Cheat on Slow-Start. Should You?

This isn't much of a secret. As it says in the article Google are lobbying to change the initial window size in the RFC. A lot of people here at Yahoo! want to see that too, and personally I think we should be more aggressive with our initial window, RFC be damned.<p>This topic was covered really well by Amazon's John Rauser at Velocity Conf: <a href="http://velocityconf.com/velocity2010/public/schedule/detail/11792" rel="nofollow">http://velocityconf.com/velocity2010/public/schedule/detail/...</a><p>To address the points in the conclusion:<p>1. Fast is good. Fast is also profit.<p>2. The net-neutrality argument here is totally bogus, anyone that knows how can up their slow-start window today if they choose to. There doesn't really have anything to do with traffic shaping.<p>3. Google have been using their usual data driven approach to support their proposal for IETF. We need a lot more of that. It's great. The only way we can really find out how the Internet in general will react to changes like this is to test them in some real world environment.<p>4. I agree, slow-start is a good algorithm with a very valid purpose. The real problem here is that the magic numbers powering it aren't being kept inline with changes to connectivity technology and increases in consumer/commercial bandwidth.

There's all sorts of latency problems caused by the congestion window size (and how it gets reset), because of how the algorithm works unless you're sending a continuous stream of data (which allows the congestion window to grow) than the window gets reset to it's initial size which can mean waiting for an ack round-trip before you get the whole message.<p>While it's not that big a deal if your users are local to you, if they're on a different continent each extra roundtrip can easily add 100ms.<p>I used to do TCP/IP tuning for low latency trading applications (sometimes you need to use a third party data protocol so can't just use UDP), this sort of stuff used to bite us all the time.<p>If latency is important it is worth sitting down with tcpdump and seeing how your website loads (i.e how many packets, how many acks, etc.) as often there are ways of tweaking connection setting (either via socket options or kernel settings) that can result in higher performance.<p>(Try using tcp_slow_start_after_idle if you're using a recent linux kernel; this won't give you a bigger initial window, but it means once your window size has grown it won't get reset straight away if you have a gap between data sends)

This is interesting, but the article and I differ greatly at this point: "Being non-standards-compliant in a way that privileges their flows relative to others seems more than a little hypocritical from a company that's making such a fuss about network neutrality."<p>No, no it's not. This has nothing to do with network neutrality; it's a purely server-side change/fix. Not only that, they're benefiting users without requiring anyone else to change while they wait for standards bodies to catch up. This is a similar scenario to HTML5 video, and distinctly more clear-cut than e.g. '802.11n draft' wireless routers in my opinion.

Google is proposing this should be allowed as a modification to rfc-3390. Their draft is <a href="http://tools.ietf.org/html/draft-hkchu-tcpm-initcwnd-01" rel="nofollow">http://tools.ietf.org/html/draft-hkchu-tcpm-initcwnd-01</a>. Active discussion of the issue may be found at <a href="http://www.ietf.org/mail-archive/web/tcpm/current/maillist.html" rel="nofollow">http://www.ietf.org/mail-archive/web/tcpm/current/maillist.h...</a>

Really interesting research, but man, if you really, <i>really</i> have to worry about premature optimization for your web app, I'd start with the usual bottlenecks first - i.e. anything that involves disk IO and/or processor work, such as databases and mathematical calculations.<p>Unless you are serving static content only (in which case you are hardly creating an "app"), the milliseconds you might save with TCP-level optimizations are <i>peanuts</i> in comparison to the multiple seconds your database and computations will be requiring.

There was a large discussion earlier about the subject. I posted detailed comments in that thread so I won't repost but just link. <a href="http://news.ycombinator.com/item?id=1143317" rel="nofollow">http://news.ycombinator.com/item?id=1143317</a>

Can anyone else repeat his experiment?<p>I tried repeating the experiment. I'm in Sweden, so, annoyingly, a request to google.com redirects to google.se. If I send my request directly to google.se, I get 9k response in 130ms and the initial window looks like 4 to me, i.e. I can't see anything unexpected happening.<p>I then tried repeating on Amazon EC2. I can't see anything unexpected there either, but the RTT from EC2 to google is only about 3ms, which means I can't assume that the ACKS don't get there.<p>(The original article author looks at how long the initial 3-way handshake takes and then assumes that all packets take that long, or, probably, half as long, i.e. he assumes that ACKS sent up to one RTT before a packet from google can't have arrived at google in time to affect that packet)<p>Can anyone else reproduce the experiment?<p>Other ideas: repeat from Sweden, but send a cookie so that I really get google.com. Repeat from EC2, but make sure I never send any ACKs after the three-way handshake. I'm not curious enough to do the latter, it's a fair bit of work.

It seems Linux does not have option to skip slow start and just use receiver's advertised window. Does anybody know where in net/ipv4/tcp.c this should be set?

Does anyone know what you would do to easily tune this for FreeBSD?

I believe the current limit for slow-start are not adapted to the current Internet anymore.<p>According to my own observations, the first 30Ko of my pages seem to be transfered faster then the next 30ko. It is not until much more is sent that the average throughput eventually get up to what it was during the first 30ko.<p>This is definitely weird.<p>Note: I am using Ubuntu on EC2 hosted VMs.<p>As a result, for as much as I can, I try to keep the size of my content below 30ko, using multiple concurrent HTTP requests.<p>I believe this is related to "slow-start" being pessimistic.<p>Unfortunately, "slow-start" is not configurable on Linux and I don't feel confident enough to go with some kernel level patch...<p>Any clue?

"It is better to ask for forgiveness than permission" - Rear Admiral Grace Hopper

We have been measuring google's "reachability" performance and it is quite amazing. The results of their tuning is that they can achieve downloading of their initial HTML in under ~250 milliseconds and many locations under 100 ms. The other thing the data shows is the standard deviation on the download times are very small making the site consistently load fast.<p><a href="http://www.yottaa.com/url/4be004065df8ca5a730001fb/reachability" rel="nofollow">http://www.yottaa.com/url/4be004065df8ca5a730001fb/reachabil...</a>

<i>"They actually managed to deliver the whole response in just 70ms, 30ms of which was spent generating the response"</i><p>Isn't part of that just the network latency? Based on the timestamps for the SYN and SYN-ACK it looks like a RTT of about 16ms.<p>EDIT: Nevermind.<p>Request was sent by the client at 00.017437<p>Request ACK was received by the client at 00.037139<p>RTT of about 20ms, so the request was received by the server around 00.027<p>First packet of the response was received by the client at 00.067151<p>67-27=40. Assuming a latency of 10ms it took 30ms to generate the request.

One should also note that when IE is talking to IIS, the request will be sent in the first packet and the initial response will be sent in the first ACK. You can actually complete a request and response (if small enough) in 3 packets. Also, when tearing down the connection, it's left half-open.<p><a href="http://osdir.com/ml/mozilla.devel.netlib/2003-01/msg00018.html" rel="nofollow">http://osdir.com/ml/mozilla.devel.netlib/2003-01/msg00018.ht...</a>

Mike Belshe - An Argument For Changing TCP Slow Start (Jan 11, 2010):<p><a href="http://sites.google.com/a/chromium.org/dev/spdy/An_Argument_For_Changing_TCP_Slow_Start.pdf" rel="nofollow">http://sites.google.com/a/chromium.org/dev/spdy/An_Argument_...</a>

On Ubuntu 8.04 (at least), you can set this per route via something like:<p>ip route change default via x.x.x.x dev eth0 initcwnd 6<p>but please test thoroughly if trying this.

Do app engine apps also serve like this?

Very interesting. Is such a thing configurable in Apache or nginx? It seems to be a rather rude behavior, but I'm curious how accessible it is.

very interesting post.

Interesting. But it really annoys me when people use "begs the question" incorrectly. Look it up!

Interesting, but there are so much more important things to consider before worrying about the load time. (i.e. 0 user experiencing 30 ms is far worst..)