Stock Ubuntu needs SR-IOV driver to get to the actual bandwidth limit on ec2, it makes a lot of difference. We routinely get to ~2 Gbps down from S3 with that setup (using largest instance types).<p>edit: Gbps not GBps
If you are pulling large files from S3 we have found that they can be sped up by requesting multiple ranges simultaneously. It is easy to hit 5Gb/s or 10Gb/s on instances with the necessary bandwidth, accessing a single file, or multiple files. We have not encountered a limit on S3 itself. YMMV.
AWS has a limit on the total throughput any one <i>account</i> can have to S3, so the more CPUs OP adds, the worse OPs performance will be on each one. I suspect the other providers have the same restriction.<p>I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node.<p>This did not seem to be accounted for.<p>EDIT: OP says below it was from one instance, so what I said doesn't apply to this writeup.
When I see things like "data set size 150GB" and "1000 CPUS" I just naturally assume they are all in memory and never come from disk :-)
with kernel tuning, S3 performance improves (and will probably improve on GC/Azure as well). Also, author uses Ubuntu 14.4 (see <a href="https://twitter.com/Zbjorn/status/684492084422688768" rel="nofollow">https://twitter.com/Zbjorn/status/684492084422688768</a>), which doesn't use AWS "Enhanced networking" by default. Would be interesting to see results for tuned systems.
Very interesting comparison, glad to see it. I don't have a comment on the content itself but I do have a note on the presentation.<p>The colors used for S3 and Azure Storage in the graphs are very near indistiguishable to me, as I have moderate red-green colorblindness. It's easier to tell apart on the bar graphs, since the patches of color are much larger, although I still have to work at it, and use the hints of the labels, but on the line graphs, it's basically impossible to tell apart. A darker shade of green would solve the problem for me personally, but I'm not all that bad a case, nor an expert on the best shades to pick for general color-blindness accessibility.<p>Just something to think about when presenting data like this.
In S3 tests on c3.8xlarge instances, I've seen 8 Gbps throughput on both uploads and downloads using parallelized requests. Testing with iperf between two of the same instances maxed out about 8 Gbps as well so the throughput limitation is likely EC2 networking rather than S3.<p>These tests were done over a year ago so bandwidth limitations on EC2 may have changed since.<p>This testing was with <a href="https://github.com/rlmcpherson/s3gof3r" rel="nofollow">https://github.com/rlmcpherson/s3gof3r</a>
Thanks for sharing your research - I've been up to the neck in EC2 migrations and trying to benchmark as I go... S3 is the neck chunk of work. Rock on!
What missing from description is network setup.
Is it ec2 classic, VPC? Is ec2 getting to s3 through IG? Hopefully not through NAT. There is also VPC endpoint to s3. Which all may have different performance profiles especially with multiple instances.
I'd be interested to see how AWS' Elastic File System (EFS) compares (though I'd imagine it's not great, given it's mounted via NFS)
How reliable is Azure? For example the story of Gitlab on Azure was a disaster: <a href="https://news.ycombinator.com/item?id=10781263" rel="nofollow">https://news.ycombinator.com/item?id=10781263</a> Something like that wouldn't happen on AWS, GC, Softlayer, etc.