I applaud the effort but is this really "big data" - the largest data sets they seem to test are ~150GB, that would fit comfortably on my Mac Book Pro a number of times over. Many of these systems being tested are designed to scale efficiently when the data starts peaking > 5TB and therefore I am dubious about the median response time results - things that work well for small datasets (where small is defined as < 1TB) easily fall apart when you scale them up a little bit more.
The author had a terrible brain cramp in the sentence "Redshift uses columnar compression which allows it to bypass a field which is not used in the query."<p>That totally confuses columnar compression with columnar I/O, an error I've been railing against for several years, e.g. in <a href="http://www.dbms2.com/2011/02/06/columnar-compression-database-storage/" rel="nofollow">http://www.dbms2.com/2011/02/06/columnar-compression-databas...</a> (I.e., ever since Oracle tried to popularize the confusion.) But this is a particularly bad instance.
Spark is a big deal (<a href="http://spark.incubator.apache.org/" rel="nofollow">http://spark.incubator.apache.org/</a>). It's a next-gen open source cluster-computing system built on top of the Berekely Data Analytics Stack (BDAS - <a href="https://amplab.cs.berkeley.edu/software/" rel="nofollow">https://amplab.cs.berkeley.edu/software/</a>), which includes Mesos, Spark, SparkStreaming, Shark, and GraphX (to name a few).<p>Mesos is the foundation of the stack, and Spark started out as a research project because they needed something to run on Mesos. But you can also run Hadoop on Mesos, and you can run Spark and Hadoop on the same Mesos cluster. Twitter runs almost everything on Mesos and works directly with AMPLab on the project.<p>See Benjamin Hindman's presentation on "Managing Twitter Clusters with Mesos" (<a href="http://www.youtube.com/watch?v=37OMbAjnJn0&list=PL9F5093F238695612&index=5" rel="nofollow">http://www.youtube.com/watch?v=37OMbAjnJn0&list=PL9F5093F238...</a>).<p>SparkStreaming replaces the need for Storm and handles failures/stragglers better. As Nathan Marz the creator of Storm said, "Spark is interesting because it extends MapReduce with a new primitive that allows Pregel to be built on top of it. So Spark is both Hadoop and Pregel" (<a href="http://nathanmarz.com/blog/thrift-graphs-strong-flexible-schemas-on-hadoop.html#comment-334743458" rel="nofollow">http://nathanmarz.com/blog/thrift-graphs-strong-flexible-sch...</a>).<p>GraphX (<a href="https://amplab.cs.berkeley.edu/publication/graphx-grades/" rel="nofollow">https://amplab.cs.berkeley.edu/publication/graphx-grades/</a>) is new, and it's GraphLab2 built on Spark, which enables fast processing of Pregel-like algorithms. GraphLab2 (<a href="http://graphlab.org/" rel="nofollow">http://graphlab.org/</a>) includes a suite of machine learning tools (similar to Mahout).<p>Berkeley's "Analyzing Big Data with Twitter" series (<a href="http://www.youtube.com/playlist?list=PLE8C1256A28C1487F" rel="nofollow">http://www.youtube.com/playlist?list=PLE8C1256A28C1487F</a>) includes a couple of presentations related to the project.<p>The last presentation is by the Spark-lead Matei Zaharia (<a href="http://www.cs.berkeley.edu/~matei/" rel="nofollow">http://www.cs.berkeley.edu/~matei/</a>), and he gives a good high-level overview: "Analyzing Big Data with Twitter: Spark" (<a href="http://www.youtube.com/watch?v=rpXxsp1vSEs&list=PLE8C1256A28C1487F&index=15" rel="nofollow">http://www.youtube.com/watch?v=rpXxsp1vSEs&list=PLE8C1256A28...</a>).<p>There is another presentation in the series by GraphLab-lead Joey Gonzalez (<a href="http://www.cs.cmu.edu/~jegonzal/" rel="nofollow">http://www.cs.cmu.edu/~jegonzal/</a>): "GraphLab: Big Learning with Graphs" (<a href="http://www.youtube.com/watch?v=E1LwqtBdPYs" rel="nofollow">http://www.youtube.com/watch?v=E1LwqtBdPYs</a>).<p>See also the GraphLab paper "PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs" (<a href="https://www.usenix.org/system/files/conference/osdi12/osdi12-final-167.pdf" rel="nofollow">https://www.usenix.org/system/files/conference/osdi12/osdi12...</a>) and presentation (<a href="https://www.usenix.org/conference/osdi12/167-powergraph-distributed-graph-parallel-computation-natural-graphs" rel="nofollow">https://www.usenix.org/conference/osdi12/167-powergraph-dist...</a>).<p>AMPLab has plenty of sponsors (<a href="https://amplab.cs.berkeley.edu/sponsors/" rel="nofollow">https://amplab.cs.berkeley.edu/sponsors/</a>), both Twitter and Yahoo are adopting it, and evidently Facebook and Amazon may too (<a href="http://www.wired.com/wiredenterprise/2013/06/yahoo-amazon-amplab-spark/all/" rel="nofollow">http://www.wired.com/wiredenterprise/2013/06/yahoo-amazon-am...</a>). The more I learn about the BDAS stack, the more I think it's going to usurp Hadoop/Storm.<p>For more on AMPLab, see...<p>AMPLab Stack Presenations: <a href="http://www.youtube.com/user/BerkeleyAMPLab/feed?activity_view=5" rel="nofollow">http://www.youtube.com/user/BerkeleyAMPLab/feed?activity_vie...</a><p>Slides/Summaries: <a href="http://ampcamp.berkeley.edu/amp-camp-one-berkeley-2012/" rel="nofollow">http://ampcamp.berkeley.edu/amp-camp-one-berkeley-2012/</a>
For a starter guide to Amazon Redshift there is a book
<a href="http://www.amazon.com/Getting-Started-Amazon-Redshift-Stefan/dp/1782178082/" rel="nofollow">http://www.amazon.com/Getting-Started-Amazon-Redshift-Stefan...</a>