Introducing DataFrames in Spark for Large Scale Data Science

Has anyone had some good experiences with Spark?<p>I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler.<p>After doing a bit of searching, it seems this is actually a known issue <a href="https://issues.apache.org/jira/browse/SPARK-4454" rel="nofollow">https:&#x2F;&#x2F;issues.apache.org&#x2F;jira&#x2F;browse&#x2F;SPARK-4454</a> and there&#x27;s been a fix on their github for a while: <a href="https://github.com/apache/spark/pull/3345" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;apache&#x2F;spark&#x2F;pull&#x2F;3345</a> and yet in that time two releases have swung by bringing a tonne of features.<p>I ended up having to drop Spark ultimately because I wasn&#x27;t confident about putting it in to production (the random OOMs and NPEs during development weren&#x27;t great either). Does anyone have any positive experiences?

Spark the platform seems awesome. I&#x27;m somewhat less convinced by mllib - I&#x27;m not sure there are as many use cases for distributed machine learning as people seem to think (and I would bet that a good deal of companies that use distributed ML don&#x27;t really need it). I&#x27;ve seen a lot of tasks that could be handled by simpler, faster algos on large workstations (you can get 250 GB RAM from AWS for like $4.00&#x2F;hr). I&#x27;d love to hear counterarguments, though!

I&#x27;m one of the authors of the blog post as well as this new API. Feel free to ask me anything.

Hey... don&#x27;t downvote Reynold Xin, author of the post as dupe when he says AMA.

A replica of this benchmark on my laptop running R has this running in about 1&#x2F;4 second. Seems like a pretty trivial benchmark?<p>library(data.table)<p>x = data.table(a=sample(10,10e6,replace=TRUE),num=sample(100,10e6,replace=TRUE)) t1=proc.time(); x[,sum(num),by=a]; print(proc.time()-t1)<p><pre><code> user system elapsed 0.209 0.032 0.245</code></pre>

Native question - how is this different than Spark SQL and things like project zeppelin?

This is great news! Where do I see the source for this?

Will the DataFrame API work with Spark Streaming?