2016 Spark Summit East Keynote

Very excited to hear the plans for GraphFrames - finally GraphX getting some attention!<p><a href="https:&#x2F;&#x2F;spark-summit.org&#x2F;east-2016&#x2F;events&#x2F;graphframes-graph-queries-in-spark-sql&#x2F;" rel="nofollow">https:&#x2F;&#x2F;spark-summit.org&#x2F;east-2016&#x2F;events&#x2F;graphframes-graph-...</a>

How advanced is the Structured Streaming functionality? Looking at the JIRA [1] I cannot find even design prototype there, which is kind of strange if they want to have it ready by end of April. But as there was a presentation on the topic at the summit [2], I hope it&#x27;s just developing it without discussion on JIRA.<p>[1] <a href="https:&#x2F;&#x2F;issues.apache.org&#x2F;jira&#x2F;browse&#x2F;SPARK-8360" rel="nofollow">https:&#x2F;&#x2F;issues.apache.org&#x2F;jira&#x2F;browse&#x2F;SPARK-8360</a> [2] <a href="https:&#x2F;&#x2F;spark-summit.org&#x2F;east-2016&#x2F;events&#x2F;keynote-day-3&#x2F;" rel="nofollow">https:&#x2F;&#x2F;spark-summit.org&#x2F;east-2016&#x2F;events&#x2F;keynote-day-3&#x2F;</a>

I think the more exciting announcement was Databricks community edition, which allows you to use 2.0:<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11126179" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11126179</a>

Slide 10:<p>&gt; CPU speeds have not kept up with I&#x2F;O in the past 5 years.<p>I presume he means the other way around?<p>Also, what does he mean by native memory management? Does he mean off-heap allocation?<p>And what&#x27;s he referring to regarding code generation?

Has it become easier to run ad hoc queries with spark? I remember a year ago that the only available solution was the job server by ooyala. Which seems to be a missing feature of core Spark, and isn&#x27;t something I was willing to bet my product on.<p>Datastax evangelized people to use Spark to run queries over Cassandra but it looks so awkward and time consuming to copy jars around to the master, basically you need a dev ops team to this and even more scriptology for production.

Are Spark streams ever going to reach a point where you can just have a table sitting in memory aggregating data and then you run queries on the <i>whole</i> thing without having to worry about windowing or anything?

Last Spark Summit the videos were up on Youtube 1-2h after each talk. Anybody knows where to find the ones from this summit?

Started using Spark 1.6 a few months ago. Excited for the Kafka Connector feature.