Joining a billion rows 20x faster than Apache Spark

Am I reading this correctly? The testbed was a single laptop? A big part of spark is the distributed in-memory aspect so I'm not sure I understand why any of these numbers mean anything.

I apologize in advance, but whenever people claim to use a in-memory big-data system, how exactly does this end up working?<p>You can only stuff so much into memory, so you can scale up vertically in-terms of memory, unless you buy a massive big-iron POWER box, you scale out horizontally. But with each of these in-memory appliances, what happens when you need to spill out to disk?<p>In essence why should one bother with these in-memory appliances as opposed to buying boxes with fast SSD&#x27;s instead? Sure you spill out to disk, but do you take that big of a hit compared to the enormous cost of keeping everything in memory?

Lol was hoping it was a combination of awk and paste :)<p>That always makes me chuckle.<p>Honestly though ... Jenkins + bash + cloud storage and you&#x27;ll be surprised at how many big data problems you can solve with a fraction of the complexity.

This seems like impressive stats about a relational database technology. But the scrolling on their website doesn&#x27;t work on mobile. So in grand HN tradition, I left and now tell you all about it here, instead of the main point of their invention :)

What is the algorithm used to join the tables? Is it a hash join on `id` and `k` or using the fact that the ids are sorted and using a kind of galloping approach?

Python 2.7 can do it in 0.0867 usec (Intel i7);<p><pre><code> $ python2.7 -m timeit &#x27;n=10**9; (n*n + n) &#x2F; 2&#x27; 10000000 loops, best of 3: 0.0867 usec per loop </code></pre> (Admittedly, I killed `n=10<i></i>9; sum(range(1,n+1))`.)

Great article, actually. Typical HN comments on performance optimizations are complaints like &quot;this isn&#x27;t a real world use case&quot; or things like that. Most of which, they miss that comparing baseline performance metrics against two systems is still genuinely interesting in and of by itself, and acts as a huge learning catalyst to understanding what is going on. I think this article did a great job of making an honest comparison and discussing what is going on, so props to the team! (We did something similar as well, where we compared cached read performance against Redis, and were 50X faster - here: <a href="https:&#x2F;&#x2F;github.com&#x2F;amark&#x2F;gun&#x2F;wiki&#x2F;100000-ops-sec-in-IE6-on-2GB-Atom-CPU" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;amark&#x2F;gun&#x2F;wiki&#x2F;100000-ops-sec-in-IE6-on-2...</a> ).

I know its just a benchmark for comparison, and it is awesome. I love seeing cool comparisons like this, but why do I care that this particular benchmark is faster than Spark? What sort of analytics will be affected by this improvement, and will it actually be saving me time on real world use cases?

why would you choose values between 1 and 1000 for the right side? why not 1000 values between 1 and 1 billion?

In case the author reads this: I can&#x27;t read well with that font, unless I zoom in all the way. Doesn&#x27;t happen with anything else (Win10, 14in laptop, Chrome)