Show HN: my database engine for GPU

Excellent work! I have a very small (marketing-related) nitpick:<p>Would you consider hosting your project on github instead of SourceForge? I don't say this to be trendy, github just has a much more pleasing interface in my opinion. It's also faster, subjectively at least.

Congratulations, well done!<p>Our team has built an in-memory DB (hybrid, both column and row store available, also using advanced compression). We have investigated GPUs for scans, we found that while they can outperform CPUs, the bottleneck is the transfer of the data to the card. Since the card's memory is only in the GB's, and our databases are typically much larger this is kind of limiting, since you have to copy data back and forth.<p>Do you know what the maximum amount of GPU memory for a blade-like system currently is?

Super cool, and thanks for sharing!<p>Are you familiar with APL/K/Q/J ? These implement column store databases which a programming language significantly more expressive and usable. K and J are also significantly faster than any other SQL database -- it would be interesting to compare K's ultra-optimized in-memory processing to the GPU -- I'm not at all sure GPU is going to win here in practice.<p>See <a href="https://github.com/kevinlawler/kona" rel="nofollow">https://github.com/kevinlawler/kona</a> for an open source K implementation (the wiki has docs). Same language, newer version (with database integration) and english language instead of ascii chars is called Q see <a href="http://kx.com/q/d/" rel="nofollow">http://kx.com/q/d/</a> (kdb1, q1 if you like longer, kdb, q if you like terser), and <a href="http://code.kx.com/wiki/Main_Page" rel="nofollow">http://code.kx.com/wiki/Main_Page</a>

Very, very cool. I'm always happy to see more ways to utilize the tremendous performance/price of GPUs.<p>So, what are the pros and cons of this approach? For a given hardware and energy budget, what kinds of task would be better suited to this kind of GPU implementation? Conversely, what kind of tasks would still be faster to run on a Core i7?

now this is the kind of thing i come to HN for... but why oh why sourceforge?

This is awesome. :)<p>Someone did something similar for PostgreSQL here:<p><a href="http://wiki.postgresql.org/wiki/PGStrom" rel="nofollow">http://wiki.postgresql.org/wiki/PGStrom</a>

That looks absolutely brilliant. So is the database all stored in the GPU memory? Or in main memory? Or does your engine handle storing to disk as well?<p>Pretty novel use of a GPU though, and if it's quick, it could be really useful.

Looks like a very interesting approach to me given the recent developments.<p>Still I have a few questions regarding your numbers. Can you specify how much data you actually stream from disk? Because when doing some back of the envelope calculations (8 columns out of a 1.8B rows assuming 4b each) you would need to stream ~300MB/s from disk. Which seems very unlikely for your setup (except you have a super fast SATA drive, or SSD).<p>Now, with only 4GB RAM, your are constantly filling your RAM and transporting data to GPU. Plus you need to compress the data somewhere (on GPU?).<p>Do I understand the manual.txt correctly, that you can only achieve the performance when the data is written compressed to disk before? (While sorting it?)<p>I may be wrong, but the group by and join looked like versions requiring sorted data.<p>Can you please give a little more details on that?

Interesting. I wonder how a GPU could best be put to use in a database engine. Joins, I suppose? Would be cool to read more about this implementation, however the readme is a bit sparse.

antonmks, did you consider using OpenCL instead? If yes - why CUDA outweighed it?

Column store, compare it with MonetDB (open source) and similar. This design is suitable for read-mostly databases. It's a trade-off, faster reads and much better compression. But the price is writes usually become an order of magnitude slower. It isn't good for transactional work (OLTP).<p><a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS" rel="nofollow">http://en.wikipedia.org/wiki/Column-oriented_DBMS</a><p>Also the author talks (other comments) about transfer rate when the real issue with GPUS is latency. Most GPUs don't have decent RAM caches and it's very hard to cover the latency.<p>Also: <a href="http://reddit.com/r/programming/comments/oxq6a/a_database_engine_that_runs_on_a_gpu_outperforms/c3kvts6" rel="nofollow">http://reddit.com/r/programming/comments/oxq6a/a_database_en...</a>

Very nice. I'm glad people are using their GPU's for something more productive than farming bitcoins.

nvidia gpus only, right? does it work with ati as well? CUDA is probably more advanced, so it's no surprise.

I love this kind of novel thinking. I'd definitely like to read more detailed articles on how this works.

What is the point of using "å" instead of "a"?